What Data for This Category Looks Like
Data for this category mainly comes from registered process documents, raw material batch ledgers, DCS systems, weighing records, LIMS systems, chromatography data systems, and QMS quality management systems. Data updates follow the rhythm of each production batch and each process step. Daily documents include batch production records, material balance ledgers, reaction and purification process records, cleaning validation records, impurity profile reports, deviation and change application documents. Most document fields include batch identifiers, reaction parameters, yield data, impurity content, solvent residue levels, and similar items. Corresponding units include degrees Celsius, kilograms, milligrams per kilogram, ppm, and others.
Constraints on Document Parsing and Chunking
Documents in this category are mostly a mix of structured and semi-structured process records, with strong logical correlations between fields. A single document may contain continuous production data across multiple steps. Frequently updated batch data requires retaining contextual associations after chunking, to avoid splitting that breaks the integrity of process steps. Documents aggregated from multiple sources have inconsistent formats. Some documents include handwritten annotations or scanned file formats, so parsing logic for non-standard text must be adapted. Additionally, documents such as impurity profile reports contain a large number of specialized chemical terms. The integrity of these terms must be retained during chunking to avoid splitting that causes professional information to break.
How to Configure Settings
This configuration is based on FastGPT v4.17.0.
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Documents in this category include long paragraphs of process steps and professional data. This range can retain complete information for a single reaction step or impurity analysis, and avoid splitting professional terms |
chunkSplitter | By paragraph + semantic boundary | Process steps and field correlations in documents of this category are tightly linked. Splitting by semantic boundaries can avoid destroying logical correlations, and adapt to mixed structured and semi-structured documents |
customPdfParse | Enabled | Some documents in this category are scanned files or annotated PDFs. Enabling custom parsing can optimize the extraction effect of non-standard text and improve text integrity before chunking |
PARSE_FILE_TIMEOUT_SECONDS | 120 seconds | A single batch production record or impurity profile report may contain multiple pages. This duration can cover the parsing and chunking process for complex documents, and avoid timeout failures |
chunkSettingMode | Manual configuration | The professional characteristics and format differences of documents in this category are large. Manual configuration can accurately adapt to the chunking needs of different document types and meet the requirements of retaining professional information |
UPLOAD_FILE_MAX_SIZE | 500 MB | Batch documents in this category may include multi-page chromatography data attachments. This upper limit can cover common large-size document upload requirements |
Three Easy-to-Make Mistakes
- Phenomenon: Fields are empty after parsing, such as solvent residue data in impurity profile reports not being extracted. Cause:
customPdfParseis not enabled, which cannot adapt to professional document formats with tables or annotations, resulting in loss of structured fields. - Phenomenon: Semantic breaks appear in chunking results, such as the temperature and yield of a single reaction step being split into different chunks. Cause:
chunkSplitteris configured to split by fixed characters, and semantic boundary logic is not enabled, destroying the associated integrity of process steps. - Phenomenon: Task times out, and the system returns an error related to parsing timeout. Cause: The
PARSE_FILE_TIMEOUT_SECONDSparameter is not adjusted, and the parsing duration of a single multi-page batch production record exceeds the default setting, causing the process to interrupt.
How to Confirm Correct Configuration
- Upload a typical impurity profile report, and check whether the parsed text includes complete professional fields such as impurity content and solvent residue levels.
- Select a segment of batch production record containing continuous process steps, and verify whether the chunking results retain complete parameters and associated data for a single reaction step.
- Upload a single batch document with fewer than 500 pages, and confirm that no timeout error occurs during the parsing task.
- View the knowledge base chunking preview interface, and confirm that no professional chemical terms are split in the chunked text.
The values provided are common starting points and should be measured against the reader's own samples.