What This Category's Data Looks Like
The core data for this category comes from registered process documents, raw material batch records, DCS systems, weighing systems, LIMS, chromatography data systems, and QMS. Daily generated documents include batch production records, material balance documents, reaction and purification step records, cleaning validation records, impurity profile reports, and deviation and change application documents. Data is updated once per production batch or process step. Fields include batch number, reaction temperature, yield, solvent residue content, impurity content, and others. Some documents are scanned PDFs or Excel files with nested tables.
Constraints on Document Parsing and Chunking
Multi-source heterogeneous data formats and high-frequency update cycles require chunking rules to balance semantic coherence and timeliness. Scanned PDFs and Excel documents with nested tables increase parsing difficulty and require targeted handling. Incorrect splitting of strongly related content such as reaction steps and impurity profiles will damage the complete information required for process logic and compliance reviews. High-frequency batch-level updates require chunking operations to quickly adapt to new document structures, preventing missing key associated data during retrieval due to rigid chunking rules.
Configuration Settings
This configuration is based on FastGPT v4.17.0.
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Ensures semantic coherence for long documents such as reaction steps and impurity profiles, avoiding splitting that disrupts chemical route logic |
chunkSplitter | By paragraph + semantic boundary | Adapts to the multi-step nested structure of batch production records, preserving the integrity of process logic |
customPdfParse | Enable OCR enhancement mode | Handles scanned impurity profile reports and cleaning records, resolving text recognition issues for scanned documents |
PARSE_FILE_TIMEOUT_SECONDS | 300–600 seconds | Adapts to the time consumption of multi-source data combined parsing, avoiding parsing timeout for large batch records |
chunkSettingMode | Manual configuration | Adjust chunking rules for different document types (such as chromatography data tables, deviation reports) to adapt to field differences |
Three Common Misconfigurations
- Uploading Excel material balance files results in incorrect vector indexing, with missing fields or inconsistent numerical units during retrieval. This occurs when custom chunking rules are not configured for Excel documents with nested tables, causing table content to be split incorrectly.
- Scanned impurity profile reports fail to recognize text after upload, with empty parsing results or garbled characters. This occurs when OCR enhancement mode for
customPdfParseis not enabled, and only the basic PDF parsing function is used. - API calls to upload large batch production records result in timeout errors. This occurs when the
PARSE_FILE_TIMEOUT_SECONDSparameter is not adjusted to the recommended range, causing parsing time to exceed the default threshold.
How to Verify Correct Configuration
- Upload a single impurity profile report, check that the parsed chunks retain complete semantic units including impurity names, residue amounts, and detection methods.
- Enable
customPdfParse, upload a scanned cleaning record, verify that the parsing result has no garbled characters and that cleaning cycle and equipment number fields can be retrieved. - Adjust
PARSE_FILE_TIMEOUT_SECONDSto 600 seconds, upload a batch production record larger than 10MB, confirm that no timeout error occurs during parsing. - Upload an Excel material balance file with tables, check that table rows and columns are not split incorrectly after chunking, and that field correspondences are correct.
The values provided are common starting points and should be measured against the reader's own samples.