What the data for this category looks like
Daily documents for this category include batch production records, material balance sheets, reaction and purification records, cleaning records, impurity profile reports, deviation and change documents. Data comes from registered process documents, raw material batch records, DCS systems, weighing systems, LIMS, chromatography data systems, and QMS. Update frequency is once per production batch and per process step. Most documents contain professional fields such as multi-step reaction parameters, impurity content data, solvent residue values. Units include molar ratios, ppm, batch numbers and other specialized identifiers.
Constraints for Document Parsing and Chunking From These Characteristics
The professional relevance of multi-step reaction sequences and impurity profile data requires retaining context associations during parsing, to avoid splitting that breaks process logic. High-frequency updates of single-batch documents and heterogeneous formats from multiple systems require parsing tools to support non-standard content such as tables and handwritten annotations converted to electronic format. Unique batch numbers and yield data for each batch must be retained as metadata, and bound to corresponding identifiers during chunking to prevent cross-batch data confusion. Bulk multi-page documents require a stable parsing process to avoid anomalies caused by processing too much content at one time.
Configuration Settings
The following configurations are based on FastGPT v4.17.0.
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Adapts to the context length of multi-step reactions and impurity profiles, avoiding splitting that breaks professional logic |
chunkSplitter | Split by paragraph + semantics | Matches the paragraph-based structure of process documents, preserving the integrity of reaction steps |
customPdfParse | Enabled | Adapts to heterogeneous PDF formats exported from multiple systems, supporting tables and handwritten annotation content |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Supports bulk upload of multi-page documents per batch, covering the needs of hundreds-page process records |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Adapts to the parsing time of large-volume PDFs, avoiding timeout errors during bulk parsing |
chunkSettingMode | Manual configuration | Adjusts chunking rules for professional fields, ensuring that batch numbers and impurity data are not split |
Three Common Configuration Mistakes
- Symptom: Parsing failure occurs when running bulk parsing for hundreds-page PDFs. Cause: The
PARSE_FILE_TIMEOUT_SECONDSandUPLOAD_FILE_MAX_SIZEparameters are not adjusted, so large-volume documents exceed default limits or time out. - Symptom: Batch numbers or impurity content fields are missing from post-parsing chunked results. Cause:
customPdfParseis not enabled, so professional field metadata in heterogeneous documents cannot be recognized. - Symptom: The proportion of duplicate fragments in chunked results is abnormally high, or recall results do not meet expectations. Cause: The
chunkSplitterrule is not adjusted based on the professional structure of process documents, leading to repeated splitting of the same reaction steps.
How to Verify Proper Configuration
- Upload a single process record PDF with more than 500 pages, and confirm that the parsing status shows no timeout errors.
- Randomly select 10 chunked content samples, and confirm that each chunk contains complete process steps or impurity data, with no missing fields.
- View the system parameter configuration page, and confirm that the values of
UPLOAD_FILE_MAX_SIZEandPARSE_FILE_TIMEOUT_SECONDSmatch the preset requirements. - Test bulk upload of multiple batch documents, and confirm that all chunked results are bound to corresponding batch number metadata.
The values provided are common starting points and should be measured against the reader's own samples.