What the data for this category looks like
Daily documents for this category include batch production records, material balance records, reaction and purification records, cleaning records, impurity profile reports, deviation and change documents. Data sources cover registered process documents, raw material batch records, DCS systems, weighing systems, LIMS, chromatography data systems, and QMS. Updates are triggered per production batch and per process step. Document structures mostly combine structured tables and long text paragraphs, containing fields such as batch identifiers, process parameters, test data, and approval workflows. Field units include degrees Celsius, kilograms, milligrams, hours, and others.
What constraints these characteristics impose on the document parsing and chunking link
Multi-source heterogeneous document sources and high-frequency update rhythms require the parsing link to support structured and unstructured content in multiple formats. Document structures that bind professional process parameters to batches require retaining context associations during chunking, to avoid splitting reaction records for a single step and corresponding impurity test data. Single-batch document volume is moderate but there are many batches with high-frequency updates, so parsing time must be controlled to meet bulk import requirements. Professional chemical terminology and fixed field formats require precise identification of field boundaries during parsing, to prevent incorrect splitting of parameters and descriptive text.
How to set the configuration
This configuration is based on FastGPT v4.17.0.
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Documents in this category include continuous process steps and test data. This range can retain complete context for a single-step reaction or a single test report |
chunkSplitter | By paragraph + heading | Documents in this category mostly use process steps and test items as paragraph and heading divisions, which ensures the chunking logic matches the document structure |
customPdfParse | Enabled | PDF documents in this category often contain embedded tables and professional formatting. Enabling this improves parsing accuracy for structured content |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Multi-document collections for a single batch may reach a large volume, which needs to support bulk import requirements |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Parsing complex impurity profile reports or multi-step records requires longer time, to avoid timeout interruptions |
chunkSettingMode | Manual configuration | Document structures vary widely in this category. Manual configuration allows flexible adjustment of chunking rules to adapt to different document types |
Three Common Misconfigurations
- Symptom: After importing PPT or DOC format documents, structured content cannot be extracted even if
customPdfParseis enabled. Cause: This configuration only applies to PDF format documents, and does not support other office document formats. - Symptom: After importing a CSV format file, only the first two columns of data are parsed correctly, and subsequent column data is lost. Cause: The default chunking rule does not adapt to the structure of multi-column CSV documents, and multi-column table recognition configuration is not enabled.
- Symptom: Search results after chunking cannot match complete process steps or impurity profile reports. Cause: The
chunkSizevalue is too small, splitting reaction records and test data for the same step into multiple chunks, breaking context associations.
How to Confirm the Configuration Is Valid
- Upload a single complete batch production record PDF, and check whether all process steps and associated test data are fully extracted in the parsing result.
- Import a test document containing multiple columns of data, and confirm that the parsed field divisions match the original document structure.
- Bulk import 3 to 5 documents of the same type, and check that no parsing tasks experience timeout interruptions.
- Adjust the
chunkSizeparameter, and verify that the chunked content can fully cover the core information of a single process step or a single test report.
The values provided are common starting points and should be measured against the reader's own samples.