What the data for this category looks like
The data for this category comes from registered process documents, raw material batch records, DCS systems, weighing systems, LIMS, chromatography data systems, and QMS. Daily generated documents include batch production records, material balance sheets, reaction and purification records, cleaning validation records, impurity profile reports, deviation and change documents. Data is updated per production batch and each process step. Most documents contain fields such as batch numbers, process parameters, material usage, and test results, with units including degrees Celsius, kilograms, liters, and similar units.
What constraints these characteristics impose on the "document parsing and chunking" link
Multi-source heterogeneous data sources lead to differences in document formats, including both structured table records and long-text process descriptions. The parsing link must adapt to the structural characteristics of different documents. The high-frequency update feature of data per batch and process step requires that chunking operations do not cross batch or step boundaries, to avoid mixing process data from different batches. Numeric fields such as process parameters and test results contained in documents must retain the association between fields and their corresponding context, to prevent loss of parameter correspondence after splitting. Long documents such as complete impurity profile reports have large lengths, which imposes additional requirements on the capacity and timeout handling of parsing and chunking.
How to set the configurations
These configurations are based on FastGPT v4.17.0.
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Documents contain process parameters and long-text descriptions. This range balances context association and chunk granularity, and avoids cross-batch splitting |
chunkSplitter | Split by paragraph + table header | Documents contain structured tables and paragraph text. Splitting by paragraph combined with table headers can retain the association between fields and their corresponding content |
customPdfParse | Enabled | Complexly formatted PDF documents such as impurity profile reports need to be parsed. Custom parsing can retain the structural integrity of tables and text |
UPLOAD_FILE_MAX_SIZE | 1000 MB | A single batch production record may include multiple attached documents. This upper limit covers the total document size for most production batches |
PARSE_FILE_TIMEOUT_SECONDS | 600–900 seconds | Parsing hundred-page process documents requires a long time. This range avoids timeout errors for large file parsing |
chunkSettingMode | Manual configuration | Chunking parameters need to be adjusted for different document types such as batch records and impurity profile reports. Manual configuration provides greater flexibility |
Three common mistakes
- A timeout error occurs when calling the parsing interface, with logs showing waiting for parsing results timed out. Cause: The
PARSE_FILE_TIMEOUT_SECONDSparameter was not adjusted. The parsing time for large impurity profile reports exceeds the default threshold. - The chunked results after parsing include cross-batch process data, and field associations are disrupted. Cause: Splitting by batch boundaries was not set, and
chunkSplitterwas not adapted to use batch numbers as the split trigger point. - An attribute read error occurs when calling the custom PDF parsing tool for local deployment, with the interface displaying Cannot read properties of undefined. Cause: The compatible versions of FastGPT and the parsing tool were not matched, and there is a version conflict in the tool's dependency packages.
How to confirm the configuration is correct
- Upload a single production document that meets the maximum business specification, and confirm that the parsing process has no timeouts or abnormal errors.
- Extract a document containing structured tables, and check whether the chunked results retain the context association between tables and their corresponding text.
- Test the adaptation effect of chunking parameters for different types of business documents, and confirm that the splitting boundaries comply with the division rules of batches and steps.
- Verify the calling logic of the custom parsing tool, and confirm that the content returned by the tool is complete and the format meets expectations.
The values provided here are common starting points and should be measured against the reader's own samples.