What the data for this category looks like
Daily documents for this category include batch production records, material balance ledgers, reaction and purification logs, cleaning validation reports, impurity profile test results, and deviation and change request documents. Data sources cover registered process documents, raw material batch ledgers, DCS systems, weighing recording systems, LIMS laboratory information management systems, chromatography data systems, and QMS quality management systems. Data is generated and updated with each production batch and completion of each process step. Most documents contain batch identifiers, reaction parameters, impurity test items, and compliance check fields. Available formats include structured tables, long-text process records, and professionally exported test files.
What constraints these characteristics impose on document parsing and chunking
Documents in this category contain multi-step process-related data, professional test results, and compliance records. Process logical connections must be retained when chunking, and cross-process reaction flow segments must not be split. Multi-source heterogeneous document formats require the parsing module to adapt to structured tables, long-text logs, and professionally exported test files. Key fields such as batch identifiers and impurity test items must be accurately identified as chunking anchors. Frequently updated data generated per batch requires parsing tasks to quickly process single-batch documents, and parsing timeouts caused by overly large single documents must be avoided. Compliance-related deviation and change documents must fully retain associated original record segments, and front and back related content of the same event must not be split during chunking.
The following configurations are based on FastGPT v4.17.0.
How to set configurations
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Matches the length of single process descriptions in batch production records, avoids splitting critical reaction steps |
chunkSplitter | Split by paragraph + semantic anchors | Adapts to the logical connections of multi-step process documents, retains anchor fields such as batch numbers and impurity items |
customPdfParse | Enable structured table parsing mode | Adapts to complex table structures in impurity profile reports and material balance ledgers, accurately extracts test values and batch information |
UPLOAD_FILE_MAX_SIZE | 500 MB | Covers the total size of full-process documents for a single batch, prevents upload failure due to exceeding document size limits |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Adapts to the parsing time of long-text chromatography export files and multi-step process records, prevents mid-task timeout interruptions |
chunkSettingMode | Manual configuration | Adjust chunking parameters for different document types such as process records and test reports to meet compliance parsing requirements |
Three common configuration errors
- Symptom: After deploying a local PDF parsing tool, no parsing option appears when uploading active pharmaceutical ingredient manufacturing record PDFs to the FastGPT knowledge base. Cause: The
customPdfParseparameter is not correctly configured to link the local parsing service, or port mapping is not opened to the FastGPT container network. - Symptom: Multi-column test tables in impurity profile reports have misaligned fields and missing values after parsing. Cause: The structured table parsing mode of
customPdfParseis not enabled, and the default chunking does not adapt to the nested structure of professional medical tables. - Symptom: After modifying the default maximum chunk length, re-uploading the document still uses the original default value. Cause: The
chunkSettingModewas not switched to manual configuration mode, or the configuration was not saved to the FastGPT global settings file.
How to confirm the configuration is correct
- Upload a single-batch batch production record PDF, and check if the parsed content retains complete reaction step paragraphs and batch identifier fields to confirm the structured parsing function is active.
- View the FastGPT uploaded file management page, and confirm the uploaded document does not trigger a size limit warning and matches the set file size range.
- Enter the knowledge base chunk configuration page, and check that the current parameters match the manually configured content to confirm the chunking mode has been switched to manual configuration.
- Submit a parsing task for a long-text impurity profile report, and confirm the task completes within the set timeout period with no timeout errors.
The values provided are common starting points and should be measured against the reader's own samples.