What the data for this category looks like
Data for this category comes from design history files, device and material batches, MES systems, clean environment monitoring equipment, sterilization equipment, and inspection and calibration systems. Daily documents include production process specifications, batch production records, clean zone records, sterilization batch records, inspection reports, software version records, adverse event and traceability materials. Fields cover unique device identifiers, sterilization parameters, clean environment indicators, batch numbers, and more. Data updates occur once per batch or per product. Clean environment monitoring data updates on a fixed schedule. The size of individual documents increases as the volume of archived batches grows.
What constraints do these characteristics impose on document parsing and chunking
The heterogeneous multi-source document structure, high-frequency update rhythm, and high-risk nature of this category impose multiple constraints on document parsing and chunking. Different document types have significant format differences. Scanned documents and native PDFs coexist, requiring adaptation for multi-format parsing. High-frequency updated batch records and monitoring data require efficient and stable parsing processes to avoid delays that impact production links. Key fields such as unique identifiers and sterilization parameters must be fully retained. Chunking operations cannot split associated information across paragraphs, as this will prevent subsequent audit links from matching critical data. At the same time, the accuracy of parsing results directly relates to product traceability and compliance requirements. Missing parsing or incorrect content splitting must be avoided.
How to set the configuration
This document is written for FastGPT v4.17.0. The configuration items and recommendations are as follows:
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | This category of documents contains long process steps and batch-associated data. This range balances context integrity and retrieval accuracy |
chunkSplitter | Paragraph + header filtering | Documents have uniform batch record headers. Duplicate metadata such as batch numbers must be excluded, while core process and inspection content must be retained |
customPdfParse | Enable OCR mode | Some sterilization records and clean zone records are scanned documents. Text and parameter information embedded in images must be extracted |
PARSE_FILE_TIMEOUT_SECONDS | 300–600 seconds | Single-batch archived records may contain multi-page scanned content. This range avoids task failure caused by parsing timeouts |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Large multi-batch archived files have significant size. Support for large file upload and parsing is required |
chunkSettingMode | Manual configuration | Chunking requirements vary significantly across different document types, such as process specifications vs. batch records. Parameters must be adjusted flexibly |
Three common misconfigurations
- Symptom: After uploading a PDF file with embedded images, the parsing result has no corresponding text for the images, or key parameter fields are empty. Cause: The OCR mode of
customPdfParseis not enabled. Only native PDF text is parsed, and content from scanned images cannot be extracted. - Symptom: Timeout errors occur when parsing large sterilization batch records, and the interface shows task failure. Cause: The value set for
PARSE_FILE_TIMEOUT_SECONDSis too low, and does not match the parsing time required for multi-page scanned files. - Symptom: The table dataset option cannot be selected when creating a knowledge base. Cause: The table dataset parsing function is not enabled in system configuration, or this configuration item is not available in the current version.
How to confirm the configuration is correct
- Upload a single-page scanned clean zone monitoring record. Check if the parsing result includes key fields such as temperature and humidity, and monitoring time. Confirm that the OCR mode is active.
- Upload a multi-page sterilization batch record. Wait for parsing to complete. Check if the chunking result retains the association between sterilization parameters and batch numbers. Confirm that the chunk length is appropriate.
- Upload a multi-batch archived file larger than 100 MB. Confirm that the upload and parsing process is not interrupted. Verify that the file size configuration is reasonable.
- View the parsing logs. Confirm that no timeout errors are triggered. Verify that the timeout setting matches the current document size requirements.
The values provided are common starting points and should be evaluated against the reader's own samples.