What Data Looks Like for This Category
Data for this category primarily comes from cell banks and seed lots, process analytics, cleanroom monitoring, MES, LIMS, and equipment integrity testing. Daily documents include cell bank or seed lot records, culture and purification records, aseptic filling records, environmental monitoring, media fill simulations, sterility testing, and batch release materials. Documents contain fields such as batch numbers, culture temperatures, cleanroom sedimentation bacteria counts, and equipment run durations. Units include degrees Celsius, colony-forming units, hours, and others. Data update frequencies fall into three categories: per batch, real-time, and per monitoring plan. Document formats vary across sources. Some use structured table formats, while others are process description texts.
What Constraints These Characteristics Impose on the "Document Parsing and Chunking" Step
Documents in this category contain tightly linked batch information and precise parameters. Chunking operations must avoid splitting continuous records from the same batch. Doing so breaks the contextual connections required for cross-review. Document formats vary across sources. Some documents exported from LIMS include nested tables and compliance annotations. The parsing module must support multi-format text extraction. Real-time cleanroom monitoring data requires fast parsing. Batch release materials generated per batch need complete chunking. The parsing and chunking workflow must adapt to different update rhythms to avoid timeouts or missed critical content. Additionally, compliance annotations and parameter notes in documents must be fully extracted. Failure to do so compromises compliance judgments for subsequent cross-review.
How to Set Configurations
| Configuration Item | Suggested Setting | Rationale for This Setting |
|---|---|---|
chunkSize | 800–1200 characters | This category’s documents contain batch-linked precise parameters. Chunking must retain contextual integrity to avoid splitting critical parameter groups |
chunkSplitter | Split by paragraph + semantic meaning | Daily documents include structured records and process descriptions. Semantic splitting preserves the independence of individual records |
customPdfParse | Enabled | Compliance documents for this category are mostly PDF formats with tables. Custom parsing accurately extracts batch and parameter fields from tables |
PARSE_FILE_TIMEOUT_SECONDS | 300–600 seconds | Single-batch documents may include multiple subsidiary records. Processing times are long, so the setting must avoid timeout interruptions to the parsing workflow |
chunkSettingMode | Manual configuration | Chunking requirements vary across document types such as culture records and environmental monitoring records. Manual configuration adapts to specific scenarios |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Batch release materials for this category may include multiple attachments. A larger file upload capacity must be allowed |
Three Common Mistakes
- Logs show split-related error messages, or parsed chunk results lack critical parameters. The cause is incorrect setup of the semantic splitting rules for
chunkSplitter, which leads to splitting continuous text containing linked parameters. - In FastGPT v4.17.0 deployed via Docker, the file parsing module returns empty content. The cause is incorrect configuration of the container’s file mount path, which prevents the parsing service from accessing uploaded document files.
- Parsing takes longer than expected. The cause is failure to adjust the value of
PARSE_FILE_TIMEOUT_SECONDS, or settingchunkSizetoo small which leads to too many chunks and increased processing load.
How to Confirm the Configuration Is Correct
- Upload a typical cell bank record document. Check if the parsed text fully extracts core fields such as batch numbers and culture parameters.
- Review the
chunkSizeandchunkSplitterconfigurations. Confirm that chunked results do not split linked parameter records from the same batch. - Test the parsing workflow for the largest standard compliance document. Confirm that parsing does not trigger timeout errors.
- Call the FastGPT parsing interface. Confirm that returned chunked data includes all expected document content with no missing fields.
All provided configuration values are common starting points. Adjust them based on your own specific samples and use cases.