What This Category’s Data Entails
The core documents for this category include cell bank and seed lot records, culture and purification records, aseptic filling records, environmental monitoring data, media fill simulation reports, sterility test results, and batch release documents. Data sources include cell bank management systems, process analysis equipment, cleanroom monitoring systems, MES, LIMS, and equipment integrity testing tools. Data updates follow production batches, real-time monitoring nodes, or established monitoring plans. Most documents combine structured tables and formatted text. Fields include batch number, culture temperature, sterility test results, environmental sedimentation bacteria concentration, and more. Units cover professional production parameters such as degrees Celsius, colony count, and milliliter.
What Constraints Do These Characteristics Impose on Document Parsing and Chunking?
The mixed structured documents and batch/real-time update patterns of this category impose multiple constraints on document parsing and chunking. For documents that mix tables and formatted text, avoid splitting complete table content across pages. This ensures that parameters and associated fields within a table are not split into different chunks. Batch-associated data must be kept within a single chunk. This prevents loss of batch context during retrieval. The binding relationship between professional production parameter fields and their units must be retained during parsing and chunking. This stops parameters and units from being separated in later use. High-volume real-time monitoring data requires limiting individual chunk size to fit vector model processing capabilities, while preserving the integrity of key parameter combinations.
How to Configure Settings
The following are chunking and parsing configuration recommendations for FastGPT v4.17.0:
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | This category of documents contains large volumes of structured tables and professional parameters. This range balances single-chunk information density and retrieval recall accuracy |
chunkSplitter | Split by Markdown headings + table separators | Most documents in this category use batch numbers and test types as heading levels, with tables as core data carriers. Following this rule preserves full document structural integrity |
customPdfParse | Enabled | Most documents for this category are compliance-formatted PDFs. Enabling custom parsing preserves table borders and cell association relationships, preventing scrambled table content after parsing |
PARSE_FILE_TIMEOUT_SECONDS | 300–600 seconds | Parsing large batch release documents or multi-batch combined files takes extended time. This range covers typical document parsing durations, avoiding mid-process timeout interruptions |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Some archived documents in this category contain multi-batch historical data, with large individual file sizes. This upper limit supports uploads of typical archived files |
chunkSettingMode | Manual configuration | Documents in this category have significant structural differences. Manual configuration allows adjusting chunking rules for different document types, meeting the need to preserve professional parameters |
Three Common Configuration Mistakes
- Phenomenon: Uploaded compliance PDF table documents appear complete in the knowledge base preview, but table content is missing when generating responses. Cause: The
customPdfParseconfiguration is not enabled. The default parsing logic cannot fully preserve the association between table cells, causing the table structure to break during output. - Phenomenon: Environmental monitoring auxiliary data is returned before core batch release parameters in retrieval results. Cause: Chunking did not divide core data blocks using document heading levels, and no priority recall rules were configured for professional parameters. This leads to retrieval sorting that does not align with business scenario requirements.
- Phenomenon: Documents processed with custom splitting rules locally fail to match expected retrieval results after being uploaded to the server. Cause: The unified server-side chunking configuration was not used for splitting. Different splitting logic creates chunk boundaries that do not match the semantic encoding of the server-side vector model.
How to Verify Correct Configuration
- Upload a typical document containing structured tables, and confirm the table structure in the knowledge base preview matches the source file.
- Run a chunking test on a complete single-batch document, confirming that core parameters and associated records for the same batch are grouped into a single chunk.
- Upload a single large archived document, confirming the parsing process does not trigger timeout or file size limit errors.
- Adjust chunking configurations, then compare chunking results across different configurations to ensure the binding relationship between professional parameters and their units remains intact.
The values provided are common starting points and should be measured against the reader's own samples.