What data for this category looks like
Documents for this category use one record per single batch. They include raw milk acceptance records, standardized ingredient lists, pasteurization or UHT heat treatment records, CIP cleaning records, filling records, and microbial test reports. Data comes from milk source detection systems, MES production systems, homogenization and heat treatment equipment, online temperature and flow collection modules, CIP systems, and LIMS laboratory management systems. Updates are generated per milk tank, per batch, or in real time. Fields include batch numbers, physical and chemical test values, microbial test values, equipment operating parameters, and timestamps. Dedicated units such as degrees Celsius, seconds, and colony-forming units are included.
What constraints do these characteristics impose on document parsing and chunking?
Document characteristics for this category create multiple constraints for parsing and chunking:
- Frequently generated real-time or batch-level documents require chunking workflows to support incremental parsing. This avoids performance loss from full reprocessing.
- Heterogeneous data from multiple sources — including equipment operation logs, laboratory test reports, and production process records — must retain source associations during chunking. This prevents cross-document field confusion.
- Batch-bound dedicated parameters such as heat treatment time and temperature, and CIP cleaning parameters require chunking to retain contextual links. This avoids losing the connection between batches and parameters after splitting.
- Batch isolation requirements for perishable raw materials require chunking to split data by batch. This prevents cross-batch data mixing from affecting subsequent judgments.
How to set configurations
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Matches the length of batch parameters and test records in dairy processing documents, avoiding splitting critical parameter combinations |
chunkSplitter | Split by paragraph + batch markers | Aligns with the batch-based segment structure of documents, retaining contextual links between batches and parameters |
customPdfParse | Enabled | Adapts to the format of embedded equipment parameter tables and batch number watermarks in documents of this category, improving parsing completeness |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Supports multi-page integrated heat treatment and CIP record files per batch, preventing large file upload failures |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Adapts to the parsing duration of multi-page test reports and long-cycle equipment logs, stopping timeout interruptions |
chunkSettingMode | Manual configuration | Adjusts chunking rules for the specialized document structure of this category, adapting to batch-bound field logic |
This page's configuration rules apply to FastGPT v4.17.0.
Three common mistakes
- Issue: Parsing task errors occur after uploading local files, returning file size limit or timeout status codes. Cause: The values of
UPLOAD_FILE_MAX_SIZEandPARSE_FILE_TIMEOUT_SECONDSwere not adjusted, failing to meet the parsing needs of multi-page test reports and long-cycle equipment logs. - Issue: Core fields such as batch numbers and test values are missing after parsing some PDF documents. Cause: The
customPdfParseconfiguration was not enabled, failing to recognize embedded tables and batch watermark formats in documents, leading to incomplete parsing. - Issue: Cross-batch parameters are merged into a single chunk in chunking results, losing batch association information. Cause:
chunkSplitterwas not configured to split by batch markers. The default character-length splitting rule was used, damaging the document's batch structure.
How to confirm the configuration is correct
- Upload a complete single raw milk acceptance record PDF. Check that the parsed text includes all batch numbers and test values.
- Adjust the
chunkSizeandchunkSplitterconfigurations, then manually split the test document. Confirm that chunks do not split core parameters across batches. - Upload an integrated CIP record that matches the
UPLOAD_FILE_MAX_SIZEvalue. Verify that the parsing task does not trigger timeout or file limit error messages. - Review the contextual associations of chunking results. Confirm that each chunk contains complete batch-bound parameters.
The values provided are common starting points and should be measured against the reader's own samples.