What this category of data looks like
This category of data primarily comes from daily business records of commercial concrete batching plant control systems, project laboratories, quality control and scheduling departments. Updates occur for each vehicle’s production record, each batch of raw material inspection report, and daily summarized scheduling reports. Daily documents include mix proportion notices, opening identification forms, production and transportation slips, raw material inspection reports, test block strength records, etc. Fields cover aggregate moisture content, water consumption, slump, strength grade, and units include percentage, kilograms per cubic meter, millimeters, and others.
What constraints do these characteristics impose on the "document parsing and chunking" link
The high-frequency updated production slips per vehicle and batch raw material inspection data require document parsing to support fast splitting of single files, avoiding overloading single chunks that would impact subsequent retrieval. In scenarios with mixed document types, the chunker needs to adapt to different document structures. For example, parameter lines in mix proportion notices and table content in test block records must be processed separately to prevent mixing of cross-type content. The feature of fields with multiple units requires parsing to retain original unit information, avoiding business judgment errors caused by separation of units and values after chunking. High-frequency updated data requires that after chunking, the latest raw material inspection data can be quickly associated, preventing loss of cross-chunk information.
How to set the configurations
This configuration is based on FastGPT v4.17.0, customized for the document characteristics of ready-mixed concrete production scenarios:
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Adapts to the field density of single documents such as mix proportion notices and test block records, preventing key parameters from being scattered after single-chunk splitting |
chunkSplitter | Split by paragraph + heading | Adapts to the mixed structure of multiple document types, distinguishing parameter lines in mix proportions from test conclusion paragraphs to avoid mixing cross-type content |
customPdfParse | Enable table recognition mode | Preserves the original row and column structure for table data in raw material inspection reports and strength reports, preventing field confusion after parsing |
UPLOAD_FILE_MAX_SIZE | 500 MB | Compatible with the maximum size limit of single-batch raw material inspection summary files, meeting batch document upload requirements |
PARSE_FILE_TIMEOUT_SECONDS | 300 seconds | Adapts to the parsing duration of single large production and transportation slips, preventing parsing failures caused by timeout interruptions |
chunkSettingMode | Manual configuration mode | Allows adjustment of chunking rules for different document types, adapting to the multi-document structure scenario of this category |
Three common misconfigurations
- Issue: Parsing fails after uploading audio or video files, with an unsupported file format prompt returned. Cause: No parsing plugin for non-document files is configured. The equipment inspection and maintenance documents of this category are primarily text and tables, and do not meet audio and video parsing requirements.
- Issue: An incorrect parameter error is returned by the interface when uploading images via the workflow API. Cause: The image file was not passed in according to the multipart/form-data format, and the file upload field parameters of the interface were not correctly configured.
- Issue: Chunked documents cannot be associated with original business batch information. Cause: The metadata retention function was not enabled, and the batch numbers of mix proportion notices and production and transportation slips were lost during chunking, making it impossible to match corresponding business records after vectorized storage.
How to verify correct configuration
- A standard mix proportion notice PDF may be uploaded, and the parsed text is checked to confirm retention of unit information for original fields such as aggregate moisture content and water consumption.
- The file upload API may be called, with a test block record image passed in, and the parsed result returned by the interface is checked to confirm inclusion of the image’s text content and table structure.
- The chunking logs of the knowledge base may be viewed, and the single-chunk content length is confirmed to fall within the configured range of
chunkSize, with no excessive splitting or merging. - The data storage association may be checked, and the vectorized data after chunking is confirmed to be associated with the batch number of the original document, allowing corresponding business records to be quickly located via the batch number.
The values provided here are common starting points and should be evaluated against the reader's own samples.