What this category of data looks like
Ready-mixed concrete production data sources include engineering orders, sand, gravel, cement and admixture inspection records, moisture content measurement data, mixing plant control system logs, and laboratory test reports. Daily documents include mix proportion notices, batch opening identification, production and transport orders, raw material inspection forms, test block records, and strength reports. Fields include sand and gravel usage (kg/m³), slump (mm), strength grade (MPa), test block age (days), and others. Data update frequency follows per vehicle production record, per batch raw material inspection, and daily summary reports. There is no fixed batch cycle.
What constraints do these characteristics impose on the "document parsing and chunking" link
The high-frequency updates and structured document features of this category require document parsing and chunking to retain field integrity, and avoid splitting cross-group mix proportion or inspection data. Dense table-based documents need to adapt to column alignment and header association, otherwise key parameters will be lost during vectorization. For batch uploads of large-volume summary files, the parsing link must have sufficient timeout fault tolerance and chunk limits, while adapting to structural differences across different document types, to avoid generic parsing logic destroying fixed-format report content.
How to set the configuration
This configuration applies to FastGPT v4.17.0.
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Most documents in this category are tables with fixed fields and short explanatory paragraphs. This range can retain complete information for a single set of mix proportions or inspection sheets, and avoid splitting cross-field content |
chunkSplitter | Split by mixed tables and paragraphs | This category of documents contains a large number of structured tables and explanatory paragraphs. Mixed splitting can retain both table structure and contextual association |
customPdfParse | Enabled | Native PDF parsing easily disrupts table column alignment. Custom parsing can adapt to the fixed header format of ready-mixed concrete documents |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Monthly batch raw material inspection reports and test block record summary files can reach hundreds of MB. This upper limit covers conventional batch upload requirements |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Parsing large-volume summary files takes a long time. This setting avoids mid-parsing timeout interruptions |
chunkSettingMode | Manual configuration | The document structures of this category vary widely. Manual configuration can accurately match the parsing needs of different document types |
Three common mistakes
- Phenomenon: After uploading files larger than 10 MB, more than 1000 chunks are generated, and some chunks have vectorization exceptions. Normal state resumes after repeated clicking of retry. Cause: The
chunkSettingModeis not configured to manual mode. The system's default chunking logic does not adapt to the table-intensive documents of this category, leading to loss of table association information during splitting and triggering vectorization exceptions. - Phenomenon: After importing documents generated from Feishu online spreadsheets, some fields are empty or the format is distorted. Cause: The
customPdfParseconfiguration is not enabled. Native parsing cannot adapt to the non-standard format exported from Feishu spreadsheets. - Phenomenon: Parsing times out after uploading batch summary files. Cause: The
PARSE_FILE_TIMEOUT_SECONDSis not adjusted to a value suitable for large files. The default timeout duration is insufficient to complete parsing of summary files.
How to confirm the configuration is correct
- Upload a single-batch mix proportion notice PDF, and check if the parsed chunks retain complete mix proportion parameters and header information.
- Upload a monthly raw material inspection summary file, and verify that the number of generated chunks meets expectations with no obvious splitting errors.
- Adjust
chunkSizeto different ranges, compare the parsed chunk content, and confirm that no cross-field splitting occurs. - Upload batch files within 1000 MB, and confirm that the parsing process does not trigger timeout interruptions.
The values provided are common starting points and should be measured against the reader's own samples.