What this type of data looks like
Data sources include engineering orders, sand, gravel, cement admixture inspection records, moisture content measurement data, concrete mixing plant control system logs, and laboratory test block records. Daily documents cover mix proportion notices, opening identification forms, production and transport orders, raw material inspection reports, test block strength reports, and similar items. Update frequency matches each production batch, each raw material delivery, and daily laboratory tests. Document fields include mix proportion parameters, aggregate moisture content, water usage, slump indicators, test block strength values, and similar items. Units include cubic meters, kilograms, percentage, megapascals, and others.
What constraints these characteristics impose on document parsing and chunking
High-frequency updates for each production batch and raw material delivery require parsing speed to align with real-time needs, to avoid timeout interruptions. Daily documents include structured PDF reports exported from mixing plant systems and scanned handwritten laboratory records. This demands parsing logic that adapts to multiple formats. Professional fields such as slump and megapascal-related attributes require retaining context during chunking. This prevents splits that break parameter logic. File sizes vary widely for high-volume batch uploads. This requires limiting single-file size and parsing timeout thresholds to prevent service blocking.
How to configure the settings
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Ready-mixed concrete documents contain continuous mix proportion parameters and moisture content correction logic. This length preserves complete context for a single set of parameters, avoiding splits that break associated relationships |
chunkSplitter | Mixed paragraph + fixed length | Daily documents include both structured table paragraphs and continuous test instructions. Mixed splitting balances format integrity and reasonable chunking |
customPdfParse | Enabled | Structured PDF reports exported from mixing plant systems require parsing. Enabling custom parsing preserves the original layout and associated relationships of table fields |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Single-batch raw material inspection reports and monthly production records have large file sizes. This upper limit covers conventional batch upload requirements |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Large monthly production log PDFs take longer to parse. This threshold avoids mid-parsing timeout interruptions |
chunkSettingMode | Manual configuration | Chunking rules must be adjusted for different document types such as mix proportion notices and test block reports. Manual configuration adapts to specialized scene requirements |
Three common errors
- Symptom: After local deployment, uploading PDF files results in empty data processing, with no content in search tests. Cause: The
customPdfParseconfiguration is not enabled. Structured PDF reports exported from mixing plant systems cannot be parsed, so file content cannot be extracted. - Symptom: Some PDF files are recognized, while others show empty content. Cause: The mixed splitting rule for
chunkSplitteris not applied. The text extraction logic for scanned PDF documents does not take effect, leading to parsing failures for some files. - Symptom: Data processing times out during knowledge base training. Cause: The
PARSE_FILE_TIMEOUT_SECONDSsetting is too low. It does not match the parsing time required for large monthly production log files.
How to confirm correct configuration
This verification process uses FastGPT v4.17.0:
- Upload a single mix proportion notice PDF. Wait for parsing to complete, then view the parsed text fragments. Confirm that fields such as mix proportion parameters and aggregate moisture content are fully displayed.
- Batch upload 10 single-vehicle production and transport order PDFs. Check that the data processing progress bar completes normally, with no timeout errors.
- Access the search test function. Input "moisture content of a batch of sand and gravel". Confirm that the returned results include fragmentary content from the corresponding documents.
- Check the file upload limit in the system backend. Confirm that the single-file size does not exceed the
UPLOAD_FILE_MAX_SIZEconfiguration value.
The values provided here are common starting points, and should be evaluated against the reader's own samples.