What the data for this category looks like
Data for this category comes from online composition analysis, DCS systems, laboratories, kiln and grinding equipment, raw fuel ledgers, and energy metering devices. Data update frequencies fall into three categories: real-time, shift-based, and daily. Daily documents include raw material batching plans, kiln operation records, laboratory reports, quality control notices, equipment inspection records, and energy reports. Document fields include equipment number, inspection point, detection value, and operation parameters. Corresponding units include megapascals, degrees Celsius, tons per hour, and others.
Constraints on document parsing and chunking
Real-time updated DCS and inspection data requires parsing and chunking to support high-frequency incremental imports. This avoids data disconnection caused by overly long chunk intervals. Multiple document types have significant structural differences. Long-text operation records and short-form inspection records use different segmentation logic. Chunking logic for mixed document structures must be adapted. Document fields contain values with multiple units. During chunking, context for associated fields must be retained. This prevents values from becoming separated from equipment numbers and detection times after splitting. Daily generated laboratory reports and quality notices must be chunked by batch. This enables subsequent precise recall. Batching plan documents during raw fuel fluctuations contain multiple linked parameter sets. During chunking, the association between parameters must be retained. This avoids loss of adjustment logic after splitting.
Configuration settings
This document is compatible with FastGPT v4.17.0.
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Adapts to the mixed structure of long-text operation records and short-form inspection records for this category, retains parameter-related context |
chunkSplitter | By paragraph + semantic boundary | Matches the natural segmentation logic of multiple document types, avoids splitting that disrupts the complete logic of inspection records and batching plans |
customPdfParse | Enabled | Adapts to PDF documents with tables such as laboratory reports and quality notices, accurately extracts fields and values |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Supports batch import of large-volume documents such as monthly kiln operation records and energy reports |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Addresses parsing time for large-volume PDFs and multi-table documents, avoids mid-process interruptions |
chunkSettingMode | Manual configuration | Adjusts chunking parameters for different document types, adapts to the differentiated needs of real-time inspection data and monthly reports |
Three common errors
- An error indicating unsupported format appears when uploading Excel format files. The cause is that custom parsing rules adapted to multi-column tables are not configured, and only basic text parsing is enabled.
- The chunking result after parsing loses the association between values and equipment numbers. The cause is that
chunkSizeis set too small, which disrupts the context binding between fields during splitting. - A connection failure error occurs when calling the parsing service after deployment. The cause is incorrect Docker container port mapping configuration, and the parsing service port is not exposed to the host machine.
How to confirm correct configuration
- Upload a single Excel-format inspection record table, check whether all columns of fields and values are fully extracted after parsing.
- Upload a laboratory report PDF containing multiple tables, confirm that the chunked result retains the associated data within the tables.
- Submit a large-volume monthly kiln operation record document, check that no timeout error occurs during the parsing process.
- Adjust
chunkSizeto different ranges, verify that the chunked text retains the complete association between parameters.
The values provided are common starting points and should be measured against the reader's own samples.