What this category of data looks like
Data for this category originates from slab or billet batches, heating furnaces, mill automatic control systems, online thickness and width measuring equipment, surface inspection devices, and MES systems. Daily documents include rolling plans, heating schedules, rolling specifications, plate shape and thickness records, surface quality reports, and quality certificates. Fields cover coil numbers, thickness (unit: mm), heating temperature (unit: ℃), surface defect levels, batch association information, and more. The data update frequency is per coil or per billet, with real-time synchronization of detection and control data from production links. Document structure presents a mixed feature of standardized and non-standardized formats based on differences in business links.
What constraints do these characteristics impose on the "document parsing and chunking" link
Multi-source and heterogeneous data sources require parsing components to support multiple document formats including CSV, Excel, and PDF, while retaining metadata associations across documents from different systems. The high-frequency real-time update feature demands adaptation to small-batch, high-frequency parsing tasks, to avoid excessive single-task time consumption. Documents contain precise numerical values and structured tables, so chunking logic must avoid splitting key business fields, while retaining contextual associations. The business requirement for coil-level traceability means core metadata such as coil batch numbers must not be broken up during chunking, to ensure complete business links can be associated during retrieval.
How to set the configurations
The following configurations are adapted for FastGPT v4.17.0.
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | This category of documents contains precise numerical values and batch association information. Excessively long chunks will lose contextual associations, while excessively short chunks will increase retrieval overhead. This setting is calibrated based on the average length of rolling specifications and plate shape records |
chunkSplitter | Split by paragraph + title | This category of documents has clear chapters for schedules and specifications. Splitting by title retains the integrity of business logic, and adapts to the batch-level structure of rolling plans |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Documents such as single-batch quality certificates and surface quality reports contain multi-page detection data. Sufficient time is required for OCR and text extraction to avoid timeout truncation |
customPdfParse | Enabled | Quality certificates and rolling specifications in this category are mostly scanned documents or encrypted PDFs with tables. Enabling custom parsing optimizes table recognition and chunking effects |
UPLOAD_FILE_MAX_SIZE | 1000 MB | When uploading multiple documents from the same batch in bulk, this setting accommodates complete detection records and specification files for a single coil. This value is calibrated based on actual measurements of total document size per batch in a hot rolling mill |
chunkSettingMode | Manual configuration | This category of documents has large format differences (MES export tables, log files, PDF reports). Manual configuration allows precise adaptation of chunking rules for different document types |
Three common configuration mistakes
- Symptom: Unable to select table datasets in the knowledge base. Cause: The
customPdfParseconfiguration is not enabled. The default parsing logic recognizes tables in rolling records as plain text, and does not generate associable structured table entries. - Symptom: Surface defect description sections are missing from parsed file content. Cause: The
chunkSplitteris configured to split by fixed character count, without retaining paragraph boundaries. This causes the latter half of long paragraphs to be lost when truncated. - Symptom: Parsing tasks time out and fail, returning status code 504. Cause: The
PARSE_FILE_TIMEOUT_SECONDSparameter is not adjusted. Parsing a single document containing multi-page online detection data takes longer than the default threshold.
How to confirm the configuration is correct
- Upload a typical hot rolled plate shape and thickness record document. Check if the parsed text retains complete coil numbers, thickness values, and detection time fields.
- Adjust the
chunkSizeparameter. Compare chunking results across different values, to confirm that the chunk length matches the average paragraph length of the document. - Upload a quality certificate PDF containing multi-page tables. Check if parsed tables are correctly recognized and retain row and column structures.
- Submit a bulk parsing task. Confirm that no tasks experience timeout errors, and that the
PARSE_FILE_TIMEOUT_SECONDSparameter does not trigger abnormal truncation.
The values provided are common starting points, and should be evaluated against the reader's own samples.