What data for this category looks like
Core data sources for this category include hot metal and scrap data, L2 process control data, laboratory test reports, MES system data, energy medium system logs, and equipment status monitoring data. Daily generated documents include heat plans, batching lists, converter smelting records, composition reports, continuous casting process records, slab judgment sheets, with update frequencies covering per heat, real-time, and per shift. Documents contain fields such as heat number, cast number, process temperature, element content, and equipment operating parameters, with units including degrees Celsius, percentage, ppm, and ton.
What constraints these characteristics impose on document parsing and chunking
Since data is updated in real time per heat and cast, documents must be strictly split by production unit to avoid ledger summary errors caused by cross-heat content merging. In scenarios with multiple document types, parsing logic must adapt to both structured tables (such as composition reports) and unstructured long text (such as smelting records). Format differences across multiple data sources require parsing configurations to support flexible adjustments to accommodate PDF, Excel, and other files exported from different systems. In addition, the consequences of business errors are severe. The chunking process must retain key process and quality parameters to prevent field loss that affects subsequent summary and report generation.
How to configure settings
The following configurations are based on FastGPT v4.17.0:
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Adapts to single-section content of a single heat smelting record, avoids splitting key process parameters or merging information across paragraphs |
chunkSplitter | Custom delimiter (newline character + heat number marker) | Matches the hierarchical structure of business documents split by heat or cast, prevents content from different heats from being merged |
customPdfParse | Enable table extraction + text block retention | Adapts to structured table data in composition reports and continuous casting process records, avoids loss of extracted fields |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Reserves sufficient parsing and chunking time when processing multi-page continuous casting process records or batch heat ledgers |
chunkSettingMode | Manual mode | Allows adjustment of chunking parameters for different document types (such as smelting records vs slab judgment sheets) to adapt to the diversity of business documents |
Three common mistakes
- Phenomenon: Cross-heat merged paragraphs still appear after setting custom delimiters. Cause: The heat number is not used as a mandatory delimiter marker, only the default newline character is used for splitting, and the heat-level structure of business documents is not matched.
- Phenomenon: An error pop-up with incorrect information appears when processing PDF ledgers, or the parsing process stops directly. Cause:
PARSE_FILE_TIMEOUT_SECONDSis not configured to extend the timeout period, or the uploaded file exceeds theUPLOAD_FILE_MAX_SIZElimit, causing parsing to interrupt. - Phenomenon: The element content field of the composition report is empty in the chunking result. Cause: The table extraction function of
customPdfParseis not enabled, and only plain text is extracted, resulting in loss of structured data.
How to confirm the configuration is correct
- Upload a single heat smelting record PDF, check whether the chunking result splits paragraphs by heat number, with no cross-heat content merging.
- Upload a composition report PDF, verify that the chunking result contains complete fields such as element content and detection time, with no missing data.
- Upload a single ledger file over 50MB, confirm that the parsing process does not trigger a timeout error, and the corresponding chunks are generated after completion.
- Adjust the
chunkSizeparameter, verify that the chunk length conforms to the preset character range, with no excessively long or short single chunks.
The values provided are common starting points and should be measured against the reader's own samples.