What This Category of Data Entails
This category covers enterprises producing hot rolled, cold rolled, section steel, and steel pipe products. Their daily documents include rolling plans, heating schedules, rolling specifications, plate shape and thickness records, surface quality reports, and quality certificates. Data is sourced from slab or billet batches, heating furnaces, mill automatic control systems, online thickness and width measurement equipment, surface inspection devices, and MES systems. Updates occur in real time for each coil or individual steel coil. Document fields include batch number, furnace number, rolling temperature, reduction rate, thickness deviation, surface defect type, and other items. Some documents include physical dimension and process parameter values.
Constraints on Document Parsing and Chunking
Real-time batch-level data requires document parsing to support quick reading of associated documents for a single coil, avoiding full loading of redundant content. Multiple types of process documents have significant structural differences: rolling specifications are structured parameter lists, while surface quality reports are text-and-image combined records. This requires chunking logic that adapts to different document structures. Single-coil data has dense fields and physical units. Chunking must retain the association between fields and units, preventing parameters from being separated from their units during splitting. Bid documents need to associate batch information across multiple documents. Chunking must retain batch numbers as association anchors, ensuring subsequent retrieval can accurately match corresponding process data.
Configuration Settings
This configuration is based on FastGPT v4.17.0. The specific parameter configurations are as follows:
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | This category of documents contains dense process parameters and batch information. This range preserves the integrity of parameter groups and avoids splitting that breaks their associated relationships |
chunkSplitter | Paragraph-based + fixed character truncation | Adapts to the structured list format of rolling specifications and the text-and-image paragraph structure of surface quality reports, balancing chunking efficiency and content completeness |
customPdfParse | Enable structured parsing mode | Most PDF documents in this category contain standardized process parameter tables. Structured parsing preserves the association between fields and units |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Parsing large quality certificates or multi-batch rolling plan documents requires a long time. This duration prevents timeout interruptions for conventional document parsing |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Meets the total upload volume requirement for multiple associated documents per batch. Calibrated based on actual measurements to adapt to daily document scales of enterprises |
chunkSettingMode | Manual configuration | Adjustment of chunking rules for different document types is required to adapt to structural differences between rolling plans and surface quality reports |
Three Common Misconfigurations
- Symptom: The parsed text stream is plain text with no formatting, resulting in monotonous front-end display. Cause: Structured parsing mode for
customPdfParsewas not enabled, only raw text was extracted without retaining format structure. - Symptom: Content cannot be parsed from internal network Confluence pages. Cause: A parsing whitelist for internal network access permissions was not configured. FastGPT only supports externally accessible web links.
- Symptom: Chunking results split complete process parameter groups into multiple chunks. Cause: The
chunkSizevalue is too small, breaking the association between parameters and units, resulting in chunked content that cannot independently support process descriptions.
How to Verify Proper Configuration
- Upload a rolling specification PDF document, check if the parsed result retains table and parameter formatting, confirming that
customPdfParseis enabled. - Upload a single 100 MB quality certificate document, verify that the parsing process does not trigger a timeout error, confirming that
PARSE_FILE_TIMEOUT_SECONDSis set appropriately. - Adjust
chunkSizeto different ranges, upload a document containing continuous process parameters, and check if the chunking result retains the integrity of parameter groups. - Test uploading a document link accessible via the internal network, confirm that corresponding access permissions have been configured, and verify internal network resource parsing capability.
The values provided are common starting points and should be measured against the reader's own samples.