What this category of data looks like
Data for this category originates from slab or billet batches, heating furnaces, mill automation control systems, online thickness and width measurement equipment, surface inspection devices, and manufacturing execution systems (MES). Daily documents include rolling plans, heating schedules, rolling specifications, plate shape and thickness records, surface quality reports, quality certificates, and other materials. Data updates in real time for each coil or individual steel product. Document structures include structured process tables, semi-structured inspection reports, and unstructured process descriptions. Fields include coil number, batch number, thickness, width, heating temperature, defect type, and additional relevant items. Corresponding units use industrial standard measurements such as millimeters (mm), degrees Celsius (℃), and similar units.
Constraints on Document Parsing and Chunking
Real-time data updates and coil-level traceability requirements demand that document parsing and chunking accurately match individual coil or batch business units. Cross-coil data must not be split or merged. Multiple document types in use require parsing tools to support structured tables, inspection reports with images, and plain-text process specifications. Parsing must differentiate structural features of different documents to complete chunking. Fixed fields and units require the parsing process to fully retain unit information for industrial parameters. Unit loss or incorrect conversion must be prevented. High-frequency, small-volume document updates require the parsing workflow to have fast response speeds. Parsing delays that disrupt the timeliness of subsequent ledger summary and report generation must be avoided.
Configuration Settings
These configurations are based on FastGPT v4.17.0.
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | This category of documents requires complete process and inspection information for a single coil or batch. This range avoids splitting core data for one coil, while ensuring chunked content fits the context window of subsequent large models |
chunkSplitter | Split by paragraph + filter coil number headers | Rolling documents have fixed coil number headers. Filtering duplicate headers prevents cross-coil data from being merged. Splitting by paragraph preserves logical integrity of process and inspection content |
customPdfParse | Enabled | PDF documents for this category contain large numbers of structured thickness record tables and surface inspection images. Custom parsing preserves table field relationships and extracts defect annotation information embedded in images |
PARSE_FILE_TIMEOUT_SECONDS | 120 seconds | Complete ledger PDFs for a single coil may include multiple pages of online inspection data. This duration covers the full process of multi-page parsing and image extraction, preventing parsing interruptions from timeouts |
UPLOAD_FILE_MAX_SIZE | 50 MB | Daily documents corresponding to a single coil are mostly under 50 MB. This setting avoids wasting parsing resources on invalid large files, while supporting the high-frequency document upload requirement |
chunkSettingMode | Manual mode | Chunking rules must be adjusted for different document types such as rolling plans and quality certificates. Manual mode flexibly adapts to the business requirements of coil-level traceability |
Three Common Mistakes
- Symptom: After calling the knowledge base creation API, real-time statuses for parsing in progress, ready, or failed cannot be retrieved. Only a single request result is returned. Cause: The supporting API is not used to poll the parsing status interface. Only the return information from the initial creation request is relied upon.
- Symptom: After uploading a quality certificate PDF containing surface inspection images, the parsing result does not extract defect description content associated with the images. Cause: The image parsing configuration for
customPdfParseis not enabled. Text or annotation information embedded in images is not captured as a result. - Symptom: The chunking result splits thickness records from two different steel coils into the same chunk. Cause:
chunkSplitteronly splits by character length, and does not filter duplicate coil number headers in the document. Cross-coil data is merged into a single chunk as a result.
How to Verify Proper Configuration
- Upload a test PDF containing a complete single-coil ledger. Check if the chunking result includes all core fields for that coil, with no mixed cross-coil content.
- Call the knowledge base creation API. Use the supporting interface to poll parsing status. Confirm that real-time status updates for parsing in progress, ready, and failed can be retrieved.
- Upload a test document containing surface inspection images. Check if the parsing result includes defect descriptions and annotation information corresponding to the images.
- Review the parsed chunk data. Confirm that all industrial parameter units are fully retained, with no unit loss or incorrect conversion.
The values provided here are common starting points. They should be evaluated against the reader's own samples.