What the data for this category looks like
This category covers hot rolling, cold rolling, section steel, and steel pipe enterprises. Daily generated documents include rolling plans, heating schedules, rolling specifications, plate shape and thickness records, surface quality reports, and quality certificates. Data sources include slab or billet batches, heating furnaces, mill automatic control systems, online thickness and width measuring equipment, surface inspection devices, and MES systems. Update frequency is real-time, per coil or per piece. Documents contain both structured tables and unstructured process descriptions. Fields include batch number, thickness (unit: mm), rolling speed (unit: m/s), surface defect level, and others. The structure of each document varies significantly by type.
What constraints do these characteristics impose on document parsing and chunking?
Data for this category updates in real time, per coil or per piece. The volume of documents associated with a single batch varies. This requires document parsing and chunking to support high-frequency uploads and processing of document collections of different sizes. Documents include both structured tables and unstructured process descriptions. Parsing rules must distinguish between different document types, without breaking the field associations of structured data. Coil-level traceability business requirements demand that chunking retains batch association information, to prevent cross-batch data confusion. Parsing failures or chunking errors may cause serious consequences such as steel coil downgrading or equipment damage. This requires the parsing link to have complete timeout and format error fault tolerance mechanisms.
How to configure the settings
The following configurations are based on FastGPT v4.17.0.
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | This category’s documents include both long-text rolling specifications and compact thickness record tables. This range balances context completeness and retrieval accuracy |
chunkSplitter | Split by paragraph + table | This category’s documents contain structured tables and unstructured process descriptions. Mixed splitting preserves data associations |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Archived documents associated with a single batch may reach a large volume. Sufficient time must be reserved for processing multi-source data merging |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Adapts to the maximum possible volume of archived documents associated with a single batch, and supports batch upload requirements |
chunkSettingMode | Manual configuration | The document structure of this category varies significantly. Manual adjustment matches the chunking logic for different document types |
customPdfParse | Enabled | Some quality certificates for this category are scanned PDF files. Enabling custom parsing optimizes OCR recognition results |
Three common misconfigurations
- An error indicating unsupported file format appears when uploading a CSV rolling plan file. The cause is that parsing rules for structured table documents are not adapted. The default parsing logic does not recognize the structured fields of CSV files.
- Parsing fails when using Qwen series models to parse surface quality report PDFs. The cause is that the
customPdfParseparameter is not enabled. OCR processing for scanned PDF files is not triggered. - Irrelevant cross-batch data appears in retrieval results. The cause is that
chunkSplitterdoes not split by batch tags, causing document blocks from different coils to be mixed together.
How to confirm the configuration is properly set
- Upload a single coil’s thickness record CSV file. Check that the parsed data retains core fields such as batch number and thickness, with no missing fields or garbled text.
- Upload a scanned PDF format quality certificate. Check that the parsed results include surface defect descriptions and batch information, with no obvious recognition errors.
- Upload a 100 MB rolling specification collection. Check that parsing time does not exceed 600 seconds, with no timeout error prompts.
- Adjust
chunkSizeto 600 characters and 1200 characters separately. Upload the same rolling specification document, and compare the chunking results. Confirm that the chunking logic can be adjusted according to the document type.
The values given are common starting points and should be measured against the reader's own samples.