What this category of data looks like
Typical enterprises include hot rolling, cold rolling, section steel, and steel pipe enterprises. Daily documents include rolling plans, heating schedules, rolling specifications, plate shape and thickness records, surface quality reports, and quality certificates. Data sources cover slab or billet batches, heating furnaces, mill automatic control systems, online thickness and width measuring equipment, surface inspection devices, and MES systems. Data is updated in real time for each coil or individual piece. Documents are organized by steel coil batch, with fixed fields such as thickness (unit mm), plate shape index (unit I), rolling temperature, batch number, and others. A complete record for one coil can cover tens to hundreds of pages.
What constraints these characteristics impose on document parsing and chunking
Documents for this category are organized by steel coil batch and include time-stamped process parameters. When chunking, the association between batches and process parameters must be preserved to avoid splitting that breaks the integrity of single-coil data. The real-time update feature requires the parsing process to support batch processing of large files. A single batch of documents can reach hundreds of pages, which may cause parsing timeouts. The presence of fixed fields and units requires parsing to retain structured information; plain text splitting easily loses key process parameters. Additionally, quality traceability requires recalling complete process segments for a specific batch, so the chunking granularity must balance information integrity and retrieval efficiency. It should not be too fine that context is broken, nor too coarse that retrieval accuracy is affected.
How to set configurations
The following configurations are based on FastGPT v4.17.0, adjusted for the steel rolling and finishing scenario:
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
UPLOAD_FILE_MAX_SIZE | 1000 MB | Adapt to the upload requirement of batch quality reports for multiple coils per batch, preventing large files from being blocked |
PARSE_FILE_TIMEOUT_SECONDS | 600–900 seconds | Handle rolling record documents with hundreds of pages per coil, avoiding interruptions caused by insufficient time |
customPdfParse | Enable structured parsing | Retain the field association of rolling specifications and thickness records, preventing loss of process parameters via plain text splitting |
chunkSize | 800–1200 characters | Balance complete information of single-coil process segments and retrieval granularity, meeting context requirements for quality traceability |
chunkSplitter | Split by paragraph + page boundary | Avoid splitting data across coils or process steps, ensuring the integrity of batch data |
chunkSettingMode | Manual configuration | Flexibly adjust chunking parameters based on the fixed field structure of steel rolling documents |
Three common misconfigurations
- An error occurs when using the parsing tool to process hundreds of pages of PDF files, while documents with tens of pages can be parsed normally. The cause is failure to adjust the
PARSE_FILE_TIMEOUT_SECONDSparameter; large file parsing duration exceeds the default threshold, triggering a timeout error. - A Cannot read properties of undefined error occurs when calling the locally deployed parsing tool. The cause is failure to configure the structured parsing rules for
customPdfParse, leading to failure to properly recognize specific process fields. - Excessive duplicate content appears after chunking. The cause is failure to split by the paragraph + page boundary specified by
chunkSplitter, leading to an overly large overlapping range between adjacent chunks.
How to confirm the configuration is correct
- Upload a 100-page surface quality report per coil, check whether the parsing task completes within the duration configured by
PARSE_FILE_TIMEOUT_SECONDS. - Review the chunked text fragments, confirm that key fields such as rolling batch number and thickness values are not split into different chunks.
- Input a plate shape defect problem of a specific batch for retrieval testing, confirm that the returned chunks include the process parameters and inspection records of the corresponding batch.
- Verify the upload file size limit: upload a document close to the value configured by
UPLOAD_FILE_MAX_SIZE, confirm that upload and parsing can be completed normally.
The values provided are common starting points and should be measured against the reader's own samples.