What this category of data looks like
Documents and materials for refinery and chemical production mainly come from process parameter records, equipment operation and maintenance logs, safety compliance documents, maintenance plans, raw material quality inspection reports, and similar sources. Data update frequencies fall into three categories: real-time (shift inspection logs), daily (daily production operation reports), and on-demand (temporary maintenance plans). Document structures take a mixed form of long professional text paragraphs, process tables with numerical units, structured fields such as equipment ID and job title, and unstructured content. Most fields include industrial parameters such as temperature, pressure, and flow with fixed units.
What constraints do these characteristics impose on the document parsing and chunking link
The mixed structure and precise parameter requirements of refinery and chemical production documents bring multiple constraints to the parsing and chunking link. Documents mixing long text and tables need to retain cell associations and parameter context to avoid losing process logic after chunking. Real-time inspection logs need to support fast upload and parsing to reduce waiting time. Industrial parameters with precise units cannot have their associations with preceding and following text broken during chunking, otherwise complete process logic cannot be matched during retrieval. Single document volume varies greatly, from a few pages of inspection records to dozens of pages of annual compliance documents, so parsing needs to adapt to different volume requirements.
How to set the configurations
Reference only; verify against the target FastGPT release and deployment documentation.
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Refinery and chemical production documents mostly contain continuous process parameters and professional terms. This value range avoids breaking the association between parameters and preceding and following text |
chunkSplitter | Split by paragraph + table header | Adapts to document structures mixing process tables and equipment logs, and retains context independence of different modules |
customPdfParse | Enable table parsing mode | Refinery documents contain a large number of process tables with numerical units. This mode can fully extract parameters and formatting within table cells |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Adapts to the volume requirements of single annual compliance documents and large maintenance plans, and avoids upload interception |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Prevents task failure due to exceeding default parsing duration when processing long documents |
chunkSettingMode | Manual configuration | Adjusts chunking rules for the professional structure of refinery documents, and adapts to context association requirements of industrial parameters |
Three common errors
- Phenomenon: After uploading a PDF document containing process tables, table content is not displayed normally in knowledge base retrieval. Cause: The dedicated table parsing mode of
customPdfParseis not enabled, only plain text content is extracted, and cell associations and formatting of the table are lost. - Phenomenon: In retrieval results after chunking, auxiliary data such as equipment ID has higher matching priority than core process parameters. Cause:
chunkSettingModeis not configured for field priority of refinery and chemical documents, and context weight of core parameters is not retained during chunking. - Phenomenon: After uploading a single annual safety log exceeding the default size, the parsing task times out and fails. Cause:
PARSE_FILE_TIMEOUT_SECONDSis not adjusted to a duration suitable for long document parsing, triggering the default timeout limit.
How to confirm the configuration is correct
- Upload a refinery and chemical document containing process tables, check the parsed preview content to confirm that cell content of the table is fully displayed without content loss.
- Randomly select a segment of text containing continuous process parameters, set
chunkSizeto 800–1200 characters, and check that the chunking result does not break the association between parameters and preceding and following text. - Upload a document exceeding the default upload limit, confirm that the upload is successful and the parsing task completes normally without error prompts.
- Check the chunked text fragments to confirm that core process parameters, their corresponding units and equipment IDs are not split into different chunks.
The values provided are common starting points and should be measured against the reader's own samples.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-09-28.