What Data for This Category Looks Like
Refinery production, process, scheduling, equipment, and laboratory departments generate daily documents including operating procedures, production plans, material balances, distillate quality reports, interlock ledgers, shift handover records, and change management documents. Data sources cover DCS/APC, LIMS, tank farm metering, crude oil evaluation, equipment management systems, and energy systems. Update frequencies include real-time, per-shift, and daily. Document structures include long-form text descriptions and multi-column structured tables. Fields involve process parameters with units such as crude oil density, distillation range, and product flash point.
Constraints on Document Parsing and Chunking
Documents from multiple sources have notable format differences: PDF operating procedures, XLSX material balance reports, and PPT change management files. The parsing module must support multiple formats and accurately extract structured content. Update frequencies range from real-time to daily, so chunking logic must adapt to documents with varying update rhythms. This prevents index lag caused by overly large chunks for high-frequency updated documents. A large number of process parameter tables with units appear in the documents. When chunking, the row and column structure of tables and field associations must be preserved to avoid losing data correspondence after splitting. The business characteristic of coupled multiple units requires chunked content to be linked to corresponding units or batch information, preventing mixed arrangement of cross-scenario content.
Configuration Settings
This guidance uses FastGPT v4.17.0. Recommended configurations are listed below:
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Adapt to the paragraph length of process parameter-containing documents in petroleum refining, to avoid a single chunk containing unrelated content across multiple units |
chunkSplitter | Split by paragraph + table | Match the mixed structure of both long-form procedures and structured tables in petroleum refining documents |
customPdfParse | Enable structured parsing mode | Accurately extract fields and table content from PDF operating procedures and interlock ledgers |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Adapt to the upload requirements of large crude oil evaluation reports and multi-unit joint change documents |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Meet the parsing time requirements of large XLSX material balance reports and multi-page PDF interlock ledgers |
chunkSettingMode | Manual configuration mode | Adjust chunking parameters for different types of petroleum refining documents, such as shift handover records vs. crude oil evaluation reports |
Three Common Misconfigurations
- Phenomenon: Uploading a PPT-format change management document causes a parsing task failure. The interface displays the error: "the argument ‘windows-1252’ is invalid encoding". Cause: Correct encoding parsing rules are not configured for common multilingual mixed content in petroleum refining documents, such as Chinese process descriptions paired with English equipment models. The default encoding adaptation is not suitable.
- Phenomenon: After uploading an XLSX-format material balance report, table rows in the knowledge base chunking results are incorrectly merged, and multiple columns of process parameters cannot be matched. Cause: Structured table chunking rules are not enabled, and the default chunking logic splits by character length without retaining the table row and column structure.
- Phenomenon: Uploading a large crude oil evaluation report causes a parsing task timeout failure. Cause: The
PARSE_FILE_TIMEOUT_SECONDSparameter is not adjusted to a sufficient duration, and the default timeout cannot cover the parsing time of large documents.
How to Verify Proper Configuration
- Upload a typical petroleum refining document, such as an XLSX distillate quality report. Check if the parsed text fully retains the table row and column structure and parameter units.
- Submit multiple test documents of different formats. Confirm that no encoding errors or timeout prompts appear for parsing tasks.
- Retrieve chunked content in the knowledge base. Verify that chunked content corresponds one-to-one with original document paragraphs and table areas.
- View chunk metadata. Confirm that each chunk is linked to the corresponding document name and source path.
The values provided here are common starting points, and should be evaluated against the reader's own samples.