What This Category of Data Looks Like
The data for this category comes from DCS/APC systems, LIMS laboratory management systems, tank farm metering systems, crude oil evaluation reports, equipment management systems, and energy management systems. Daily documents include operating procedures, production plans, material balance sheets, distillate export quality reports, interlock ledgers, shift handover records, and change management documents. Data updates occur in real time, per shift, and daily. Documents contain fields such as unit number, process parameter thresholds, equipment number, change order number, and others. Units include MPa, ℃, cubic meters, hours, and additional standard units.
Constraints for Document Parsing and Chunking
Multiple continuous production units are coupled via material and hydrogen networks. Documents often include cross-unit collaborative parameters. Chunking must retain upstream and downstream association logic to avoid breaking process workflows. Raw material property fluctuations lead to wide parameter ranges in documents. Chunking must not excessively truncate key process thresholds and equipment interlock values. High-frequency documents updated in real time or per shift require fast parsing to avoid timeouts. Interlock ledgers and shift handover records related to equipment inspection and maintenance must retain operational context, and key process steps must not be split. Documents exported from different source systems have significant format differences, so multiple parsing rule types must be supported.
Configuration Settings
This configuration is based on FastGPT v4.17.0.
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Adapts to the length of paragraphs containing multiple sets of process parameters and interlock logic in refining documents, avoiding splitting of key thresholds |
chunkSplitter | Split by paragraph + fixed character count | Matches the natural segmentation logic of shift handover records and operating procedures, retaining the integrity of operational steps |
customPdfParse | Enabled | Adapts to formatted PDF documents exported from equipment management systems, retaining field layout and association relationships |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Addresses parsing time requirements for large crude oil evaluation reports and multi-unit interlock ledgers |
chunkSettingMode | Manual configuration | Adjust chunking parameters for different document types such as quality reports and shift handover records to adapt to detailed business scenarios |
Three Common Configuration Errors
- Uploading a large interlock ledger PDF results in empty parsed content or incomplete displayed content. The cause is failure to adjust the
PARSE_FILE_TIMEOUT_SECONDSparameter; large document parsing exceeds the default time limit and is terminated. - Importing Java interface files with their file name suffixes renamed to .txt yields no valid parsing results. The cause is failure to configure custom parsing rules to adapt to code-style text; only splitting as plain text leads to truncation of key logic.
- Parameter and description misalignment occurs when parsing distillate export quality reports. The cause is an overly small
chunkSizeconfiguration, which splits complete paragraphs containing parameters and their corresponding descriptions.
How to Verify Correct Configuration
- Upload a typical petroleum refining operating procedure PDF, check if the parsed chunks retain complete operational steps and parameter thresholds.
- Upload an interlock ledger document with more than 100 pages, verify that the parsing time does not exceed the duration configured by
PARSE_FILE_TIMEOUT_SECONDS. - Import a Java interface file with a renamed file suffix, check if the code structure can be recognized and valid chunks are generated.
- Compare the chunking results of manual configuration and automatic mode, confirm that manual configuration better matches the structural characteristics of business documents.
The values provided are common starting points and should be measured against the reader's own samples.