This document is compatible with FastGPT v4.17.0.
What the data for this category looks like
Documents in petroleum refining scenarios mainly come from DCS/APC systems, LIMS laboratory management systems, tank farm metering systems, crude oil evaluation databases, equipment management systems, and energy management systems. Daily generated documents include operating procedures, production plans, material balance sheets, distillate export quality reports, interlock ledgers, shift handover records, and change management documents. Data update frequencies fall into three categories: real-time, per shift, and daily. Document fields typically include unit numbers, process parameter thresholds, material flow rates, test indicator values, and units such as degrees Celsius, cubic meters, and megapascals.
Constraints for Document Parsing and Chunking
The heterogeneous document sources and multi-tempo update characteristics of this scenario require document parsing to support multiple formats including tables, plain text, and structured reports. This avoids missing cross-unit associated parameter context. The coupled logic of multiple continuous units means a single document may contain process parameters from multiple stages. Chunking must retain adjacent process-related information to prevent loss of process logic after splitting. Real-time and high-frequency updated documents require parsing and chunking workflows to adapt to short-cycle batch processing needs. This avoids data synchronization delays caused by timeouts. Additionally, the precision requirements for process parameters mean chunks cannot randomly merge or split complete indicator description paragraphs.
Configuration Settings
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Adapts to the length of single-section parameter descriptions in petroleum refining process documents, avoiding splitting complete process indicator descriptions |
chunkSplitter | Line break + custom delimiter | Compatible with line breaks in shift handover records and entry separators in interlock ledgers, retaining the original document logic |
customPdfParse | Enable Marker parsing mode | Adapts to process PDF files containing tables and encrypted metadata, improving the accuracy of structured content extraction |
UPLOAD_FILE_MAX_SIZE | 3000 MB | Meets the upload requirements for single large crude oil evaluation reports or multi-unit joint plan documents |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Covers the parsing and chunking time required for large process documents, avoiding task failure due to timeouts |
chunkSettingMode | Manual configuration | Allows adjustment of chunking parameters for different types of refining documents, adapting to heterogeneous data scenarios |
Three Common Mistakes
- Uploading a 3 MB PDF process document triggers an upload error. The cause is failure to adjust the
UPLOAD_FILE_MAX_SIZEconfiguration. The default threshold is insufficient to accommodate documents of this size. - After setting a custom delimiter, chunking results either merge multiple paragraphs or split a single complete paragraph. The cause is failure to match the delimiter logic with the
chunkSizevalue, and failure to balance line breaks in the document and process entry separation rules. - Processing a PDF after deploying Marker and configuring environment variables triggers an error. The cause is failure to enable the corresponding parsing mode for
customPdfParse, leading to inability to recognize structured table content in refining documents.
How to Verify Correct Configuration
- Upload a single typical distillate export quality report PDF, and check if the parsed text fully retains test indicators and corresponding values.
- Import a process plan document containing multi-unit associations, and verify that the chunking results retain parameter association information for adjacent processes.
- Upload a crude oil evaluation report larger than 3 MB, and confirm that the upload and parsing processes do not trigger timeouts or errors.
- Adjust
chunkSizeto different values, and verify that the chunk length matches the preset character range.
The values listed here are common starting points, and should be assessed against samples used in the target deployment.