What Data Looks Like for This Category
Petroleum refining scenario data covers multiple system sources, including DCS/APC operation data, LIMS laboratory reports, tank farm metrology records, crude oil evaluation reports, equipment management ledgers, and energy system data. Daily generated documents include operating procedures, production plans, material balance sheets, distillate outlet quality reports, interlock ledgers, shift handover records, and change management documents.
Data update cycles cover real-time, shift-based, and daily frequencies. Document structures are mostly a mix of structured tables and process-oriented text, with fields including process parameters such as distillation range, flash point, and sulfur content, with clear unit identifiers attached.
What Constraints Do These Characteristics Impose on the "Document Parsing and Chunking" Link
The multi-source mixed data and document characteristics of petroleum refining scenarios impose multiple constraints on the document parsing and chunking link.
Documents produced by multiple systems have diverse formats, including PDF procedures, Excel multi-sheet reports, and plain text shift handover records. Parsing tools must support these different file types and structured content.
Parsing tasks must support short response times for high-frequency real-time and shift-based data, to avoid timeouts that disrupt quality traceability processes.
Documents with mixed structures include process tables, flowcharts, and parameter text. Chunking processes must retain the integrity of structured information, to avoid splitting that breaks cross-field association logic.
Multi-unit coupled business workflows require parsing results to associate document content across different units. Parsing tools must support cross-document context association.
How to Configure Settings
This configuration is based on FastGPT v4.17.0.
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
PARSE_FILE_TIMEOUT_SECONDS | 300–600 seconds | Refining scenarios include multi-sheet Excel files and long PDF procedures, requiring adaptation to large file parsing durations |
chunkSize | 800–1200 characters | Process documents include long paragraphs and structured tables; this range can retain complete parameter context |
chunkSplitter | Split by paragraph and table boundaries | Refining documents mix text and tables; splitting by boundaries avoids damaging structured information |
customPdfParse | Enabled | Need to retain original images and table structures in PDFs to avoid misparsing graphic and text content via OCR |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Refining scenarios include large-capacity crude oil evaluation reports and multi-unit summary reports |
chunkSettingMode | Custom mode | Need to adjust chunking rules for different document types (such as Excel reports, PDF procedures) |
Three Common Misconfigurations
- The symptom is that after parsing a PDF document, the returned result contains no embedded original images, only text content identified via OCR. The cause is that the
customPdfParseconfiguration is not correctly enabled, or the text-only extraction option is selected. - The symptom is that after parsing an Excel report with multiple sheets, only the content of the default sheet is extracted, and data from other sheets is not included. The cause is that chunking rules are not configured to support multi-sheet recognition, or the cross-sheet extraction function is not enabled during parsing.
- The symptom is that when parsing a large-capacity crude oil evaluation report, the task returns a timeout error with a 504 status code. The cause is that the
PARSE_FILE_TIMEOUT_SECONDSconfiguration value is lower than the actual required parsing duration, and adaptation to large file parsing rhythms is missing.
How to Verify the Configuration Is Correct
- Upload an Excel report with multiple sheets, check whether the parsing result includes content from all sheets, and confirm that the multi-sheet extraction configuration is active.
- Upload a PDF process procedure with original images, check whether the parsing result retains embedded original images, and confirm that the
customPdfParseconfiguration is correctly set. - Upload a large-capacity document, verify that the parsing task does not trigger a timeout, and confirm that the
PARSE_FILE_TIMEOUT_SECONDSconfiguration meets actual needs. - View chunked text fragments, confirm that there is no unreasonable splitting across tables or paragraphs, and verify that the
chunkSizeandchunkSplitterconfigurations are adapted to the document structure.
The values provided are common starting points and should be measured against samples from the actual deployment scenario.