What this category of data looks like
Daily documents for this category include operating procedures, production plans, material balances, distillate export quality reports, interlock ledgers, shift handover records, and change management documents. Data sources include DCS/APC, LIMS, tank farm metering, crude oil evaluation, equipment management systems, and energy systems. Data updates happen in real time, per shift, and daily. Most documents are structured tables, process step descriptions, and numerical reports. They include fields such as raw material properties, unit load, product fractions, and quality indicators. Units cover mass, volume, temperature, time, and other dimensions.
What constraints these characteristics impose on document parsing and chunking
This category has heterogeneous document types and multi-source data. Parsing and chunking must adapt to multiple formats, including structured tables and text-graphics combined process descriptions. Real-time, shift-based, and daily update cycles require short parsing time thresholds. Delays must be avoided to prevent harm to ledger summary efficiency. High-risk interlock ledgers and change management files require chunking to retain contextual associations. Splitting must not break logical connections. Material data from multiple coupled units require parsing to distinguish fragments from different data sources. Cross-unit data confusion must be avoided.
Configuration Settings
The following configurations are based on FastGPT v4.17.0. They adapt to document parsing needs for petroleum refining scenarios.
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Adapts to the average length of structured numerical paragraphs and process text in petroleum refining ledgers, avoiding loss of contextual associations for unit load and material balance data after splitting |
chunkSplitter | Semantic segment + fixed character truncation | Adapts to multi-format documents, splits Excel tables by sheet and row blocks, retains graphic-text structure for PDFs, matching the heterogeneous content characteristics of refining ledgers |
customPdfParse | Enable original image retention mode | Avoids forced OCR on operation manual PDFs, retains original images and layout, meeting the text-graphics combined requirements of refining operating procedures |
PARSE_FILE_TIMEOUT_SECONDS | 300–600 seconds | Covers parsing time for large crude oil evaluation reports and multi-sheet ledgers, adapting to real-time/shift-based document processing cycles |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Supports uploading and parsing of single large unit operation logs and annual material balance summary files |
chunkSettingMode | Manual configuration | Adjusts chunking rules for different document types (Excel, PDF, DOC) to adapt to parsing needs of heterogeneous data sources |
Three Common Misconfigurations
- After importing a text-graphics combined refining operating procedure PDF, the parsing result only contains OCR text without original images. Cause: The original image retention mode of
customPdfParseis not enabled, and the default forced OCR parsing logic is activated. - After importing an Excel file containing multiple unit ledgers, only the default sheet data is read, and other sheet names and corresponding content are not obtained. Cause: The chunking rules are not configured to adapt to the Excel multi-sheet structure, and the sheet name recognition logic is not enabled.
- After importing a PPT or DOC format refining change management document, targeted parsing logic is not triggered, and the parsing result has chaotic formatting. Cause: Incorrectly assuming that PDF enhancement functions cover all non-PDF documents, and the corresponding parsing configuration is not selected based on document type.
How to Verify Proper Configuration
- A single refining operating procedure PDF with original images is uploaded. The parsing result is checked for retention of original images and text layout.
- An Excel production ledger with three or more sheets is imported. The parsing result is checked for inclusion of all sheet names and corresponding data.
- A large material balance report of around 100 MB is uploaded. The parsing time is checked to fall within the preset
PARSE_FILE_TIMEOUT_SECONDSrange. - Parsing trigger logic for documents of different formats (DOC, PPT, PDF) is tested. Confirmation is made that PDF enhancement only applies to PDF documents.
The values provided here are common starting points and should be measured against the reader's own samples.