What this type of data looks like
The data for this category comes from production operations, process, scheduling, equipment, and laboratory departments of oil refining enterprises. Daily documents include operating procedures, production plans, material balance sheets, distillate export quality reports, interlock ledgers, shift handover records, and change management documents. Data sources cover DCS/APC systems, LIMS, tank farm metering, crude oil evaluation, equipment management, and energy systems. Update frequencies include real-time, per-shift, and daily. Most documents are structured tables and process descriptions, with fields including material flow rates, operating parameters, component proportions, and others. Units involve tons per hour, degrees Celsius, cubic meters, and similar measurements.
What constraints do these characteristics impose on document parsing and chunking
The data characteristics of this category impose multiple constraints on document parsing and chunking. Multi-source heterogeneous system data has significant format differences, so parsing tools must adapt to different document structures. The high-frequency update rhythm requires parsing tools to support fast incremental processing to avoid delays. Structured tables with multi-dimensional parameters, such as material balance sheets and distillate export quality reports, require accurate identification of headers and associated row data to prevent chunking from splitting business logic. Documents containing trigger conditions, such as interlock ledgers, require retention of contextual associations to avoid logical breaks after splitting. Differences in field units across different documents require synchronous retention of metadata during chunking to prevent loss of unit information during subsequent vectorization. Additionally, the business characteristic of coupled multiple continuous units requires retention of logistics association information between units during chunking, to avoid splitting cross-unit process descriptions.
How to configure the settings
These configurations are written based on FastGPT v4.17.0.
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Most documents in this category are structured content with parameters. 800–1200 characters can retain the complete logic of a single operation or material balance, avoiding splitting critical associated information |
chunkSplitter | Split by paragraph + header | Documents in this category include operating procedures and tabular content. Splitting by paragraph and header can retain the integrity of business units and adapt to multi-format documents |
customPdfParse | Enabled | PDF documents in this category are mostly scanned or encrypted process files. Enabling custom parsing can adapt to complex layouts and improve text recognition accuracy |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Oil refining bid documents may include process drawings and historical data files for multiple units, so large-capacity file uploads need to be supported |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Parsing large or multi-page process documents requires longer processing time. 600 seconds can cover the parsing duration of most conventional documents |
chunkSettingMode | Custom mode | Documents in this category have large format differences. Custom mode allows flexible adjustment of chunking rules to adapt to different types of bid documents and data files |
Three common mistakes
- Issue: After uploading a scanned PDF format shift handover record, the parsing result contains a large amount of garbled text or no valid text. Cause: The
customPdfParsecustom parsing function is not enabled, and the default parsing logic does not adapt to the OCR recognition requirements of scanned layouts. - Issue: After uploading a material balance Excel file, some material flow rate parameters are not correctly associated after vectorization indexing, and key information is missing from retrieval results. Cause: The
chunkSplitterrule adapted to structured tables is not configured, and the default chunking logic splits the association between headers and corresponding data rows. - Issue: When uploading a bid document containing process descriptions for multiple units, the upload task triggers a file size limit error. Cause: The
UPLOAD_FILE_MAX_SIZEparameter is not adjusted, and the default limit cannot accommodate large-capacity bid document packages.
How to confirm the configuration is correct
- Upload a typical distillate export quality report PDF, and check whether the parsed text completely retains the parameters and corresponding values.
- Upload a material balance Excel file, verify that the chunked content includes complete headers and corresponding row data, with no field splitting.
- Upload a test document that meets the business capacity requirements, confirm that the parsing task does not trigger timeout or file size limit errors.
- Adjust the
chunkSizeparameter, compare the chunking results under different values, and confirm that the integrity of a single set of business logic is retained.
The values provided are common starting points and should be measured against the reader's own samples.