What data for this use case looks like
Technical document sources for petroleum refining scenarios cover DCS/APC systems, LIMS systems, tank area metering systems, crude oil evaluation reports, equipment management ledgers, and energy scheduling systems. Document types include operating procedures, production plans, material balance sheets, distillation outlet quality reports, interlock trigger ledgers, shift handover records, and change management documents.
Data update frequencies include real-time (process fluctuation logs), per-shift (shift handover records), and daily (material balance and quality reports). Document fields include unit number, crude oil density, distillation range temperature, interlock threshold, shift identifier, and more, with corresponding physical units attached.
Constraints for the document parsing and chunking workflow
Multiple continuous units couple via logistics and energy networks. Document chunks must not sever process association logic. Cross-unit material balance descriptions must not be split.
Real-time and frequently updated documents require parsing workflows to support batch processing and fast response. Parsing workflows must also accommodate large-volume annual crude oil evaluation reports.
Process parameter fields use fixed units. Parsing must retain the binding relationship between parameters and units. Units must not be separated from their corresponding parameters during chunking.
Long, multi-page single documents increase parsing time. Timeouts or abnormal resource usage often result.
Configuration Settings
| Configuration Item | Recommended Value Range | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Refining documents include coherent process flow descriptions and parameter groups with units. This length balances context integrity and chunk granularity |
chunkSplitter | Mixed paragraph-based and fixed-length splitting | Adapts to document structures with both coherent operating procedure paragraphs and batch laboratory data tables. It avoids splitting table rows or process steps |
PARSE_FILE_TIMEOUT_SECONDS | 600–900 seconds | Parsing multi-hundred-page crude oil evaluation reports or interlock ledgers takes a long time. This threshold covers most large-file parsing requirements |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Supports uploading complete single-unit operating procedures or annual material balance reports. It adapts to full-document import needs |
customPdfParse | Enabled | Adapts to common table nesting, header/footer annotations, and process flow diagram comments in refining documents. It improves parsing accuracy |
chunkSettingMode | Manual configuration | Flexibly adjust chunking parameters for different document types such as shift handover records and crude oil evaluation reports. It matches business scenarios |
Three Common Configuration Errors
- An error occurs when using
doc2xto parse multi-hundred-page PDFs. Files with tens of pages work normally. The cause is failure to adjust thePARSE_FILE_TIMEOUT_SECONDSparameter. Large-file parsing exceeds the default threshold and triggers a timeout. - The PDF parsing function fails after locally deploying FastGPT v4.17.0. The cause is failure to enable the
customPdfParseconfiguration, or failure to correctly connect the dependent environment of the third-party parsing component. - After integrating pdf-marker, large-file parsing always times out while waiting for results. The cause is failure to set reasonable
UPLOAD_FILE_MAX_SIZEandPARSE_FILE_TIMEOUT_SECONDSvalues. It also stems from failure to configure a parsing queuing mechanism, leading to backlogged concurrent requests.
How to Verify Correct Configuration
- Upload a typical 100-page PDF of a refining operating procedure. Check whether the parsing completion status returns results within the preset timeout period.
- View the parsed chunk content. Confirm that process parameters and their corresponding units are not split into different chunks.
- Upload a complete crude oil evaluation report larger than 500 MB. Confirm that the upload and parsing processes are not blocked.
- Test the parsing effect of two different document types: shift handover records and interlock ledgers. Confirm that the chunking logic matches the structural characteristics of the corresponding documents.
The values provided are common starting points and should be measured against the reader's own samples.