What Data for This Category Looks Like
Documents related to process specifications and technical standards for petroleum refining scenarios originate from DCS/APC systems, LIMS laboratory management systems, tank farm metering systems, crude oil evaluation reports, equipment management systems, and energy management systems. Document types include operating procedures, production plans, material balance sheets, distillate export quality reports, interlock ledgers, shift handover records, and change management documents. Data update frequencies cover real-time, per-shift, and daily cycles. Distillate export quality reports and interlock ledgers have relatively high update frequencies. Material balance sheets and crude oil evaluation reports have relatively fixed update cycles. Documents include fields such as process parameters, equipment status, and material flow rates, using units including degrees Celsius, cubic meters, and tons.
What Constraints These Characteristics Impose on the "Document Parsing and Chunking" Link
Documents from multiple sources have significant format differences. They include structured PDFs and CSV files exported from systems, plus scanned shift handover records. This requires parsing components to support broad format compatibility. Process documents with multiple coupled units contain cross-unit logistics and energy association information. Chunking must retain logical hierarchies to avoid splitting associated parameters into different chunks. Real-time updated documents require parsing components with fast response capabilities. Excessive parsing timeout will disrupt the real-time requirements of process scheduling. Different document types have large structural differences. Structured table documents and semi-structured text documents need differentiated chunking strategies. General chunking rules cannot adapt to all document types.
Configuration Settings
The following configurations are formulated based on FastGPT v4.17.0.
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Petroleum refining process documents contain multi-unit coupled parameter logic. Chunks that are too long will lose associations, while chunks that are too short will damage the integrity of process units. This range is determined after actual measurement and calibration |
chunkSplitter | Split by paragraph + title anchor | Process specifications and ledger documents have clear device and step title hierarchies. Splitting by titles can retain logical boundaries and avoid splitting cross-device parameters |
customPdfParse | Enable OCR mode | Some documents are scanned shift handover records or encrypted PDFs. Enabling OCR can fully extract text content from images |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Large files such as crude oil evaluation reports and multi-device interlock ledgers have large sizes. This upper limit covers conventional batch upload requirements |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Parsing large process plan documents requires longer processing time. 600 seconds prevents parsing timeout interruptions for conventional large documents |
chunkSettingMode | Manual configuration mode | Different document types require differentiated chunking parameters. Manual configuration allows flexible adjustments to adapt. The default strategy is to split by fixed character count |
Three Common Mistakes
- The symptom is missing parsed document content, with only partial page text included. The cause is that
customPdfParseis not enabled, so text content in scanned or encrypted PDFs cannot be recognized. - The symptom is that chunking results split cross-device process associated parameters into different chunks. The cause is using the default fixed character count splitting strategy instead of switching to title anchor splitting mode, which damages the logical hierarchy of the document.
- The symptom is a timeout error returned after a parsing task is triggered. The cause is that the set
PARSE_FILE_TIMEOUT_SECONDSvalue is too small to complete full parsing of large crude oil evaluation reports.
How to Verify Proper Configuration
- Upload a scanned shift handover record, and check if the parsed result contains all handwritten and printed text content.
- Upload a process specification PDF containing a multi-unit coupled process, and check if chunking results divide content by device title boundaries.
- Upload an interlock ledger file larger than 500 MB, and confirm that the upload and parsing process is not interrupted.
- Call the parsing interface to obtain the returned text stream, and check if the paragraph and title structure of the original document is retained, consistent with the reading logic of process documents.
The values provided are common starting points and should be measured against the reader's own samples.