This configuration is based on FastGPT v4.17.0.
What data looks like for this category
Daily documents for continuous basic chemical production include operating procedures, process index cards, DCS trends, laboratory reports, interlock bypass records, special work permits, shift logs, and more. Data sources include DCS/SIS systems, online analyzers, LIMS laboratory management systems, equipment condition monitoring tools, energy metering devices, and raw material tank metering data. Data update frequencies fall into three categories: real-time, per-shift, and daily. DCS trends update in real time, shift logs are generated per shift, and annual process ledgers are archived periodically. Document fields include process parameters (with standard units such as temperature, pressure, and flow), interlock point numbers, alarm disposal procedures, shift output, raw material consumption values, and more. Some documents are structured forms with fixed formats, while others are free-text records.
What constraints do these characteristics impose on document parsing and chunking
The data characteristics of this category impose multiple constraints on the document parsing and chunking step. First, documents from multiple sources with varying update frequencies require support for both small real-time files (such as single-shift logs) and large historical archive files (such as annual DCS trend reports). The chunking logic must balance fast parsing with complete information retention. Second, document types include structured forms and free text. Parsing rules must adapt to fixed-format process index cards and unstructured shift log records, to avoid splitting critical interlock information. In addition, documents related to strong interlock control must maintain parsing accuracy. Chunks must not lose core content such as point numbers, alarm thresholds, and disposal requirements, otherwise the accuracy of subsequent ledger summary and report generation may be compromised.
How to configure the settings
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Adapts to the mixed structure of basic chemical ledgers, which includes both short fields (such as process indicators) and long text (such as shift log records), and avoids splitting critical interlock point information |
chunkSplitter | Semantic chunking | Adapts to the structure of different types of documents such as DCS trends and shift logs, and retains contextual associations |
PARSE_FILE_TIMEOUT_SECONDS | 300–600 seconds | Addresses hundreds of pages of historical ledger files, and avoids parsing timeouts for large files |
UPLOAD_FILE_MAX_SIZE | 500–1000 MB | Compatible with large-volume files such as annual process reports archived by enterprises |
customPdfParse | Enable pdf-marker adaptation mode | Adapts to process procedure PDFs with annotations and page numbers, and retains the original format structure |
chunkSettingMode | Manual configuration | Adjust chunking parameters for different document types (such as shift logs, interlock records) to avoid one-size-fits-all settings |
Three common misconfigurations
- An error occurs when using doc2x to parse hundreds of pages of PDF files, while it works correctly for dozens of pages. Phenomenon: The backend returns a parsing timeout error, and the interface shows file parsing failed. Cause: The default parsing timeout setting is too short, and does not align with the parsing time required for large-volume ledger files.
- A JSON parsing error occurs during online processing. Phenomenon: When parsing laboratory reports exported from LIMS in JSON format, missing fields or format errors are returned. Cause: Custom parsing rules are not configured for the structured JSON fields of chemical ledgers, and the default parsing logic cannot recognize proprietary field names.
- After integrating pdf-marker, parsing results remain pending, and the file times out. Phenomenon: After uploading a process report that exceeds the threshold, the task queue has no response, and finally returns a timeout status code. Cause: No file queuing mechanism is configured, and large files occupy parsing resources, causing subsequent tasks to block and exceed the preset timeout time.
How to verify correct configuration
- Upload a single-page shift log PDF and a 500-page historical trend report, and confirm both parsing tasks complete within the set
PARSE_FILE_TIMEOUT_SECONDSduration. - Parse a document containing interlock bypass records, and check that the chunked content retains key information such as point numbers, trigger times, and disposal requirements.
- Review the
UPLOAD_FILE_MAX_SIZEsetting, and confirm it matches the maximum volume requirements of enterprise archived files. - Test parsing laboratory reports exported from LIMS in JSON format, and confirm all preset fields are correctly extracted.
The values provided are common starting points and should be measured against the reader's own samples.