What data for this category looks like
Data sources for basic chemical continuous production include DCS/SIS systems, online analyzers, LIMS laboratory management systems, equipment condition monitoring modules, energy metering systems, and raw material tank monitoring data. Daily documents cover operating procedures, process index cards, DCS trend charts, laboratory reports, interlock bypass records, special work tickets, and shift logs. Data update frequencies are divided into real-time (such as DCS trends), per shift (such as shift logs), and daily (such as laboratory reports). Documents include numerical fields with clear units such as process temperature (℃), pressure (MPa), flow rate (m³/h), as well as structured and semi-structured content such as interlock logic and work steps.
What constraints these characteristics impose on the "document parsing and chunking" link
Real-time, high-frequency updated DCS trends and shift logs require parsing processes with low latency, to avoid data lag caused by timeouts. Multiple types of documents coexist, including structured process index cards and semi-structured special work tickets. Chunking logic must adapt to different document structures. Key content related to strong interlocks must retain complete logic, and must not be split arbitrarily. Otherwise, the accuracy of subsequent Q&A will be affected. Numerical fields with units must be bound to corresponding parameters, to prevent loss of unit information after parsing that causes data distortion. Large archived documents uploaded in batches must limit single-file size and parsing duration, to avoid task failure.
How to set configurations
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Basic chemical documents include long-text process steps and short numerical parameters. This range balances context integrity and retrieval accuracy |
chunkSplitter | By paragraph + fixed character boundary | Process specifications have clear paragraph structures, while avoiding splitting key continuous content such as interlock logic |
customPdfParse | Enable structured parsing mode | Adapt to charts and numerical fields in DCS trend charts and laboratory reports, and retain the association between original data and units |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Compatibility with large batches of archived DCS trend files uploaded in bulk, to avoid failures during the upload phase |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Reserve sufficient parsing and rendering time for large PDF documents containing multi-page interlock records |
chunkSettingMode | Manual configuration | Adjust chunking parameters for different document types (such as process cards, work tickets) to meet differentiated scene requirements |
Three common mistakes
- Running the
docker run --gpus all -itd -p 7231:7231command results in container startup or image pull errors. The cause is incorrect configuration of host machine GPU driver permissions, or use of a marker_images version incompatible with FastGPT v4.17.0. - FastGPT v4.8.12 cannot properly parse complex chemical formulas in PDFs, while v4.9.0 can return parsing results normally. The cause is differences in support for formula rendering logic between built-in PDF parsing modules of different versions.
- Clicking chunk preview for a PDF file in the knowledge base shows that the file content cannot be read. The cause is failure to enable the
customPdfParseconfiguration, single file size exceeding theUPLOAD_FILE_MAX_SIZElimit, or too shortPARSE_FILE_TIMEOUT_SECONDSsetting causing parsing interruption.
How to confirm configurations are set correctly
- Upload a typical process index card PDF, check that the chunk preview completely retains process parameters and their corresponding units, with no missing content or garbled text.
- Upload a document containing DCS trend charts, confirm that the parsed chunks include chart annotations and corresponding numerical values, with no field loss.
- Upload an archived file with a single size close to
UPLOAD_FILE_MAX_SIZE, confirm that the parsing task completes normally with no timeout error prompts. - Adjust the
chunkSizeparameter, compare the length of chunked text, and confirm that the context coverage range meets scene requirements.
The values provided here are common starting points, and should be evaluated against the reader's own samples.