What this category of data looks like
Core data for the basic chemical continuous production category comes from DCS/SIS systems, online analyzers, LIMS systems, equipment condition monitoring modules, energy metering systems, and raw material tank monitoring modules. Daily documents include operating procedures, process index cards, DCS trend charts, laboratory reports, interlock bypass records, special work permits, and shift logs. Data update frequencies include real-time (such as DCS trends), per-shift (such as shift logs), and daily (such as laboratory reports). Document fields include process parameters (temperature, pressure, flow, with units of ℃, MPa, m³/h), interlock status bits, shift identifiers, and timestamps. The data mixes text, tables, and time-series data, and has no unified fixed format.
What constraints these characteristics impose on document parsing and chunking
Data sources for this category are scattered and have diverse formats. Real-time DCS trends and per-shift shift logs require chunking to retain time-series and shift context, to avoid splitting that breaks associations between corresponding parameters. Mixed-format documents contain text, tables, and embedded images. Parsing and chunking must adapt to both structured tables and unstructured text, and cannot rely solely on fixed character lengths for splitting. Documents related to critical interlocks, such as interlock bypass records, must retain complete operational context. Chunking must not separate interlock numbers from their corresponding disposal processes. If high-frequency real-time data is chunked into overly large segments, retrieval will fail to match the latest process index changes.
How to set configurations
These configurations are formulated based on the official rules of FastGPT v4.17.0.
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Adapts to mixed-format documents, can fully accommodate the content of a single process index card, and prevents long-text operating procedures from being split into overly fragmented chunks |
chunkSplitter | By paragraph + heading hierarchy | Adapts to operating procedures with chapter structures and time-series DCS trends, retains the logical context of documents |
customPdfParse | Enabled | Parses formatted chemical process PDF documents, retains the basic structure of tables, text hierarchies, and embedded images |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Prevents timeout interruptions during parsing of large archived shift logs or DCS trend files |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Supports uploading compressed packages containing multiple batches of archived documents, meets the long-term data retention needs of continuous production |
chunkSettingMode | Manual configuration | Adjusts chunking rules for different document types (such as shift logs, interlock records) to meet context retention requirements for specific scenarios |
Three common mistakes
- Issue: When uploading a DOCX operating procedure with embedded images, an
Invalid image fileerror is returned. Cause: ThecustomPdfParseconfiguration is not enabled, so the embedded vector image formats common in chemical documents cannot be recognized. - Issue: When parsing a knowledge base, the log reports
slow operation xxxxms, and MongoDB itself responds normally. Cause: ThePARSE_FILE_TIMEOUT_SECONDSsetting is too low, and large DCS trend archive files trigger a timeout before completing full parsing. - Issue: The output of a Markdown-format laboratory report is truncated, and complete test data rows cannot be displayed. Cause: The
chunkSizesetting is too small, splitting complete table rows into different chunks, which prevents complete data from being associated during retrieval.
How to confirm the configuration is correct
Upload a test document containing process index card tables, DCS trend text, and embedded images, and check if the parsed chunks retain complete table rows and continuous blocks of time-series data. Upload an archived shift log compressed package within 1000 MB, and confirm that there are no timeout-related errors during the upload and parsing process. View the parsing log to confirm that there are no abnormal records related to Invalid image file or slow operation. Manually adjust chunkSize to the 800–1200 character range, and verify that long-text operating procedures are not split to separate critical operational steps.
The values listed above are common starting points and should be measured against the reader's own samples.