What this category of data looks like
This category of data comes primarily from production systems of basic chemical enterprises including chlor-alkali, synthetic ammonia, methanol, and soda ash facilities. Sources include DCS/SIS systems, online analyzers, LIMS, equipment status monitoring, energy metering, and raw material storage tanks. Daily documents cover operating procedures, process index cards, DCS trends, laboratory reports, interlock bypass records, special work permits, and shift logs. Update frequencies range from real-time, per-shift, to daily. Documents include clear process parameter fields with corresponding units, such as temperature (℃), pressure (MPa), and concentration (%). Interlock-related records contain key information like trigger time and tag number.
What constraints do these characteristics impose on document parsing and chunking?
The multi-source, multi-update-frequency, and multi-format documents of this category impose multiple constraints on parsing and chunking. Real-time generated DCS trend data must be chunked by time windows to avoid splitting sequential logic. Shift logs and special work permits generated per shift are discrete documents, so chunking rules must be matched to document types. Content related to critical interlocks must retain complete logic, and key tag numbers and trigger conditions cannot be split. Mixed-format documents (tables, timing diagrams, text) require structured parsing to ensure that information from laboratory report tables and process flow diagrams is not lost. Additionally, the confidence of parsing results is directly linked to production safety, so key interlock records or process parameters must not be omitted.
How to configure the settings
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | This category of documents includes long-form process descriptions and structured tables. This range preserves the integrity of individual process index or interlock records |
chunkSplitter | Chunk by paragraph + semantics | Process documents have clear chapter divisions. Semantic chunking avoids splitting interlock logic or continuous process steps |
customPdfParse | Enable structured parsing mode | Documents include DCS trend diagrams and laboratory report tables. Structured parsing preserves table fields and text information associated with diagrams |
PARSE_FILE_TIMEOUT_SECONDS | 300–600 seconds | Parsing large DCS trend PDFs or batch shift logs requires sufficient time to avoid interrupting the parsing process due to timeout |
chunkSettingMode | Manual configuration | Chunking parameters must be adjusted for different document types such as interlock bypass records and shift logs to adapt to multi-format content |
Three common configuration mistakes
- Issue: Parsed process index card tables are output as plain text, and column alignment and unit information cannot be restored. Cause: The structured table parsing mode of
customPdfParseis not enabled, and document content is only split as plain text. - Issue: After calling the knowledge base creation API, real-time updates of the parsing status are not obtained, and the system stays in the parsing state for a long time. Cause:
PARSE_FILE_TIMEOUT_SECONDSis set too short, exceeding the parsing time required for large DCS trend documents, which terminates the status reporting process prematurely. - Issue: Chunked content loses the trigger time and tag number information of interlock bypass records. Cause:
chunkSizeis set too small, which splits complete interlock logic statements and destroys the integrity of key information.
How to verify correct configuration
- Upload a single process index card PDF, view the chunked content parsed by FastGPT v4.17.0, and confirm that table fields and corresponding units are fully preserved.
- Call the knowledge base parsing API, and check if the returned parsing status field includes the enumeration values of parsing in progress, ready, and failed.
- Upload a document containing DCS trend diagrams, and confirm that the chunked content is associated with the process parameter description corresponding to the diagrams.
- Upload batch shift log files, confirm that there are no timeout interruption logs during the parsing process, and that the final number of generated chunks matches the number of documents.
Note: The configuration values listed above are common starting points. Adjust them based on your own specific samples and use cases.