What this category of data looks like
The main entities in basic chemical continuous production include enterprises engaged in chlor-alkali, synthetic ammonia, methanol, soda ash, and similar production. Daily documents include operating procedures, process index cards, DCS trends, laboratory reports, interlock bypass records, special work permits, and shift logs. Data sources include DCS/SIS systems, online analyzers, LIMS systems, equipment condition monitoring, energy metering devices, and raw material storage tanks. Data update frequencies include real-time, per shift, and daily. Documents contain process parameter values and corresponding engineering units, with mixed structures including a large number of structured tables and continuous parameter text.
What constraints do these characteristics impose on the document parsing and chunking link
Multi-source, multi-format documents in this category impose clear constraints on chunking rules. Real-time updated DCS trend and shift log data must avoid splitting parameter associations at critical time nodes. Documents containing process interlock logic must retain complete interlock numbers and disposal procedures to prevent logical breaks. Structured table documents such as laboratory reports and process index cards must accurately identify row and column structures, and must not split parameters and units within cells. Large-volume historical data files require appropriate chunking thresholds to avoid excessive splitting that leads to lost context.
How to set the configurations
These configurations are adapted for FastGPT v4.17.0.
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Adapts to the parameter density of basic chemical documents, avoids splitting process interlock logic or table rows |
chunkSplitter | Split by semantic and format boundaries | Retains contextual integrity of process parameters and interlock records, adapts to documents with mixed formats |
customPdfParse | Enabled | Accurately parses tables and numeric fields in DCS trend PDFs and laboratory report PDFs |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Adapts to large-volume files such as batch-uploaded shift logs and DCS historical trends |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Addresses parsing time for large-volume process documents, prevents task termination due to mid-task timeout |
chunkSettingMode | Manual configuration | Adjusts chunking rules for different document types such as interlock records and shift logs |
Three common mistakes
- Phenomenon: After uploading DCS trend or shift log files larger than 10 MB, over 1000 chunks are generated, and some chunks fail vectorization. Cause: The
chunkSizeandUPLOAD_FILE_MAX_SIZEparameters are not adjusted, large files are excessively split, exceeding the processing limit of the vectorization service. - Phenomenon: When importing laboratory report data from Feishu online spreadsheets, table fields and corresponding values cannot be correctly associated. Cause:
customPdfParseis not configured to adapt to structured parsing rules for online spreadsheets, leading to chaotic field splitting. - Phenomenon: The dataset.csv exported from the knowledge base only uses question-and-answer template formatting, and the original document structure cannot be directly reused. Cause: Format metadata of original documents is not retained in chunking configurations, leading to loss of table and paragraph structure information during export.
How to confirm the configurations are correct
- Upload a single 100 MB process operating procedure PDF, check whether the number of generated chunks falls within the expected range, and verify that no empty chunks or formatting errors exist.
- After enabling
customPdfParse, upload a laboratory report PDF containing multiple tables, confirm that the parsed chunks completely retain the row and column structure of the tables and numerical units. - Upload an interlock bypass record document, check whether adjacent chunks contain complete interlock numbers and disposal procedures, with no logical splitting breaks.
- Upload a single large-volume DCS trend file, confirm that the parsing task does not have timeout errors, and that the generated chunks have no content truncation.
The values provided are common starting points and should be measured against the reader's own samples.