What this category of data looks like
Core data for pulp and paper scenarios comes from wood pulp and waste paper batches, DCS/QCS systems, on-machine measuring equipment, laboratory testing, energy and water treatment systems. Daily documents include pulp preparation sheets, beating records, paper machine operation logs, online quality curves, laboratory reports, sheet break and grade change records. Data update frequencies include real-time, per shift, per roll. Document fields include beating consistency, paper machine speed, grammage, moisture content, and other metrics, with dedicated associated units.
What constraints do these characteristics impose on document parsing and chunking
Real-time, per-shift, and per-roll update frequencies require that chunks not be overly large, to avoid recall lagging behind process adjustment needs. Single-line document formats such as pulp preparation sheets and beating records require accurate recognition of line breaks as chunk boundaries, to prevent multiple lines of data from merging into a single chunk. Structured table data in online quality curves and laboratory reports requires retaining the association between fields and units. Without this association, precise matching for process Q&A cannot be supported. Process records from high-speed continuous production have strong coherence. Chunks must not interrupt complete process steps such as beating or paper making.
How to configure the settings
The following configurations are based on FastGPT v4.17.0.
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Matches the length of a single process record, avoids splitting complete steps such as beating or paper making |
chunkSplitter | Split by line breaks + paragraphs | Adapts to the single-line format of pulp preparation sheets and beating records, resolves merged multiple-line data issues during Excel import |
customPdfParse | Enable table structured parsing | Accurately extract numerical values and associated units from online quality curves and laboratory reports |
PARSE_FILE_TIMEOUT_SECONDS | 120 seconds | Adapts to the parsing time required for large paper machine operation log files, avoids timeout interruptions |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Adapts to the maximum file size of single-roll paper machine operation logs |
chunkSettingMode | Manual configuration | Flexibly adjust chunking parameters for different document types such as records and reports |
Three common configuration mistakes
- Excel-imported pulp preparation sheets are merged into a single chunk: The phenomenon is that each row of records in the uploaded Excel pulp preparation sheet is automatically merged into one chunk. The cause is that
chunkSplitteris not configured to split by line breaks, and the default chunking logic does not recognize single-line data boundaries. - Identical matching content cannot be recalled: The phenomenon is that chunks in a specific knowledge base that exactly match the question are not retrieved. Creating a knowledge base with the same configuration restores normal function. The cause is that
customPdfParsewas not enabled during the first import, resulting in incomplete document parsing and missing key matching fields in chunks. - Multiple PDFs cannot be extracted individually after upload: The phenomenon is that batch-uploaded paper machine operation record PDFs cannot be split and extract information by individual roll documents. The cause is that
chunkSplitteris not configured to split by document metadata, and the default logic splits all content uniformly.
How to verify the configuration is correct
- Upload a single Excel file of a pulp preparation sheet, check if the chunking result shows each row as an independent record, and verify that the
chunkSplitterconfiguration is effective. - Upload a PDF report containing online quality curves, check if the parsed chunks retain complete numerical values and corresponding units for fields such as grammage and moisture content.
- Upload a single paper machine operation log file, confirm that the parsing time does not exceed 120 seconds as set by
PARSE_FILE_TIMEOUT_SECONDS, and no timeout errors occur. - Import chunk content that matches a process question, launch a recall test with the same question, and confirm that the chunks are correctly retrieved.
The values provided are common starting points and should be measured against applicable local samples.