What data for this category looks like
This category of data primarily comes from formulation systems, raw material batch ledgers, internal mixers, Mooney viscometers, vulcanizing machines, and laboratory testing equipment. Daily documents include formulation sheets, weighing records, mixing curves, rubber quick-test reports, vulcanization process cards, mold usage records, and finished product performance reports. Data updates are generated synchronously with each batch of raw material delivery, completion of each batch of mixed rubber production, and end of each vulcanization operation. Single document fields include temperature, time, viscosity, batch number, and other parameters, with attached units such as ℃, min, MU.
What constraints these characteristics impose on the "document parsing and chunking" workflow
The high-frequency, small-batch updates and multi-dimensional process parameter characteristics of this category require document parsing to accurately match the complete parameter chain of a single production batch. Mixing curves and vulcanization process cards contain continuous time-series data, so chunking must avoid splitting critical process nodes. The uniformity of field units across different documents varies, so original unit information must be retained to avoid ambiguity. When uploading multiple production records in bulk, ensure that chunking does not mix data across batches, while adapting to the time-series length of single documents, preventing chunk sizes that are too large to allow association of complete process parameters during retrieval, or too small to lose contextual connections.
How to set up configurations
This page is written based on the FastGPT v4.17.0 version configuration logic.
| Configuration Item | Recommended Value | Rationale for This Setting |
|---|---|---|
chunkSize | 800–1200 characters | Adapts to the length of single-segment process parameters for mixing curves and vulcanization process cards, avoiding splitting continuous time-series data |
chunkSplitter | Split by document structure | Matches the fixed chapter structure of formulation sheets and process cards, retaining contextual connections between fields and units |
customPdfParse | Enabled | Retains tables and time-series curve text from original documents, avoiding loss of parameter information during format parsing |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Adapts to the need for packaged uploads of multiple documents covering the full process of a single batch mixing |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Prevents parsing timeout for large-volume mixing curve documents |
chunkSettingMode | Manual configuration | Adjust chunking parameters for different document types (such as formulation sheets, process cards) to adapt to the category-specific document structure |
Three common mistakes
- Phenomenon: After uploading a mixing curve PDF, knowledge base retrieval results only contain scattered temperature values, with no complete mixing time-series process chain. Cause:
customPdfParseis not enabled, causing the text of the time-series curve to be incorrectly split and losing contextual connections. - Phenomenon: A newly created knowledge base can retrieve target content, but an existing knowledge base with the same configuration cannot. Cause: The
chunkSizeof the existing knowledge base does not adapt to the current category's document length, or the chunking mode is not switched to split by document structure, resulting in a mismatch between the chunking logic of previously uploaded documents. - Phenomenon: After database connection configuration is completed, the generated SQL cannot be executed, and the reply contains punctuation marks that cause parsing failure. Cause: Parameters are not configured via
chunkSettingModeto filter invalid punctuation in unstructured text, resulting in interfering characters being mixed in during SQL generation.
How to confirm the configuration is correct
- Upload a single vulcanization process card PDF, check if the parsed text retains complete temperature, time, batch number fields and their corresponding units.
- Upload 3 consecutive batch mixing record documents, confirm that the chunking results do not splice data across batches, and each chunk of content corresponds to single-batch production parameters.
- Test uploading a single 800 MB mixing curve document, confirm that the parsing process does not trigger a timeout error.
- Manually adjust
chunkSizeto 1000 characters, verify that the chunked text length meets expectations and does not split critical process nodes.
The values provided are common starting points and should be measured against the reader's own samples.