What Data for This Category Looks Like
This category’s data is sourced from registered and filed formulas, raw material safety documents, production ERP/MES system data, laboratory test records, and packaging material version files. Daily documents include formula process sheets, weighing and feeding records, emulsification and filling records, cleaning and site clearance records, semi-finished and finished product inspection sheets, and label review forms. Data updates with each production batch or line change. Document fields include raw material identifiers, process parameters, batch numbers, and test results. Some fields use dedicated units: emulsification temperature is measured in degrees Celsius, and feeding quantity is measured in kilograms.
Constraints Imposed on Document Parsing and Chunking
High-frequency data updates triggered by each production batch or line change require the document parsing and chunking process to support batch parallel processing, to avoid delaying production-related review workflows. Scenarios with mixed document types require chunking strategies to adapt to different document structures, rather than relying solely on fixed paragraph splitting rules. The binding relationship between raw materials, process parameters, and batches in documents requires preserving field associations during chunking, to avoid destroying data integrity after splitting. Under compliance requirements, chunk results must retain the original document’s source identifier and batch information, to enable traceability of filed associated content during cross-review. The presence of long paragraphs of process descriptions requires chunking logic to balance content integrity and retrieval granularity.
How to Configure Settings
These configurations are based on FastGPT v4.17.0.
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Adapts to the content length of raw material groups and process steps in cosmetics manufacturing documents, preserves the integrity of associated information for single groups, and avoids breaking data binding relationships |
chunkSplitter | Intelligent splitting based on document structure | Adapts to the structural characteristics of different documents such as formula process sheets and test records, avoids destroying field associations with fixed splitting rules, and generates structured chunks for front-end parsing |
customPdfParse | Enabled | Extracts structured fields from PDF embedded forms and scanned converted text, avoids losing key information such as raw materials and test results |
PARSE_FILE_TIMEOUT_SECONDS | 300–600 seconds | Adapts to the parsing time of batch documents, avoids timeout interrupts when parsing large-volume filed materials or multi-batch records |
chunkSettingMode | Custom mode | Supports adjusting chunking rules for different document types, including parent-child chunking modes: for example, splitting process sheets by steps and inspection records by inspection items, to meet the retrieval needs of cross-review |
Three Common Misconfigurations
- Empty fields or missing parameters appear after parsing. The cause is that
customPdfParseis not enabled, so structured fields in PDF embedded forms cannot be extracted, leading to loss of key information such as raw materials and test results during chunking. - Timeout errors (status code 504) occur when uploading multiple documents in batches. The cause is that
PARSE_FILE_TIMEOUT_SECONDSis set too low, which does not adapt to the parsing time of batch documents, resulting in interruption of some large-volume documents before parsing is complete. - Process steps are split apart after chunking. The cause is that
chunkSizeis set too small, splitting complete emulsification process steps into multiple independent chunks, destroying the continuity of process logic and affecting step traceability during cross-review.
How to Verify Correct Configuration
- Upload a single typical document (such as a formula process sheet), check the parsed chunk results, and confirm that raw materials and process parameters are not split into different chunks.
- Upload multiple documents of different types in batches, check whether all parsing tasks are completed without timeout or error prompts.
- Extract the chunked text, confirm that each chunk retains the document source identifier and batch number information.
- For documents with long paragraphs of process descriptions, verify that the chunk results do not split core process steps.
The values provided here are common starting points, and should be evaluated against the user’s own samples.