What Data Looks Like for This Category
The data for this category covers the entire production process of liquid milk, yogurt, milk powder, and dairy ingredients. Daily documents include raw milk acceptance records, standardized ingredient lists, pasteurization/UHT processing records, CIP cleaning records, filling records, and retained sample and microbiological test reports. Data sources include milk source testing systems, MES production management systems, homogenization and heat treatment equipment, online temperature and flow monitoring devices, CIP cleaning systems, and LIMS laboratory management systems. Updates are synced in real time per individual milk tank and production batch. Each individual document includes structured fields such as test values, process parameters, and operation times, plus original record text.
What Constraints These Characteristics Impose on Vector Models and Indexing Workflows
Real-time batch updates, multi-source heterogeneous attributes, and dense structured fields of this category’s data impose multiple constraints on the vector model and indexing workflow. Multi-source data requires unified embedding formats to ensure vector consistency. Batch-level updates require support for incremental indexing to avoid excessive time spent on full reprocessing. Each individual document contains a large number of continuous process parameters and short-text test records. When creating chunks, care must be taken to avoid splitting critical parameter groups, and adjustments must be made to accommodate length differences across data from different sources. High-risk audit scenarios prioritize recall accuracy. Invalid chunks must be restricted from being included, to prevent production risks caused by incorrect audits.
How to Set Configurations
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
| Chunk Length | 800–1200 characters | Adapts to the length of continuous process parameter groups in individual documents, avoids splitting critical batch parameters |
| Overlap Length | 100–150 characters | Preserves process context across chunks, prevents critical parameters from being truncated by chunk boundaries |
defaultToken | Calibrated via actual measurements | Adapts to token proportion across text from different sources, balances embedding effects for structured fields and free text |
maxToken | 4096 tokens | Matches the maximum input limit of mainstream embedding models, prevents long document chunks from exceeding capacity |
| Similarity Threshold | 0.75–0.85 | Filters low-match irrelevant records, meets audit accuracy requirements for high-risk parameters in dairy processing |
| Single Search Limit | Top 8 entries | Balances recall coverage and response speed, supports parallel processing needs for batch audits |
Three Common Misconfigurations
- Scenario: After Excel-format ingredient lists or raw milk acceptance records are uploaded, field values are missing or match incorrectly in index results after vectorization. Cause: Column-level parsing configuration for structured Excel files was not enabled, causing fields with mixed text and numerical values to be incorrectly merged into a single chunk.
- Scenario: After a single batch production record is uploaded, the number of chunks displayed in the interface does not match the actual number of text segments, with duplicate index entries appearing. Cause: The overlap length parameter was not adjusted, causing overlapping sections of adjacent process paragraphs to be recognized as independent chunks, or old version caches were not cleared during incremental indexing.
- Scenario: After upgrading from an older version to FastGPT v4.17.0, previously queryable CIP records can no longer be recalled. Cause: Token calculation rules for the new version’s embedding model were updated, and the original chunk length setting exceeded the input limit of the new model, causing the corresponding chunks to not be indexed correctly.
How to Verify Configurations Are Correct
- Upload a single typical document, such as a single-tank raw milk acceptance record, and check whether the paragraph splits in the chunk preview interface retain continuous process parameter groups.
- Submit a simulated audit query, such as "Verify if the time and temperature of this batch’s pasteurization meet requirements", and confirm whether the recalled result chunks contain complete process parameters.
- Run a bulk upload test, check whether the total number of chunks after indexing matches the expected value, with no duplicate entries.
- After adjusting configurations, query existing historical documents, confirm whether no abnormal missing of recall results occurs.
The values provided are common starting points and should be measured against the reader's own samples.