What data looks like for this category
The core documents for this category include raw milk acceptance records, standardized ingredient lists, pasteurization or UHT heat treatment records, CIP cleaning records, filling records, and reserved sample and microbial test reports. Data is sourced from milk source detection systems, MES manufacturing execution systems, homogenization and heat treatment equipment, online temperature and flow sensors, CIP systems, and LIMS laboratory information management systems. Data syncs in real time per milk tank and production batch. Each individual document includes structured fields such as raw milk physicochemical indicators, heat treatment parameters, cleaning verification results, and original test data.
What constraints these characteristics impose on vector models and indexes
Real-time high-frequency updates require indexes to support incremental synchronization, to avoid performance losses from full reindexing. Multi-source heterogeneous data from production equipment, detection systems, and business systems needs unified field mapping rules to ensure consistent vector generation. Each individual document contains strongly correlated parameters such as heat treatment time, cleaning temperature, and microbial limits. Vector chunking must retain field correlation, to avoid losing critical business logic after splitting. Strict compliance requirements for raw milk and finished products mean index retrieval results must be traceable to original detection and production records, to prevent irrelevant data from interfering with judgments.
How to set configurations
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
| Chunk Length | 800–1200 characters | Matches the typical paragraph length of a single production record, retains complete context for key processes such as heat treatment and cleaning |
| Overlap Length | 150–200 characters | Ensures cross-chunk process parameters are not split or broken, avoids losing continuous process information during retrieval |
| Similarity Threshold | 0.75–0.85 | Adapts to dairy processing compliance judgment needs, filters low-relevance non-process documents |
| Single Search Limit | Top 8 results | Covers the number of associated records for a single production batch, ensures sufficient verification evidence is retrieved |
defaultToken | Set from representative sample documents | defaultToken is the default token count for direct text chunking; tune it on representative source documents. See the v4.17.2 model-field reference. It is not a vector dimension, embedding-model input cap, or retrieval/LLM context. |
maxToken | 8192 | Supports integration of complete process specifications and batch records, meets context requirements for complex Q&A |
Three common mistakes
- Symptom: After deploying FastGPT v4.17.0 locally, no image indexing model option appears in the knowledge base upload step. Cause: The open-source version does not include the image indexing extension module by default. Manually deploy the corresponding plugin to enable the feature.
- Symptom: After uploading raw milk acceptance records, searching for "raw milk antibiotic indicators" returns no matching results. Cause: No field mapping rules matching raw milk physicochemical indicators have been configured, leading to loss of critical retrieval fields during vector generation.
- Symptom: Searching for "pasteurization process" returns a large number of irrelevant filling records. Cause: Chunk length is set too short, splitting continuous heat treatment processes into multiple independent chunks and reducing correlation during similarity matching.
How to confirm configurations are set correctly
- Upload a standard raw milk acceptance record, search for "raw milk antibiotic indicators", and verify that returned results include corresponding test values and acceptance conclusions.
- Adjust the similarity threshold to 0.8, search for "pasteurization temperature", and confirm returned results only include relevant heat treatment process records.
- Upload multiple consecutive batch production records, trigger incremental indexing, and check that the indexing progress log has no error messages.
- Input a test question containing 3 associated production records, confirm that the complete context loads without truncation.
The values provided are common starting points and should be measured against the reader's own samples.