What Data Looks Like for This Category
This category's data comes from milk source testing, MES systems, homogenization and heat treatment equipment, online temperature and flow collection, CIP cleaning systems, and LIMS laboratory management systems. Daily documentation includes raw milk acceptance records, standardized ingredient lists, pasteurization or UHT heat treatment records, CIP cleaning records, filling operation records, sample retention and microbial test reports. Data is generated per milk tank, per batch, or in real time. Each individual document contains fields such as batch number, test values, equipment parameters, operation time, and some values include units like temperature, pressure, or concentration.
What Constraints These Characteristics Impose on Vector Models and Indexes
Real-time or high-frequency updated batch data requires indexes to support incremental synchronization, avoiding delays caused by full index reconstruction. Multi-source heterogeneous data formats require unified vector conversion logic to ensure that temperature and flow data and test data collected by different devices can be correctly encoded. Documents with multiple fields and associated units require retaining batch identifiers and key parameters during chunking to avoid losing critical context during retrieval. Batch-generated exception reports and process documents require indexes to support quick filtering by batch and time range, narrowing the retrieval scope. Process data for perishable raw materials has strong timeliness, so index update delay must be limited to ensure retrieval results match current production status.
How to Configure Settings
This configuration is based on FastGPT v4.17.0.
| Configuration Item | Recommended Value Range | Rationale |
|---|---|---|
| Chunk Length | 800–1200 characters | Matches the typical paragraph length of a single process document, retaining complete context of operating steps or test results |
| Overlap Length | 100–200 characters | Avoids losing cross-chunk key information such as batch numbers and equipment parameters after chunking |
maxToken | Set to the selected embedding model/service maximum supported input tokens | maxToken is the input-token maximum supported by the selected embedding model/service; verify it in that provider configuration. See the v4.17.2 model-field reference. It is not retrieval/LLM context or a FastGPT-wide fixed cap. |
| Similarity Threshold | 0.75–0.85, calibrated via actual testing | Filters low-match non-production-related documents, focusing on content from the same batch or same process link |
defaultToken | Set from representative sample documents | defaultToken is the default token count for direct text chunking; tune it on representative source documents. See the v4.17.2 model-field reference. It is not a vector dimension, embedding-model input cap, or retrieval/LLM context. |
| Single Search Limit | Top 10 entries | Controls the number of initially recalled documents, avoiding excessive redundant data occupying system resources |
Three Common Misconfigurations
- Phenomenon: Retrieval results return documents from non-current production batches, or lack unit information for test values. Cause: Batch identifiers and unit fields were not retained during chunking, or the index was not configured with filtering by batch tags.
- Phenomenon: Logs show matching documents were recalled, but the LLM reports no relevant content. Inspect the query, retrieval filters and results, prompt, conversation history, and LLM context budget. The embedding model's
maxTokenlimits an individual embedding input; it does not set total retrieval or LLM context. - Phenomenon: Retrieval requests time out or take longer than expected. Check database and embedding-service logs, query filters, result volume, and provider rate limits before changing chunk settings; do not assume
defaultTokenis the cause.
How to Verify Proper Configuration
- Upload a single raw milk acceptance record, retrieve the corresponding batch number, and check whether the recalled results include all relevant documents for that batch.
- Adjust the similarity threshold and single search limit, observe the matching degree and quantity of retrieval results, and calibrate appropriate values based on actual business needs.
- Submit a retrieval request containing multi-source equipment parameters, and check whether the large language model's returned content includes specific values and operating steps from the retrieved documents.
- View the system's incremental synchronization logs, confirm that the index update rhythm matches the production batch update frequency, and there are no delay errors from full reconstruction.
The values listed above are common starting points and should be measured against the reader's own samples.