What this category's data looks like
The core data for this category comes from water source and raw material testing, formula systems, online sugar and acidity monitoring, filling equipment, visual inspection systems, and LIMS systems. Daily documents include formula sheets, water treatment records, blending records, sterilization or aseptic filling records, packaging material inspection records, and finished product inspection records. Data is updated per batch, per shift, or via continuous collection. Individual records include fields such as raw material indicators, process parameters, test results, and batch numbers, covering process and standard-related information across the full workflow from raw materials to finished products.
What constraints do these characteristics impose on the vector models and indexes link
High-frequency, continuously updated process and test data requires indexes to support incremental synchronization mechanisms, avoiding performance loss and delays caused by full reconstruction. The mixed multi-type document structure, including formula text, numeric process parameters, and test results, requires vector models to adapt to semantic matching for both process description text and parameter values. The feature that single-batch data is associated with multiple link records requires indexes to support associative retrieval by batch dimension, avoiding interference from irrelevant cross-batch results. Additionally, the rigor required for process and standard Q&A means retrieval results must match strict process thresholds, preventing low-confidence results from impacting production decisions.
How to set the configurations
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
| Chunk Length | 800–1200 characters | Individual process records for this category contain multiple sets of parameters and text descriptions. This length can fully cover single-link process information and avoid semantic fragmentation |
| Overlap Length | 150–200 characters | Process record parameters are linked to preceding and subsequent steps. The overlap length preserves semantic associations across chunks and prevents key parameters from being split |
| Similarity Threshold | 0.75–0.85 | Process parameters for this category have strict requirements. The threshold filters low-match irrelevant records while retaining candidate results that meet process standards |
| Single Search Limit | Top 10 results | Single-batch process records are associated with multiple links. The top 10 results cover the main process and standard-related content and avoid excessive redundant results |
maxToken | Set to the selected embedding model/service maximum supported input tokens | maxToken is the input-token maximum supported by the selected embedding model/service; verify it in that provider configuration. See the v4.17.2 model-field reference. It is not retrieval/LLM context or a FastGPT-wide fixed cap. |
| Reranking | Enabled | Process parameters for this category mix numeric and text content. Reranking can improve the ranking priority of high-match results and optimize retrieval accuracy |
Three common configuration errors
- Phenomenon: When configuring a third-party embedding model, the interface prompts that no available channels are found or the connection fails. Cause: No dedicated call group was configured for the embedding model in FastGPT v4.17.0's group management, and the default group does not have channel permissions for the corresponding model enabled.
- Phenomenon: After enabling the reranking function, the response time for process Q&A is too long. Cause: The chunk length setting exceeds the reasonable range for single-link process records of this category, causing single-batch data to be split into too many chunks and increasing the computational load during the reranking stage.
- Phenomenon: A large number of irrelevant general test records are mixed into the retrieved results, and target process specifications are not matched. Cause: The selected embedding model does not adapt to the mixed structure of process parameters and standard text for this category, and cannot accurately match the semantic requirements related to processes.
How to confirm the configuration is correct
- Upload a single batch of process record documents, and check whether the chunk results fully cover the single-link process information, with no key parameters split.
- Initiate a process parameter query, verify whether the returned results are sorted according to process association priority, and adjust the similarity threshold to adapt to retrieval strictness as needed.
- Test incremental update of a new shift record, and confirm that the index can automatically synchronize new data without manually triggering full reconstruction.
- Check the embedding model call logs to confirm there are no connection failures or current limit errors, and the model call status is normal.
The values provided are common starting points and should be measured against the reader's own samples.