What the data for this category looks like
Core data for this category comes from multiple types of business documents from carbonated drink, juice, tea, energy drink, and packaged drinking water factories, including formula sheets, water treatment records, blending records, sterilization or aseptic filling records, packaging material inspection reports, and finished product inspection reports. Data is collected from water source and raw material testing systems, formula management systems, online sugar and acidity monitoring equipment, filling equipment, visual inspection devices, and LIMS systems. Updates are based on each production batch and each shift; some online equipment data is collected continuously. Document fields include raw material batch numbers, parameter values, test results, equipment operating status, and additional relevant fields.
What constraints do these characteristics impose on the vector model and indexing workflow
High-frequency continuously collected data and batch-updated documents for this category require vector indexes to support dynamic incremental updates and batch reindexing operations. Differences in fields across multiple types of heterogeneous documents require index configurations to adapt to chunking rules for different document structures, avoiding long text truncation or excessive short text splitting. Mixed storage of real-time online equipment data and static inspection reports requires indexes to support both low-latency real-time recall and precise long-text matching. Associated data for quotations and documents in bidding scenarios requires indexes to retain field-level precise matching capabilities, avoiding recall of irrelevant cross-category production data.
How to configure the setup
This configuration applies to FastGPT v4.17.0.
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
maxToken | 8192 token | Bulk bidding documents for this category typically include summaries of multiple production records, adapting to the maximum length limit for long-text embedding |
defaultToken | 1024 token | Average token count of a single basic business document for this category, adapting to default parameters for general embedding |
| Chunk Length | 800–1200 characters | Matches the average length of inspection reports and formula sheets for this category, improving embedding accuracy |
| Overlap Length | 100–150 characters | Retains temporal context association for continuously collected online equipment data |
| Similarity Threshold | 0.75–0.85 | Adapts to precise matching requirements in bidding scenarios, avoiding recall of irrelevant data |
| Single Search Limit | Top 10 entries | Controls context redundancy, focusing on core associated production batch data |
Three Common Configuration Mistakes
- Symptom: After switching to a multilingual embedding model, batch re-embedding tasks consistently show "Indexing" for more than 2 hours. Cause: Incremental indexing was not configured for the continuously collected data of this category, and excessive online equipment real-time data streams were loaded during full re-embedding.
- Symptom: The index corresponding to a manually uploaded single packaged water finished product inspection report disappears from the knowledge base after 3 hours. Cause: A persistent indexing strategy was not configured for batch-updated documents, and the temporary index was not bound to a production batch ID, leading to automatic system cleanup.
- Symptom: When searching for formula data associated with bidding quotations, a large number of blending records for juice products are recalled. Cause: The index was not configured with category-specific field filtering rules, and no restrictions were set to only recall production documents for the beverage and bottled water category.
How to Verify the Configuration is Correct
- Review batch re-embedding task logs to confirm that the token count of each document chunk does not exceed the configured
maxTokenvalue, and adjust chunk length as needed. - Manually insert a single water treatment record, wait for index completion, then run a targeted search to confirm that the document is recalled and no data from irrelevant categories is included.
- Access the knowledge base index management interface to confirm that all production batch documents are bound with persistent index tags, and no temporary index entries exist.
- Execute a multilingual search test to confirm that queries in different languages can recall matching production documents, verifying the recall effect of the new embedding model.
The values listed above are common starting points and should be measured against the reader's own samples.