What data for this category looks like
Data for this category comes primarily from enterprise ERP systems, shop floor production terminals, and supplier collaboration documents. It includes sales orders, master production schedules, material requirements plans, weekly and daily scheduling sheets, work orders, material shortage lists, delivery change notices, and similar items. Update schedules are irregular. They include daily capacity and inventory updates, weekly schedule adjustments, and temporary updates for added orders, material shortages, or equipment failures. Documents have structured fields such as order numbers, material codes, capacity man-hours, and shift headcount, plus unstructured descriptions like work order exception notes and delivery change reasons. Units use standard production terms: pieces, hours, days, and others.
Constraints on vector models and indexes
Data sources are scattered and formats are mixed, with structured numerical and unstructured text coexisting. This requires vector models to adapt to encoding logic for both types of content. Update frequencies fluctuate widely, with sudden temporary updates, so indexes must support incremental updates and fast synchronization. Data volume grows with order volume and process complexity. Single-batch retrieval needs to cover associated data across multiple processes and materials, which places requirements on index recall efficiency and accuracy. Additionally, documents contain a large number of production-specific terms and numbers. Vector indexes must retain precise semantic associations to avoid retrieval results straying from the core needs of supplier quality audits and schedule adjustments.
How to set configurations
| Configuration Item | Suggested Range | Rationale |
|---|---|---|
| Chunk Length | 800–1200 characters | Matches the average content length of work orders and scheduling sheets for this category, avoids semantic fragmentation within chunks |
| Overlap Length | 100–150 characters | Connects semantics of adjacent chunks, prevents truncation of scheduling process and material association information |
| Similarity Threshold | 0.75–0.85 | Filters low-correlation supplier delivery records and redundant scheduling entries, adapts to retrieval accuracy for quality audits |
maxToken | Set to the selected embedding model/service input limit | This is the configured embedding model's maximum token count, not retrieval or LLM context. Check the selected model or provider documentation. See the v4.17.2 model-field reference |
defaultToken | Set from representative sample documents | defaultToken is the default token count for direct text chunking; tune it on representative source documents. See the v4.17.2 model-field reference. It is not a vector dimension, embedding-model input cap, or retrieval/LLM context. |
| Single Search Limit | Top 20 entries | Balances retrieval efficiency and recall coverage, meets associated data needs for multi-process scheduling |
Three Common Configuration Mistakes
- Phenomenon: Qwen3-Embedding-8B requests through vLLM time out or return format errors. Check the vLLM endpoint, request compatibility, model logs, and input length.
defaultTokencontrols text splitting; it is not an embedding dimension. - Phenomenon: A large number of irrelevant inventory records are returned when retrieving content related to supplier quality audits. Cause: The similarity threshold is set too low, failing to filter non-core material inventory data.
- Phenomenon: Duplicate original files and embedding vector entries are found when checking knowledge base disk usage. Cause: Incremental index updates based on update frequency are not configured, leading to repeated generation of vector data for the same content.
How to confirm configurations are correctly set
- In FastGPT v4.17.2, confirm that
maxTokenfollows the selected embedding model/service input limit and thatdefaultTokenproduces the intended chunks on representative work orders. - Manually upload a test work order document, verify that the chunk length and overlap length of the segmentation results fall within the preset ranges of 800–1200 characters and 100–150 characters.
- Perform a retrieval related to supplier quality audits, confirm that the number of returned results matches the single search limit of the top 20 entries.
- View the system disk usage statistics page, confirm that the statistics scope includes three categories of data: original files, split file chunks, and embedding vectors.
The values provided are common starting points and should be measured against the reader's own samples.