What this category of data looks like
The core data for this category comes from ERP orders and inventory, BOMs and process routes, equipment capacity, staff shifts, work-in-progress, and supplier deliveries. Daily processed documents include sales orders, master production schedules, material requirements plans, weekly and daily production schedules, work orders, material shortage lists, and delivery change notices. Data fields cover order numbers, material codes, capacity values, shift time periods, process requirements, and more. Units include pieces, hours, days, and others. Data update frequencies include daily, weekly, and real-time updates triggered by orders or abnormal events.
What constraints do these characteristics place on vector models and indexing
Since data sources are scattered and update rhythms vary widely, indexes must support incremental updates to meet the recall needs of real-time exception reports. Daily documents include tabular production schedules and textual work orders, so chunking rules must adapt to different text structure formats. There are many fields with specific business units, so vector models must accurately identify semantic associations between structured fields. Sudden exception reports require fast recall of the latest data, so index recall efficiency must be optimized for high-frequency query scenarios.
How to set configurations
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
| Chunk Length | 800–1200 characters | Adapts to the average character length of daily documents such as work orders and production schedules, avoiding overly fragmented chunks or lost context |
| Overlap Length | 100–150 characters | Retains semantic connections between adjacent chunks, preventing breaks in cross-chunk process and scheduling information |
| Similarity Threshold | 0.75–0.85 | Filters low-relevance inventory and equipment data, focusing on results related to scheduling and work orders |
| Single Search Limit | Top 20 results | Controls the volume of recalled data, reducing time pressure from mixed retrieval |
maxToken | 8192 token | Meets the vectorization needs of long documents such as complete master production schedules, avoiding truncation of critical information |
defaultToken | Calibrated based on actual testing | Adapts default vectorization parameters for daily short documents, balancing accuracy and efficiency |
Three common mistakes
- Symptom: Consistent vector scores after Docker deployment. Cause: Incorrect chunking configuration, causing all uploaded documents to be uniformly truncated before vectorization, losing differentiated information from production schedules and work orders.
- Symptom: Mixed retrieval takes longer than 10 seconds. Cause: No reasonable single search limit set, recalling too much irrelevant data which increases time spent in subsequent rearrangement steps.
- Symptom: Index construction stops at the final batch. Cause: The
maxTokenparameter was not adjusted, so ultra-long master production schedule documents cannot be vectorized, causing the construction process to stall.
How to confirm the configuration is correct
- Upload a typical work order document, check that the chunked fragments cover key fields such as work order number, process requirements, and delivery time.
- Initiate a search containing keywords from production schedules, confirm that the similarity scores of returned results have reasonable differences, with no cases of all scores being identical.
- Add a new delivery change notice, trigger an incremental index update, and verify that the new document can be retrieved within a reasonable time frame.
- Check the vector index logs for FastGPT v4.17.0, confirm there are no error messages related to
maxTokenexceeding limits or chunking exceptions.
The values provided here are common starting points and should be measured against the reader's own samples.