What data looks like for this category
Data for this category primarily comes from ERP/WMS systems, barcode or RFID equipment, purchase work orders, storage location management records, AGV transport systems, and weighing equipment. Daily generated documents include receiving slips, putaway slips, material issue and return slips, inventory transfer slips, inventory count sheets, lot ledgers, first-in-first-out and expiration date records. Data update frequencies fall into three categories: real-time (such as AGV real-time location, weighing data), daily (such as material issue and return summaries), and monthly inventory counts. Document fields include material code, lot number, storage location number, expiration date, material quantity, inspection status, and others. Units include pieces, kilograms, days, and others.
What constraints these characteristics impose on the vector models and indexes link
Data sources for this category are scattered, and update frequencies vary widely. Real-time generated data requires indexes to support incremental updates and low-latency synchronization, to prevent discrepancies between ledger and physical data. The accuracy of core business fields such as lot number, expiration date, and material code directly affects ledger reconciliation results. This requires vector models to accurately capture semantic associations between structured fields, and avoid matching failures caused by field truncation. Mixed import of multiple document types requires indexes to adapt to different document structures. At the same time, business requirements for ledger-physical consistency and lot traceability require retrieval recall precision and completeness, and key associated records must not be missed.
How to set configurations
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
| Chunk Length | 800–1200 characters | Match the length of single records from documents such as receiving slips and lot ledgers, avoid splitting core business fields |
| Overlap Length | 50–80 characters | Retain key information such as lot number and expiration date across chunks, prevent field truncation that causes matching failures |
| Similarity Threshold | 0.75–0.85 | Ensure retrieval matching precision for core fields such as material identity and lot number, adapt to the high requirements of ledger reconciliation |
| Single Search Limit | Top 10 entries | Cover associated retrieval requirements for multiple storage locations and lots, avoid missing associated ledger records |
defaultToken | Set from representative sample documents | defaultToken is the default token count for direct text chunking; tune it on representative source documents. See the v4.17.2 model-field reference. It is not a vector dimension, embedding-model input cap, or retrieval/LLM context. |
maxToken | Set to the selected embedding model/service maximum supported input tokens | maxToken is the input-token maximum supported by the selected embedding model/service; verify it in that provider configuration. See the v4.17.2 model-field reference. It is not retrieval/LLM context or a FastGPT-wide fixed cap. |
Three common mistakes
- Phenomenon: Retrieval response times out, single retrieval takes more than 30 seconds. Cause: Chunk length is set too large, leading to excessive vector data volume per batch, sharply increasing index loading and retrieval calculation pressure.
- Phenomenon: Lot number and expiration date fields are empty or matched incorrectly in retrieval results. Cause: Insufficient overlap length is reserved during chunking, leading to core business fields being split and truncated, preventing vectors from accurately associating corresponding information.
- Phenomenon: Number of retrieval results is far lower than expected, unable to cover associated inventory transfer and inventory count records. Cause: Similarity threshold is set too high, filtering out weakly matched but actually associated ledger data.
How to confirm configurations are set correctly
Use the FastGPT v4.17.0 platform to perform the following verification actions:
- Upload a single standard receiving slip document, check the chunk preview interface, confirm that core fields such as material code and lot number are fully retained, with no truncation.
- Simulate a retrieval for "lot ledgers for line-side warehouse in Q2 2024", check retrieval response time, confirm it meets business expectations with no obvious timeout.
- Run the same retrieval once before and after adjusting the similarity threshold, compare changes in the number of results, confirm the threshold setting matches business matching precision requirements.
- Import bulk warehousing documents, check the index update status panel, confirm that real-time data synchronization delay meets business requirements.
The values provided are common starting points and should be measured against the reader's own samples.