What this category of data looks like
This category of data covers purchase orders, drawing specifications, supplier quality certificates, inspection equipment output data, historical quality records, and documents synced from supplier portals. Daily documents include inspection specifications, sampling plans, incoming material reports, supplier audit reports, exception contact sheets, 8D reports, and concession acceptance approval forms. Fields include batch number, material code, inspection item, judgment result, supplier number, rectification cycle, and more. Updates are triggered per incoming material batch, monthly or quarterly supplier evaluations, and irregular audits.
Constraints imposed by these characteristics on vector models and indexing
Data sources are scattered, and updates have no fixed cycle. Bulk indexing and incremental update support for multi-format documents is required. Documents contain both structured metadata and unstructured inspection descriptions. Semantic retrieval and metadata matching must both be supported. Update frequency fluctuates with incoming material batches. Index loads must adapt to dynamic data volumes. Data is linked to supplier and batch information. Indexes must retain associated metadata to enable precise recall and prevent cross-batch data confusion.
Configuration Settings
These configurations are based on FastGPT v4.17.0, and are adapted to the vector model and indexing needs of the incoming material inspection and supplier quality scenario:
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
| Chunk Length | 800–1200 characters | Ensures semantic integrity for unstructured documents such as inspection reports and 8D reports, prevents key inspection items from being split |
| Overlap Length | 100–200 characters | Retains critical metadata such as batch numbers and material codes across chunks, prevents loss of associated information |
| Similarity Threshold | 0.75–0.85 | Distinguishes semantic differences between qualified and unqualified judgments, aligns with strict judgment standards for incoming material inspection |
defaultToken | Set based on actual measurement | Adapts to token consumption requirements for different format documents such as supplier quality certificates and drawing specifications |
maxToken | 4096 | Matches the input limit of mainstream embedding models, ensures complete encoding of long documents |
| Single Search Limit | Top 10 results | Covers retrieval needs for multiple inspection items per incoming batch, balances recall accuracy and efficiency |
Three Common Misconfigurations
- Index tasks in the interface show a persistent running state with no progress updates. The cause is failure to adjust chunk parameters based on the document length of this category, resulting in failed chunk encoding for large-volume supplier audit reports.
- The chat function operates normally but vector indexing tasks have no response. The cause is incorrect setting of the
maxTokenparameter, exceeding the input limit of the embedding model and causing encoding failure. - Retrieval results include expired supplier audit reports. The cause is failure to enable incremental indexing mode; full indexing does not filter updated supplier evaluation data, resulting in recall of old data.
How to Verify Successful Configuration
- Upload a single typical incoming material report, check the chunk preview interface, and confirm that critical metadata such as batch numbers and material codes are fully retained in the chunk content.
- Run an incremental indexing task, check system logs to confirm only newly uploaded supplier evaluation data is processed, and no full indexing process is triggered.
- Initiate a retrieval test, input a query term containing a batch number or supplier number, and confirm that retrieval results only return valid documents from the corresponding dimension.
- Check the embedding model call logs, confirm that the token count for each call does not exceed the set
maxTokenvalue.
The values provided are common starting points and should be measured against the reader's own samples.