What this category of data looks like
This category of data comes from purchase orders, drawings and specifications, supplier quality certificates, inspection equipment, historical quality data, and supplier portals. Daily documents include inspection specifications, sampling plans, incoming material reports, supplier audit reports, abnormal contact sheets, 8D reports, and concession acceptance approvals. Data updates are triggered for each batch of incoming materials, and synchronized alongside monthly or quarterly supplier evaluations and irregular audits. Data fields include incoming material batch numbers, supplier numbers, inspection item parameters, non-conforming item descriptions, rectification status, and basic information such as inspection value units and timestamps.
What constraints do these characteristics impose on the vector models and indexes stage
The high-frequency updates for each batch of incoming materials require indexes to support real-time incremental synchronization, avoiding additional time consumption caused by full reindexing. There are significant structural differences across multiple types of documents. Short texts such as inspection specifications and long texts such as 8D reports need to adapt to different chunking rules to prevent semantic fragmentation. Data must be bound to meta fields such as incoming material batch numbers and supplier numbers, and retrieval must conduct precise filtering by business dimension to ensure results are associated with the corresponding supplier and batch. Unstructured non-conforming item descriptions and rectification records require vector models to accurately capture semantic associations, while also being compatible with numerical data exported from equipment and manually filled text data, to ensure unified indexing of multi-source heterogeneous data.
How to set the configuration
This configuration applies to FastGPT v4.17.0.
| Configuration Item | Recommended Setting | Rationale for This Setting |
|---|---|---|
| Chunk Length | 800–1200 characters | Adapts to the common text length of business documents such as incoming material reports and 8D reports, avoiding semantic fragmentation within a single chunk |
| Overlap Length | 100–150 characters | Retains contextual associations across chunks, ensuring semantic integrity of non-conforming item descriptions and rectification records |
maxToken | 4096 token | Adapts to the input length limit of mainstream open-source vector models, matching the average token proportion after chunking business documents |
| Similarity Threshold | 0.72–0.85 | Distinguishes semantic differences between qualified and unqualified inspection records, adapting to the accuracy requirements of supplier quality retrieval |
| Single Search Limit | Top 10 entries | Matches the result volume of daily retrieval for SQE and IQC personnel, reducing interference from invalid information |
defaultToken | 1024 token | Serves as the baseline parameter for default chunking, adapting to the initial chunking configuration of most business documents |
Three Common Mistakes
- Retrieval result count does not meet expectations, only returning a small number of irrelevant results or including irrelevant documents across suppliers. The cause is failure to filter by binding meta fields such as incoming material batch numbers and supplier numbers by business dimension, leading to retrieval scope exceeding business requirements.
- Custom vector model access fails, for example, multimodal vector models cannot initiate calls normally. The cause is failure to correctly configure authorization parameters and interface formats for model access, and failure to adapt to the input requirements of multimodal vectors.
- Token limit error occurs during vector model calls. The cause is failure to correctly configure the
maxTokenparameter, causing chunked text to exceed the model's input limit.
How to Confirm the Configuration is Complete
- Upload a single incoming material inspection report, check whether the chunked text retains core semantics completely without obvious breaks.
- Enter retrieval terms related to supplier audits, verify whether retrieval results only associate historical documents of the corresponding supplier, with no irrelevant results across suppliers.
- Connect to the specified vector model, initiate a test call, and confirm that the vector data format returned by the model meets system requirements.
- Adjust the similarity threshold parameter, verify whether the matching accuracy of retrieval results meets business expectations.
The values given are common starting points and should be measured against the reader's own samples.