What data for this category looks like
Data for this category comes from purchase orders, drawings and specification documents, supplier quality certificates, data collected by inspection equipment, historical quality records, and supplier portals. Daily documents include inspection specifications, sampling plans, incoming material reports, supplier audit reports, exception liaison sheets, 8D correction records, and concession acceptance approval forms. Data updates align with each batch of incoming materials, monthly or quarterly supplier evaluations, and irregular audits. Most documents include fields such as batch number, sampling quantity, non-conformity item level, and supplier code. Units include pieces, batches, ppm, and other standard measurement units.
What constraints these characteristics impose on the vector model and index workflow
Data sources are scattered across multiple systems. Indexes must support unified access to multi-source data and metadata association. Update frequencies vary widely. They range from high-frequency incremental updates for each batch of incoming materials to quarterly full batch synchronizations. This setup requires support for switching between incremental and full indexes. Document structures are diverse. They include plain text, tables, and embedded images. This requires support for chunking and vectorization of different formats. Key fields such as batch number and supplier code need to be used as retrieval filter conditions. Indexes must support metadata filtering functions.
Configuration Settings
This configuration is based on FastGPT v4.17.0 feature logic, as detailed below:
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
| Chunk Length | 800–1200 characters | Documents related to incoming material inspection mostly consist of short paragraphs paired with structured tables. This length preserves complete context for a single inspection specification or sampling plan |
| Overlap Length | 100–200 characters | Key information such as batch numbers and sampling rules often spans multiple paragraphs. Overlapping chunks prevent key information from being truncated |
| Similarity Threshold | 0.75–0.85 | Fine distinctions between qualified and unqualified judgment standards need to be differentiated. This range balances retrieval accuracy and recall rate |
| Single Search Limit | Top 10 results | Incoming material inspection decisions rely on only a small number of highly matched specifications and historical records. Excessive results will interfere with quick decision-making |
defaultToken | 2000 token | Adapts to the standard vectorization length of a single process specification or inspection report, avoiding exceeding model limits for a single request |
maxToken | Set to the selected embedding model/service maximum supported input tokens | maxToken is the input-token maximum supported by the selected embedding model/service; verify it in that provider configuration. See the v4.17.2 model-field reference. It is not retrieval/LLM context or a FastGPT-wide fixed cap. |
Three Common Misconfigurations
- Confusing the create training order function and the collection add data function. The symptom is redundant indexes caused by repeated execution of training order creation. The root cause is failing to clarify that training orders are used for incremental update scenarios such as batch synchronization of supplier quality certificates and historical quality data. Single-text additions are mistakenly routed through the training order process.
- Failure to associate retrieved content with vectorized images. The symptom is retrieval results only returning text fragments without associated information for corresponding images. The root cause is failing to bind image metadata to the corresponding inspection specification number during chunking, and failing to configure index association rules for image vectorization.
- Restricted model selection. The symptom is being unable to select a specified Embedding model. The root cause is failing to update the corresponding model mapping list in FastGPT configuration, or failing to correctly configure oneAPI model endpoint permissions during local deployment.
How to Verify Proper Configuration
- Upload a supplier audit report, check if chunked text fully retains key fields such as batch number and sampling quantity.
- Initiate a retrieval for incoming material inspection standards, confirm that the similarity of returned results falls within the preset threshold range.
- View training order execution logs, confirm that incremental updated historical quality data has been correctly synchronized to the index.
- Test image vectorization upload, confirm that retrieval results include associated metadata information for images.
The values provided are common starting points and should be measured against the reader's own samples.