What the Data for This Category Looks Like
The core data for this category originates from manufacturing enterprises’ MES systems, online testing equipment, laboratory testing systems, process parameter documents, customer quality standards, and historical defect databases. Daily processed documents include control plans, patrol inspection records, SPC reports, final product inspection reports, nonconforming product review forms, release forms, and quality monthly reports. Data updates align with each production batch, shift patrol inspections, and continuously collected process parameters. Each document includes fields such as batch number, inspection items, deviation values, and review conclusions. Some structured data includes physical units.
Constraints for Vector Models and Indexing
Multi-source heterogeneous data sources require vector models to support both structured inspection parameters and unstructured review text. Frequently updated batch data requires indexes to support incremental refreshes, avoiding excessive computational resource consumption. Document lengths vary widely: single segments of control plans can reach thousands of characters, while individual patrol inspection records are only tens of characters. This demands flexible chunking rules to preserve context integrity. Associated judgments for final product release also require precise matching of multiple documents from the same batch, placing high requirements on retrieval recall accuracy and relevance ranking.
Configuration Settings
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
| Chunk Length | 800–1200 characters | Adapts to the common content density of documents in this category, covering single-segment information from control plans and SPC reports |
| Overlap Length | 100–150 characters | Prevents chunking from breaking the complete logic of process parameters and inspection items, preserving context connections |
| Similarity Threshold | 0.75–0.85 | Enables precise matching of release standards and deviation items in inspection records, reducing false recall rates for nonconforming product-related documents |
| Single Search Limit | Top 10 results | Associated documents for this category are mostly inspection and review materials from the same batch, so excessive redundant results are unnecessary |
maxToken | Set to the selected embedding model/service maximum supported input tokens | maxToken is the input-token maximum supported by the selected embedding model/service; verify it in that provider configuration. See the v4.17.2 model-field reference. It is not retrieval/LLM context or a FastGPT-wide fixed cap. |
defaultToken | Set from representative sample documents | defaultToken is the default token count for direct text chunking; tune it on representative source documents. See the v4.17.2 model-field reference. It is not a vector dimension, embedding-model input cap, or retrieval/LLM context. |
Three Common Configuration Mistakes
- The interface shows files remaining in an indexing state. The symptom is that the indexing status does not update for an extended period after upload. The cause is that incremental index trigger rules are not configured, and full indexing is not adapted to frequently updated batch data.
- Vector retrieval results differ from local test results. The symptom is that documents recalled by the server do not match results after local chunking. The cause is that different vector models are used locally and on the server, and embedding dimensions and chunking rules are not aligned.
- Multiple vectors cannot be associated with a single batch. The symptom is that vectors generated from multiple documents of the same batch cannot be retrieved uniformly. The cause is that metadata indexing is not configured for batch fields, and vectors are only generated independently for each individual document.
How to Verify Proper Configuration
- Upload a control plan document. Verify that the indexing completion status changes within a reasonable timeframe, confirming that chunking rules and index refresh configurations take effect.
- Input a query containing inspection deviations. Confirm that recalled documents include inspection records and review materials from the same batch, verifying that the similarity threshold and search limit configurations are correct.
- Switch the vector model, then upload the same document. Compare generated vector hash values to confirm that embedding dimensions match the
defaultTokenparameter. - Upload an updated patrol inspection record. Verify that the index automatically refreshes, confirming that incremental update configurations take effect.
The values provided are common starting points and should be measured against applicable local samples.