What Data Looks Like for This Category
This category of data originates from manufacturing enterprise IPQC, FQC, quality engineering, and laboratory teams. Daily documents include control plans, patrol inspection records, SPC reports, final product inspection reports, non-conforming product review forms, release forms, and monthly quality reports. Data comes from MES systems, testing equipment, laboratory records, process parameter documents, customer standards, and historical defect databases. Updates are collected continuously per production batch and shift. Most documents combine structured fields and free-text descriptions. Fields include inspection items, parameter thresholds, judgment results, batch numbers, and more. There is no unified fixed format. Some documents include test values and units.
Constraints These Characteristics Impose on Vector Models and Indexing
Multi-source heterogeneous data sources require indexes to support unified vectorization of cross-system data. This avoids retrieval deviations caused by differences in data formats. High-frequency batch updates require indexes to support incremental synchronization. This prevents performance loss from full index reconstruction. Mixed structured and free-text document structures require a chunking strategy that balances field boundaries and semantic unit integrity. This stops cross-field truncation or semantic breaks. For bidding scenarios, users need to associate historical defect databases with customer standards. This requires indexes to support multi-document associated recall, ensuring retrieval results cover all business-required dimensions. Batch-level update granularity requires indexes to quickly locate associated documents by production batch. This avoids ineffective retrieval.
This configuration guide applies to FastGPT v4.17.0.
How to Set Configuration
| Configuration Item | Recommended Approach | Basis for This Setting |
|---|---|---|
| Chunk Length | 800–1200 characters | Adapts to the semantic unit length of documents such as patrol inspection records and SPC reports, avoiding cross-field truncation |
| Overlap Length | 100–150 characters | Retains semantic association between adjacent chunks, preventing information breaks between chunks |
| Similarity Threshold | 0.72–0.85, calibrated via actual testing | Matches the semantic similarity distribution of manufacturing documents, filtering invalid recall results |
| Single Search Limit | Top 8 entries | Adapts to multi-source reference needs during bidding document preparation, balancing retrieval efficiency and result coverage |
maxToken | Set to the selected embedding model/service maximum supported input tokens | maxToken is the input-token maximum supported by the selected embedding model/service; verify it in that provider configuration. See the v4.17.2 model-field reference. It is not retrieval/LLM context or a FastGPT-wide fixed cap. |
defaultToken | 1024 | Controls the output length of a single vectorized text, adapting to the input restrictions of most index models |
Three Common Mistakes
- Phenomenon: The number of chunks shown after uploading a single document does not match the final count after the document is ready, and duplicate chunks appear. Cause: No deduplication rule for incremental indexes is configured, or the chunk overlap length is set too large, causing adjacent semantically overlapping chunks to be recognized repeatedly.
- Phenomenon: The added ollama qwen2.5 language model and bge-m3 index model appear in the available model list, but no corresponding options display in the text understanding model dropdown when creating a knowledge base. Cause: The index model is not linked to the knowledge base's text understanding configuration item, or model channel permissions are not synchronized to the knowledge base module.
- Phenomenon: Only single-file training parameter adjustment and re-upload are supported. Batch triggering of vector model retraining is not available. Cause: The knowledge base's batch retraining function is not enabled, or batch processing trigger conditions and scope are not configured.
How to Confirm the Configuration Is Complete
- Upload a single typical document, such as a final product inspection report, and check if the chunk results match the preset chunk length and overlap length requirements.
- Run a retrieval test, enter a query term related to the historical defect database, and verify that the number and relevance of recall results meet business expectations.
- Submit a batch document upload task, and check that the index update progress matches the final chunk count, with no duplicate chunks.
- Navigate to the knowledge base's model configuration page, and confirm that the added index model and language model both appear in their respective dropdown options.
The values provided here are common starting points, and should be evaluated against your own specific samples.