What this category of data looks like
This category of data comes from manufacturing enterprise MES systems, online inspection equipment, laboratory systems, process parameter documents, customer quality standards, and historical defect libraries. Data is collected in real time per production batch and per shift, with some monthly report documents updated on a periodic basis. Daily documents include control plans, patrol inspection records, SPC reports, final product inspection reports, nonconforming product review forms, release forms, and quality monthly reports. Fields include batch number, inspection item, judgment standard, CPK value, defect description, audit status, and more. Some documents include numeric inspection data and units.
What constraints do these data characteristics impose on the vector models and indexing stage
This category’s data characteristics impose multiple constraints on the vector models and indexing stage. Data includes structured numerical values such as CPK values and inspection deviations, and unstructured text such as defect descriptions and review comments. This requires vector models to support vectorization processing for multiple types of features. High-frequency real-time batch data requires indexes to support incremental writing and low-latency retrieval, avoiding performance loss caused by full reconstruction. Diverse document types and closely linked fields—for example, release forms are linked to final product inspection reports—require indexes to support cross-document associated recall. Indexes must also adapt to differences in document lengths and flexibly adjust chunking rules. Historical defect library data accumulates continuously with production. Indexes must support a balance between capacity expansion and query performance. The defaultToken parameter must be used to adjust the token upper limit for single-block text to adapt to model restrictions.
How to set configurations
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
| Chunk Length | 800–1200 characters | Adapts to the content density of long documents such as control plans and SPC reports, avoiding overloading single chunks |
| Overlap Length | 100–200 characters | Retains contextual association between adjacent chunks, preventing semantic breaks in short documents such as patrol inspection records and defect descriptions |
defaultToken | Calibrated via actual testing | Adapts to the token limits of different vector models, matching the average length of documents in this category |
maxToken | No more than 90% of the model’s supported upper limit | Avoids vectorization failures caused by token limit exceedance, adapting to the input requirements of multimodal vector models |
| Single Search Limit | Top 10 results | Covers the recall scope of common historical defects and inspection standards, balancing retrieval efficiency and result completeness |
| Maximum Context | 4000–6000 characters | Matches the contextual association needs of this category’s audit scenarios, ensuring recalled content can support quality judgment |
Three common mistakes
- Phenomenon: Vectorization service logs show only single text chunks are submitted each time, with no batch request records. Cause: FastGPT’s batch vectorization configuration is not enabled. The default setting submits text one by one, and does not adapt to the array receiving rules of the vectorization service.
- Phenomenon: When retrieving chunked indexes, only partially truncated information is returned, and the complete chunked content of the original document cannot be obtained. Cause: The original text storage function of the index is not enabled. Only vectorized vector data is saved, and no chunked copies of the original document are associated.
- Phenomenon: After adding the Alibaba multimodal-embedding-v1 model, numerical deviations appear in vectorization results. Cause: Structured numerical fields in this category are not processed separately. Text and numerical values are directly mixed into the model input, and the input format requirements of the multimodal model are not followed.
How to confirm the configuration is complete
- Navigate to the vector model configuration page of FastGPT v4.17.0, check the batch vectorization switch status, and confirm it is enabled.
- Submit a test document containing multiple fields, run a vectorization task, and check the service logs to confirm batch request records exist.
- Retrieve the configured index, check the returned chunked content, and confirm complete original document fragments can be obtained.
- For multimodal model configurations, submit test data containing text and numerical values, and verify that vectorization results are normal.
The values provided are common starting points and should be measured against the reader's own samples.