What data in this category looks like
This category of data is generated and maintained by the change management team made up of R&D, process, quality, production, and procurement departments at manufacturing enterprises. Daily documents include ECN/ECR, change assessment materials, drawings and BOM versions, process verification reports, switching plans, inventory disposal documents, and customer approval documents. Data sources cover PLM, ERP, MES systems, supplier materials, verification test records, and customer special requirements. Updates occur on an irregular basis, with each change triggering a data update. The data includes structured fields and unstructured text. Some documents are linked to non-pure text content such as engineering drawings and BOM lists. Fields include version numbers, change reasons, verification results, switching nodes, and similar content.
What constraints do these characteristics impose on the vector models and indexes link
This category’s data is scattered and has diverse formats, including structured metadata, unstructured text, and non-pure text content. Indexes must support multi-source heterogeneous data access and unified parsing. Irregular change updates require indexes to use incremental update mechanisms, avoiding the long delays caused by full index reconstruction for every change. Documents linked to changes must be strictly bound to version information. Indexes must retain version metadata to distinguish between old and new versions, preventing mixing. Support for multi-modal open-source vector models is required, to enable encoding of non-text content such as engineering drawings. It must also support multiple index dimensions linked to the same document, to meet change-related retrieval needs.
How to set configurations
The following configurations apply to FastGPT v4.17.0
| Configuration Item | Recommended Range | Rationale |
|---|---|---|
| Chunk Length | 800–1200 characters | Adapts to the text density of documents such as ECN and process verification reports, avoiding information fragmentation within chunks |
| Overlap Length | 100–200 characters | Retains cross-chunk associated information such as change nodes and version numbers, preventing loss of context during retrieval |
| Similarity Threshold | 0.75–0.85 | Distinguishes change documents of different versions, avoiding confusion between old and new version retrieval results |
| Single Search Limit | Top 10 entries | Covers the number of associated documents involved in a single change, avoiding result redundancy |
defaultToken | 4096 | Adapts to vector encoding requirements for structured metadata and short text, and is compatible with the base encoding length of the ali-emb3 model |
maxToken | 8192 | Accommodates long text content such as BOM lists and customer approval documents, and matches the maximum input limit of multi-modal vector models |
Three common errors
- Retrieval results include old version process documents. This occurs when document version metadata is not bound in the index, leading to mismatched mixing of change documents from different versions.
- Index tasks stay in a pending state for extended periods. This happens when no incremental update mechanism is configured, with each change triggering a full index reconstruction that exceeds system processing limits.
- No matching results are returned after importing engineering drawings. This is caused by not enabling a multi-modal compatible vector model, using only a plain text encoding model to parse non-text content.
How to confirm configurations are properly set
- Upload a test ECR document, check the index task progress log, and confirm that incremental updates trigger normally, with no full reconstruction prompts.
- Enter a search term that includes change nodes and version numbers, verify that the document versions in returned results match the search term, and no old versions are included.
- Import a multi-modal test file such as an engineering drawing, confirm that the vector encoding task completes successfully, and that associated change assessment documents are returned during retrieval.
- Adjust the value range of configuration items, verify that retrieval result quantity and similarity meet expectations, and that no abnormal errors occur.
The values provided here are common starting points and should be evaluated against the reader's own samples.