What the data for this category looks like
Daily business documents for welded structural component manufacturing include welding procedure qualifications, welding procedure specifications, consumable issue records, welder qualification records, weld inspection reports, repair orders, and more. Data sources include design drawings, material quality certificates, welding equipment parameters, consumable warehouse management systems, non-destructive testing systems, personnel qualification ledgers, and other sources. Data updates occur per weld, per batch, or per shift. Individual records contain fields such as weld identifiers, material parameters, personnel qualification numbers, test results, and repair records. The data includes both structured table and unstructured text types.
What constraints these characteristics impose on the vector model and indexing workflow
High data update frequency and fine granularity require indexes to support incremental updates and fine-grained chunking, to avoid performance losses from full index rebuilding. Documents include both structured and unstructured formats, so mixed-format vectorization processing must be supported to ensure consistent vector extraction for structured records like consumable issue sheets and unstructured text like procedure specifications. Business data has strong relevance: multiple ledgers linked to the same weld must be recalled together, so the recall range must be precisely controlled to prevent irrelevant data from being included in summary results. Data from multiple systems requires unified field mapping to avoid retrieval deviations caused by field confusion during indexing.
How to set the configurations
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
| Chunk Length | 800–1200 characters | Matches the information density of ledgers associated with a single weld, and avoids splitting that would destroy the integrity of process parameters and test results |
| Overlap Length | 100–150 characters | Preserves key associated information such as cross-chunk weld numbers and consumable grades, preventing context breaks |
| Similarity Threshold | 0.75–0.85 | Matches highly relevant business data such as weld inspection reports and repair records, reducing the recall rate of irrelevant records |
| Single Search Limit | Top 10 entries | Covers multiple types of ledger records for the same batch or weld, meeting the information needs of production ledger summaries |
defaultToken | 8192 | Adapts to vectorization processing for long documents such as welding procedure specifications and non-destructive testing reports |
maxToken | Set to the selected embedding model/service maximum supported input tokens | maxToken is the input-token maximum supported by the selected embedding model/service; verify it in that provider configuration. See the v4.17.2 model-field reference. It is not retrieval/LLM context or a FastGPT-wide fixed cap. |
Three common configuration errors
- Phenomenon: After upgrading from FastGPT v4.9.0 to v4.9.3, previously uploaded welding ledger documents can no longer be retrieved. Cause: Incremental index synchronization was not completed. The index structure of the old version is incompatible with the new version. A full or incremental index rebuild is required.
- Phenomenon: After uploading an Excel file of consumable issue records, indexed content has missing fields or chaotic formatting after vectorization. Cause: Structured parsing configuration for Excel tables was not enabled, and business field mapping corresponding to the table header was not specified. This results in incomplete vector extraction.
- Phenomenon: After using the knowledge base search citation merging function, the generated production summary report contains redundant content. Cause: Citation merging matching rules were not configured correctly, leading to multiple citations of different chunks from the same ledger.
How to confirm the configuration is correct
- In FastGPT v4.17.0, upload consumable issue records and non-destructive testing reports corresponding to a single weld, run a keyword search, and confirm that the corresponding documents are retrieved.
- Adjust the similarity threshold, verify that highly relevant weld data is recalled first, and low-relevance data is not incorrectly recalled.
- Upload multiple types of ledger documents for the same batch, run a production summary query, and confirm that returned content covers all associated records with no obvious duplicates.
- Verify the vectorization results of long documents such as complete welding procedure specifications, and confirm that key process parameters are not truncated.
The values provided here are common starting points and should be evaluated against the reader's own samples.