What data looks like for this category
Data sources for this category include product drawings and 3D models, CAPP/CAM systems, machine tool collection data, tool management systems, MES systems, and measurement and inspection tool records. Document types cover process routes, operation cards, tool lists, CNC programs, first-piece inspection records, patrol inspection forms, and measuring tool calibration records. Updates are triggered by work orders, single-piece machining runs, spot checks, and tool replacements. The data includes two types of content: structured fields and unstructured text.
Constraints imposed on vector models and indexing
The mixed-type data structure requires a chunking strategy that balances structured field integrity and unstructured text contextual coherence. The high-frequency update feature requires indexes to support incremental synchronization, avoiding resource consumption from full reconstruction. Industry-specific professional terms and formats require chunking that is not overly fine, to prevent splitting of professional code segments. The strategy must also adapt to token limits of mainstream vector models, avoiding single chunks exceeding input upper limits. There are large granularity differences across documents: a single CNC program segment can reach hundreds of characters, while a patrol inspection form only has tens of characters. Fixed chunking parameters easily lead to content truncation or redundancy.
How to set configurations
The following configurations are based on FastGPT v4.17.0.
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
| Chunk length | 800–1200 characters | Adapts to mixed content in metal machining documents, prevents continuous G-code segments in CNC programs from being truncated unnecessarily, and keeps per-chunk token count within the maxToken limit of mainstream vector models |
| Overlap length | 100–200 characters | Preserves professional terms and code context across chunks, preventing critical information such as tool models and anomaly descriptions from being split apart |
maxToken | 1024 | Matches the input token upper limit of the specified vector model, avoiding content truncation errors during vectorization |
| Similarity threshold | 0.75–0.85 | Filters low-match degree equipment inspection anomaly descriptions, and accurately recalls relevant documents based on anomaly characteristics in metal machining scenarios |
| Single search limit | Top 8 entries | Balances recall coverage and response speed, adapting to fast Q&A needs in workshop scenarios |
defaultToken | Set from representative sample documents | defaultToken is the default token count for direct text chunking; tune it on representative source documents. See the v4.17.2 model-field reference. It is not a vector dimension, embedding-model input cap, or retrieval/LLM context. |
Three common configuration errors
- Uploading CNC program files larger than 10 MB results in system prompts for partial chunk vectorization exceptions, and repeated retries do not fully resolve the issue. The root cause is that chunk length settings do not match the
maxTokenlimit; continuous G-code segments exceed the model's input token upper limit, triggering vectorization failure. Retries only reset temporary caches and do not fix the underlying configuration issue. - After importing Excel-format operation cards, recall results for tool wear-related queries are inconsistent. The root cause is that reasonable chunking rules are not applied to structured table data, leading to incorrect splitting of header fields and corresponding data rows, and breaking contextual associations of professional fields.
- After locally deploying ollama, the response speed of equipment inspection Q&A is too slow. The root cause is that the
defaultTokenlength of Q&A context is not limited, causing the model to process excessive redundant historical data and exceeding the inference capacity of local CPUs.
How to confirm configurations are properly set
- Upload a single typical document (such as a complete CNC program) and check the chunk preview interface, confirming that continuous G-code segments are not truncated unnecessarily.
- Input typical equipment inspection questions such as tool wear or dimensional drift, verifying that recalled chunks include corresponding operation cards, tool lists, or patrol inspection form content, and that match degrees meet expected thresholds.
- Perform batch vectorization tests on the local server, confirming that no continuous vectorization exception errors occur.
- Check the token consumption records of Q&A context, confirming that they do not exceed the context window limit of the deployed model.
The values provided here are common starting points and should be measured against the reader's own samples.