What this category’s data looks like
This category of data typically originates from wafer fab MES systems, equipment FDC modules, defect detection and metrology devices, process recipe libraries, and experimental databases. Daily documentation includes process flow cards, batch operation records, equipment operation recipes, SPC statistical charts, exception handling sheets, defect maps, and yield reports. Data updates follow wafer batch progression; some real-time parameters refresh alongside equipment operation, while bulk archived documents sync daily. Data fields include nanoscale process parameters, batch numbers, equipment operation duration, and more. Most data uses structured table and semi-structured text formats.
What constraints these characteristics impose on the vector model and indexing stage
The multi-source hybrid nature of wafer fabrication data requires indexes to support vectorization adaptation for both structured process parameters and semi-structured documents. Real-time updated equipment operation data needs to support high-frequency incremental indexing to avoid business delays caused by full reconstruction. Ultra-long process chains generate large document volumes, requiring index splitting to preserve the process logic integrity of individual content blocks and prevent breaks in cross-block process correlations. Nanoscale parameter precision means vector models must be sensitive to subtle numerical changes, requiring vectorization configuration tailored to industrial scenarios. Additionally, the total index volume for single-batch data must be limited to avoid excessive cluster resource consumption.
How to set configurations
| Configuration Item | Recommended Values | Rationale |
|---|---|---|
| Chunk Length | 800–1200 characters | Adapts to the mixed structure of process parameters and text in wafer fabrication documents, avoiding splitting process logic across single chunks |
| Overlap Length | 150–200 characters | Covers cross-block process correlation information, avoiding logical breaks in process steps |
| Similarity Threshold | 0.72–0.85 | Matches the subtle difference sensitivity of nanoscale parameters, filters invalid low-match results |
| Single Search Limit | Top 8 results | Adapts to the process correlation scope of a single wafer batch, avoids excessive redundant information interfering with Q&A |
maxToken | Set to the selected embedding model/service maximum supported input tokens | maxToken is the input-token maximum supported by the selected embedding model/service; verify it in that provider configuration. See the v4.17.2 model-field reference. It is not retrieval/LLM context or a FastGPT-wide fixed cap. |
defaultToken | 2048 | Balances the average length of industrial documents and vectorization calculation efficiency, avoids truncation of long chunk content |
Three common misconfigurations
- The knowledge base remains in the indexing state for an extended period after upload. This occurs when incremental indexing rules are not configured, and full indexing takes too long to process multi-source hybrid wafer fabrication data.
- Text chunk loss occurs after setting the chunk length to 3000 characters. This happens when
defaultTokenis not adjusted to accommodate long text splitting; long chunk content exceeds the model’s vectorization capacity limit, resulting in automatic filtering of some chunks. - Q&A results fail to associate with matching exception handling sheets. This can stem from a similarity threshold set too low, which incorrectly labels unrelated process parameter records as high-match results, or from not enabling document metadata indexing.
How to confirm proper configuration
- Upload a single typical equipment recipe document, review the index completion time, and confirm it falls within the business-allowed delay range.
- Submit a query containing nanoscale parameters, verify that the matching fields in the returned results align with the parameter precision of the source document.
- After adjusting the chunk length, check that the number of text chunks generated by indexing matches the expected splitting results, with no abnormal chunk loss.
- Review the FastGPT v4.17.0 vector model call logs, confirm that the
defaultTokenandmaxTokenconfigurations do not trigger model truncation errors.
The values provided are common starting points and should be measured against the reader's own samples.