What the data for this category looks like
Data sources include silicon wafer incoming material records, MES production systems, operation logs from diffusion furnaces, coating equipment and printing presses, EL and IV test data, and laboratory quality inspection reports. Updates occur immediately per production batch and per detected wafer, with full workshop data aggregated daily. Documents include process cards, diffusion and coating parameter tables, printing and sintering records, EL inspection images, IV grading data, and yield summary documents. Fields cover process temperature, equipment operating duration, defect location coordinates, electrical performance parameter values, and more. There are no unified fixed units; values vary by inspection item.
What constraints these characteristics impose on the vector model and index workflow
This category has high-frequency updates and multi-source heterogeneous data, which creates multiple constraints for the vector model and index workflow. Per-wafer immediate inspection data quickly increases index scale, requiring adaptive logic for incremental indexing and batch updates. Mixed multi-source data including text, numerical values, and EL inspection images requires a vectorization process that supports feature extraction for different formats. Strong contextual association between process parameters and defect data requires avoiding excessive document splitting that would lose associated information. Large differences in the length of fine-grained single pieces of data require flexible adjustment of chunking rules to adapt to different data types.
How to configure the system
defaultToken and maxToken are core parameters for configuring vectorization length and context limits. The following configurations are adapted for FastGPT v4.17.0.
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
| Chunk length | 800–1200 characters | Adapts to the standard length of text data such as process cards and parameter tables, avoiding single chunks exceeding the input limits of vectorization models |
| Overlap length | 50–100 characters | Retains process parameters and process context of adjacent chunks, preventing loss of associated information after splitting |
| Similarity threshold | Calibrated based on actual testing | Adapts to the semantic association characteristics of photovoltaic cell inspection data, flexibly matching different types of audit requirements |
| Single search limit | Top 20 entries | Controls the amount of returned data per retrieval round, adapting to low-latency call requirements at the workshop level |
maxToken | Set to the selected embedding model/service maximum supported input tokens | maxToken is the input-token maximum supported by the selected embedding model/service; verify it in that provider configuration. See the v4.17.2 model-field reference. It is not retrieval/LLM context or a FastGPT-wide fixed cap. |
defaultToken | Set according to the longest text segment in the data | Unifies the base length of vectorization, adapting to format differences across different documents |
Three common configuration mistakes
- The dataset remains in the indexing state for a long time after import. Cause: Incremental indexing is not configured for high-frequency updated photovoltaic cell inspection data, and full index reconstruction with excessive data volume causes process blocking.
- Semantic similarity values show abnormally high results after switching vector models. Cause: Standardization rules for similarity calculation are not adjusted, and non-standardized values directly output by the original model are used as matching criteria.
- Index speed fails to match real-time production requirements. Cause: Index scheduling logic for multi-replica deployment is not enabled, and single-process processing capacity is insufficient to support immediate per-wafer data updates.
How to confirm the configuration is correct
- View the running logs of index tasks to confirm there are no errors for failed document splitting, and that chunk lengths match the actual lengths of imported data.
- Run retrieval tests, compare return results for different types of data, and confirm that the similarity threshold configuration meets the matching requirements of current audit needs.
- Import a single batch of test data, confirm that the index completion time meets the time limits of production scheduling, and there is no situation of remaining in the indexing state for a long time.
- View resource usage of cluster nodes, confirm that index tasks for multi-replica deployment can allocate resources normally, and there is no process blocking.
The values provided are common starting points and should be evaluated against the reader's own samples.