What this type of data looks like
Process, equipment, quality control, and production departments at crystalline silicon cell enterprises generate daily documents. These documents include process cards, diffusion and coating parameters, printing and sintering records, EL detection images, IV grading data, and yield reports. Data is sourced from silicon wafer incoming materials, MES systems, diffusion furnaces, coating equipment, printing presses, EL and IV testing equipment, and laboratory data. Data updates occur per production batch, per piece test output, and daily summary. Document fields include process parameter values, test results, batch numbers, and equipment numbers. Some documents include images and quantitative indicators.
What constraints do these characteristics impose on vector models and indexing
Data sources are scattered. Update frequencies differ. Support for multi-format data source access and incremental updates is required. The volume of parameter and test data per production batch grows with scale. Efficient index expansion support is required. Documents include structured numerical parameters and unstructured image content. Separate vector extraction logic is needed for the two data types. Process parameters have high precision requirements. Vector encoding must retain subtle numerical differences. This ensures retrieval matching accuracy. Associated retrieval of batch and equipment numbers is needed. Corresponding fields must be retained in index metadata for precise filtering.
How to set configurations
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
| Chunk Length | 800–1200 characters | Adapt to the paragraph length of documents such as process cards and parameter tables, avoid splitting that breaks the associated integrity of process parameter groups |
| Overlap Length | 100–200 characters | Retain process step connection information of adjacent chunks, prevent cross-chunk process logic from being disconnected |
maxToken | Set to the selected embedding model/service maximum supported input tokens | maxToken is the input-token maximum supported by the selected embedding model/service; verify it in that provider configuration. See the v4.17.2 model-field reference. It is not retrieval/LLM context or a FastGPT-wide fixed cap. |
| Similarity Threshold | 0.75–0.85, calibrated via actual testing | Distinguish subtle differences between parameters of the same process, avoid false matches or missed matches of valid process documents |
| Single Search Limit | Top 8 entries | Cover multiple types of process documents involved in a single production batch, avoid excessive retrieval results exceeding context carrying limits |
defaultToken | Set from representative sample documents | defaultToken is the default token count for direct text chunking; tune it on representative source documents. See the v4.17.2 model-field reference. It is not a vector dimension, embedding-model input cap, or retrieval/LLM context. |
Three common misconfigurations
- The number of retrieval results does not match the set single search limit. The automatic synchronization switch for incremental indexing is not enabled. Newly uploaded batch data is not included in the current index library.
- The large language model does not use retrieved document content to answer. Retrieved chunk content does not meet the preset similarity threshold. Or the
maxTokensetting is too small. Retrieval results are truncated and cannot be fully passed to the large language model. - Excessive matching deviation of process parameters occurs after vector encoding. Chunk length is set too small. Complete process parameter groups are split into multiple unrelated chunks. Retrieval cannot match complete process requirements.
How to confirm correct configuration
- Upload a single standard process card document in FastGPT v4.17.0. Check if the chunk preview retains complete parameter groups and process descriptions.
- Enter a query term for known process parameters. Verify that the similarity scores of retrieval results fall within the preset 0.75–0.85 range.
- Verify the incremental indexing function. Upload new batch test data. Confirm that the new data can be retrieved and hit.
- Check system logs. Confirm that no timeouts or format errors occur during the vector encoding process.
The values given are common starting points and should be measured against the reader's own samples.