What data for this category looks like
Data for this category comes from hot metal and scrap collection systems, L2 process control nodes, laboratory testing systems, MES production management systems, energy medium monitoring platforms, and equipment state sensors. Daily documents include heat plans, batching lists, converter smelting records, composition analysis reports, continuous casting process parameter records, and slab quality judgment sheets. Updates are triggered per heat, in real time, or per shift. Each individual document contains fields such as process parameters, composition values, and equipment operating status. Units use industrial standard values including temperature, flow rate, and composition percentage.
What constraints these characteristics impose on vector models and indexes
Data is updated in batches per heat with high real-time requirements. Indexes must support incremental updates to avoid interrupting high-temperature continuous production processes caused by full index reconstruction. Multi-source heterogeneous data has large format differences. Unified field mapping must be completed before vectorization to prevent deviations in semantic retrieval. The tightly coupled business logic between heats and casting runs requires indexes to associate fields including heat number, casting run number, and equipment ID. This ensures retrieval results match actual business scenarios. Each individual document contains a large number of continuous industrial parameters. The length of vectorized text per chunk must be limited to avoid exceeding model processing limits.
How to set configurations
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
| Chunk Length | 800–1200 characters | Individual steelmaking records contain multiple sets of continuous process parameters. Chunks that are too long will destroy semantic coherence, while chunks that are too short will lose process association logic. This range is calibrated based on actual testing of industrial document structures. |
| Overlap Length | 50–80 characters | Heat data has process connection logic across chunks. Overlap length covers key connection parameters to avoid semantic breaks. |
| Similarity Threshold | 0.72–0.85 | Steel equipment inspection questions mostly involve precise parameter matching. A threshold that is too low will introduce irrelevant equipment records, while a threshold that is too high will miss matching results. This range is calibrated based on actual testing of business scenarios. |
| Single Search Limit | Top 10 results | The number of equipment records related to a single heat is limited. Too many results will exceed the context window and reduce the accuracy of model inference. |
| maxToken | 4096 token | Matches the default model support limit for FastGPT v4.17.0, and is set based on the average length of steel process documents. |
| defaultToken | 2048 token | Adapts to the input length of routine inspection questions to ensure complete semantic embedding of retrieval requests. |
Three common errors
- Vector databases only store vector fields and cannot associate original document text. Retrieval results cannot be traced back to verify specific process records. Cause: No association storage logic between original documents and vectors is configured. Only vector data is written to the vector database, and no associated index for original text is retained.
- When using the bge-m3 vector model, the similarity scores returned by semantic retrieval are generally higher than normal business ranges. Cause: Unified mapping is not performed for field formats and units of multi-source heterogeneous data. Differences in original data units amplify semantic deviations in embedded vectors.
- Index status shows synchronization complete, but retrieval cannot match data related to the corresponding heat or equipment. Cause: Incremental update configuration is not bound to key business fields including heat number, casting run number, and equipment ID. Only non-business-related system logs are synchronized, and core business documents are not covered.
How to confirm proper configuration
- View the stored content of the vector database to confirm that both original document text and vector fields exist. Check that associated fields match business data.
- Input a typical inspection question, review the similarity score distribution of retrieval results, and adjust the threshold to a range that meets business requirements.
- Trigger a single heat data update, check if the index log records the incremental synchronization entry, and confirm that associated fields are correctly bound.
- Test retrieval templates and routine question-and-answer workflows to confirm consistency between returned results, and verify the adaptability of template configurations.
The values provided are common starting points and should be measured against the reader's own samples.