What data in this category looks like
Data for this category comes from iron melt and scrap charging records, L2 process control systems, laboratory composition reports, MES production management systems, energy medium monitoring, and equipment status sensors. Daily documents include heat plans, batching lists, converter smelting records, continuous casting process records, slab judgment sheets, and more. Fields cover furnace temperature, charging amount, composition test values, casting speed, equipment operating conditions, and other items. Units include degrees Celsius, tons, meters per second, and others. Data update frequency follows these rules: dedicated ledgers are generated per heat, process data is uploaded in real time, and summary statistics are completed per shift. The length of single records varies widely.
What constraints do these characteristics impose on vector models and indexes
This category of data has high update frequency and strong real-time requirements. Vector indexes must support incremental updates to avoid performance losses from frequent full index builds. Multiple types of mixed documents require vector models to adapt to mixed features of structured fields and semi-structured text. This ensures valid vectors can be generated for ledger data in different formats. The tightly coupled business logic between heats and casting batches requires indexes to support cross-document associated retrieval. This prevents damage to business relationships from split data. Data fields include specific physical quantities and units. Chunk processing must retain field integrity to prevent key business information from being truncated.
How to set configurations
This page applies to FastGPT v4.17.0. The following configurations are based on the interface and parameter logic of this version:
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
| Chunk Length | 800–1200 characters | Adapts to the length range of single heat ledger records, avoids splitting that breaks field associations between batching lists and smelting records |
| Overlap Length | 50–80 characters | Retains cross-chunk process step association information, prevents key casting links from being truncated by chunking |
| Similarity Threshold | Calibrated based on actual measurements | Adapts to the feature distribution of steel production data, avoids mismatching or missing matching of abnormal working condition data |
| Single Search Upper Limit | Top 10 results | Matches the requirements for multi-link associated retrieval in the continuous casting process, covers the complete business chain from heat to slab |
defaultToken | Set from representative sample documents | defaultToken is the default token count for direct text chunking; tune it on representative source documents. See the v4.17.2 model-field reference. It is not a vector dimension, embedding-model input cap, or retrieval/LLM context. |
maxToken | 32000 | Supports the total context required for complete production report generation, avoids triggering length limits during generation |
Three common mistakes
- Phenomenon: Manually inserted heat ledger indexes disappear automatically after several hours. Cause: Persistent storage for incremental indexes is not configured, and real-time updated ledger data is not correctly written to the vector database index files.
- Phenomenon: File status remains "Indexing" and index construction cannot be completed. Cause: The embedding model is incompatible with the FastGPT version, and the
defaultTokenparameter is not configured as required to adapt to the field length of steel data. - Phenomenon: Search results only return a single heat record, and cannot retrieve associated continuous casting process data. Cause: Chunk length is set too short, splitting destroys the association fields between heats and casting batches, preventing vectors from matching cross-document association information.
How to confirm configurations are correct
- Upload a single complete heat ledger, wait for index construction to complete, search for the batching list field of that heat, confirm that the returned results contain complete associated fields.
- Wait 24 hours and check the inserted ledger index list, confirm that all real-time updated record indexes are not lost.
- Call the retrieval interface, verify that the number of returned results matches the configured single search upper limit.
- Generate a production report, check that the context length included in the report does not trigger the
maxTokenparameter limit.
The values provided are common starting points and should be measured against the reader's own samples.