What data for this category looks like
Data comes from hot rolling, cold rolling, section steel, and steel pipe enterprises, covering slab or billet batches, heating furnaces, mill automation control systems, online thickness and width measurement equipment, surface inspection devices, and MES systems. Daily documents include rolling plans, heating schedules, rolling specifications, shape and thickness records, surface quality reports, and quality certificates. Data updates per coil or individual billet. Some parameters sync in real time. Data fields include batch number, heating temperature, rolling reduction, thickness tolerance, surface defect level, and more. Associated units include degrees Celsius, millimeters, percentage, and others. Documents exist in both structured table and unstructured report formats.
Constraints on vector models and indexing
Real-time or coil-level updated data creates demand for high-frequency index updates. Systems must support small-batch, high-frequency vector ingestion operations, to avoid excessive single-ingestion time that disrupts production data synchronization. Multiple document structure types require indexes to support both structured field association and unstructured text semantic retrieval. This avoids retrieval bias from single-index dimensions. Fine-grained data per coil or billet requires retention of batch traceability links. Indexes must bind batch number as a metadata field, to ensure retrieval results can be traced back to the corresponding steel coil. Long text content from unstructured documents such as surface quality reports and quality certificates requires appropriate chunking rules. This avoids semantic fragmentation that reduces retrieval accuracy.
Configuration Settings
| Configuration Item | Recommended Approach | Rationale |
|---|---|---|
| Chunk Length | 800–1200 characters | Matches semantic integrity of long texts such as rolling specifications and surface quality reports, avoids content fragmentation in single chunks |
| Overlap Length | 100–150 characters | Retains semantic connections between adjacent chunks, adapts to short-cycle data block handoffs for real-time updates |
| Similarity Threshold | 0.75–0.85, calibrated via actual testing | Aligns with the strictness of defect judgment in steel inspection, avoids false or missed retrievals |
defaultToken | 4096 | Adapts to maximum input limits for long documents such as quality certificates, matches default configuration for FastGPT v4.17.0 |
| Maximum Context | 8000 characters | Integrates inspection and process records from multiple associated documents, supports semantic retrieval for full-batch traceability |
maxToken | 8192 | Matches maximum input token limit for vector models, ensures long text chunks can be fully embedded into vector space |
Three Common Mistakes
- Phenomenon: Indexes for manually inserted inspection record documents disappear automatically after several hours. Cause: No persistent storage policy is configured for indexes. FastGPT’s default temporary index cache is not bound to batch metadata, so it is cleared during memory reclamation.
- Phenomenon: Recall rate for full historical rolling data shows no significant improvement after switching embedding models. Cause: The full batch re-embedding process for historical data is not executed. Only vectors for new documents are updated. Historical data still uses vectors generated by the old model, leading to mismatched vector spaces.
- Phenomenon: Vector generation fails for surface quality reports, with error prompts during retrieval. Cause: Long text chunks are not truncated per the
maxTokenconfiguration. This exceeds the input limits of the local embedding model, so the model cannot process the input content.
How to Verify Proper Configuration
- Upload a rolling plan document. Check vector generation logs to confirm chunk length and overlap length match configured values. Verify matching between
defaultTokenandmaxToken. - Manually trigger an index update for a single batch of billet data. Check the knowledge base index list to confirm the batch number tag is retained. Verify indexes are not automatically cleared.
- After switching embedding models, run the full batch re-embedding operation. Check the vector update status in the knowledge base. Confirm all historical documents have completed re-embedding.
- Initiate an inspection record retrieval. Set a batch number filter condition. Check that returned results only include documents related to the target batch. Confirm the similarity threshold filtering logic is active.
The values listed here are common starting points, and should be evaluated against the reader’s own operational samples.