What Data for This Category Looks Like
Data for this category originates from slab or billet batches, heating furnaces, mill automation control systems, online thickness and width measurement equipment, surface inspection devices, and MES systems. Daily documents include rolling plans, heating schedules, rolling regulations, plate shape and thickness records, surface quality reports, and quality certificates. Data is generated in real time for each coil or individual steel product. Fields include coil batch numbers, heating temperatures, reduction rates, thickness deviations, surface defect types, with corresponding units such as degrees Celsius, millimeters, and percentages.
Constraints on Vector Models and Indexes
Real-time per-coil data requires indexes to support incremental synchronization, to avoid performance losses from full reindexing. Documents vary widely in length across industrial types, from short heating schedules to thousands-character surface quality reports. This requires support for text chunking of varying lengths. The specific semantics of industrial terms such as reduction rate and plate shape defects require vector models optimized for industrial scenario text encoding. Batch data generation requires indexes to support processing of over 100,000 mixed structured and unstructured data items in a single operation.
Configuration Settings
The following configurations are based on FastGPT v4.17.0, and are tailored to vector model and index requirements for steel rolling and finishing scenarios.
| Configuration Item | Recommended Range | Rationale |
|---|---|---|
| Chunk Length | 800–1200 characters | Adapts to typical lengths of documents such as rolling regulations and surface quality reports, avoids semantic breaks in single chunks |
| Overlap Length | 100–150 characters | Covers cross-chunk industrial term associations, such as continuous heating temperature curve descriptions |
| Similarity Threshold | 0.75–0.85 | Distinguishes semantic similarity between qualified and abnormal process parameters, meets quality judgment requirements for steel rolling |
| Single Import Limit | 5000 items | Balances import speed and system load, adapts to batch processing of 100,000+ data items |
defaultToken | 8192 | Adapts to context encoding for long documents, covers complete rolling plan content |
maxToken | 16384 | Supports vectorization of ultra-long quality certificate text, avoids content truncation |
Three Common Configuration Errors
- Issue: Vectorization takes excessively long after uploading a PDF to the knowledge base. Cause: Chunk length was not adjusted for long documents, causing single-chunk token counts to exceed model limits and triggering multiple segment encodings.
- Issue: After importing 100,000 CSV data items, the final vector dataset only contains approximately 90,000 items, with thousands missing. Cause: A reasonable single import limit was not set, and some batch data was overwritten by subsequent tasks before indexing completed.
- Issue: The vector index functions correctly when run locally, but vector scores are inconsistent after packaging as a Docker image. Cause: Industrial scenario vocabulary files were not loaded correctly in the image, leading to deviations in semantic encoding.
How to Verify Proper Configuration
- Import a single coil’s surface quality report PDF, confirm that chunked text blocks have no semantic breaks and no content truncation.
- Import 100,000 structured CSV data items, check that the number of entries in the completed vector collection matches the number of source data items.
- Submit a query request for an exception report, verify that the similarity values of returned similar documents fall within the preset threshold range.
- Start an incremental synchronization task, confirm that newly generated steel coil data can be automatically added to the index.
The values provided are common starting points and should be measured against the reader's own samples.