What this category of data looks like
Data for inspection and process record audit in petroleum refining scenarios comes from multiple types of documents and system data from production operation, process, laboratory and other departments of oil refining enterprises. Daily documents include operating procedures, production plans, distillate export quality reports, interlocking ledgers, shift handover records, etc. Data sources cover DCS/APC systems, LIMS laboratory management systems, tank farm metering systems, crude oil evaluation databases, equipment management systems and energy systems. Data update cycles include real-time (such as DCS process parameters), shift-based (such as shift handover records), and daily (such as material balance reports). Each single data entry contains professional fields and units, such as distillation range, flash point, equipment number, operation timestamp, etc.
What constraints these characteristics impose on the "vector models and indexes" link
Multi-source heterogeneous data sources and differences in update cycles require indexes to support a combination of incremental update and batch full update modes, to adapt to access of real-time stream data and scheduled batch data. The growth in data volume brought by multiple coupled continuous units requires index configuration to adapt to high-concurrency retrieval and sharded storage, to avoid excessive load on a single node. The professional document content and field units require vector models to have semantic understanding capabilities related to petroleum refining processes, to avoid semantic deviations of general models for professional terms. The wide span of document lengths, from short-text shift handover records to long-text production plans, requires chunking strategies to adapt to vector generation for multi-length texts.
How to set the configurations
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
| Chunk Length | 800–1200 characters | Adapts to the semantic integrity of single segments of refining documents, avoids semantic breakage of long texts and sparse vectors of short texts |
| Overlap Length | 100–150 characters | Retains contextual association between adjacent chunks, avoids loss of cross-chunk process logic |
defaultToken | Set from representative sample documents | defaultToken is the default token count for direct text chunking; tune it on representative source documents. See the v4.17.2 model-field reference. It is not a vector dimension, embedding-model input cap, or retrieval/LLM context. |
maxToken | 8192 | Ensures complete semantic input when processing long documents such as production plans, avoids truncation of key process parameters |
| Single Search Limit | 5–10 entries | Balances retrieval accuracy and response speed, adapts to the multi-unit associated retrieval requirements of refining scenarios |
| Reranking | Enabled, calibrated via actual testing | Optimizes retrieval ranking for professional terms, improves matching accuracy for inspection record audits |
Three common mistakes
- Symptom: After enabling the index model, the prompt "No available channels for the current group" is displayed. Cause: Access parameters for the vector model are not configured in the default group or specified group, only a large language model channel is configured.
- Symptom: No vector recall results are returned after deploying the BGE-M3 model via ollama. Cause: The interface address and request format of the vector model are not correctly configured, and the input and output requirements of the model are not matched.
- Symptom: Retrieval response time exceeds 30 seconds. Cause: The chunk length is set too large, resulting in an excessive number of chunks per batch, or the single search limit is configured too high, exceeding system load.
How to confirm the configuration is correct
- Enter the FastGPT vector model management page, confirm that the target model status shows Connected with no error prompts.
- Upload a distillate export quality report document from a refinery, perform a manual chunking test, and check that the chunk results fall within the preset length range.
- Initiate a retrieval request for the interlocking ledger, verify that the returned results include the interlock trigger time and parameters of the target process equipment.
- View the index update task logs, confirm that both real-time data and scheduled batch data update tasks are triggered normally, with no failed records.
The values provided are common starting points and should be measured against the reader's own samples.