What this category’s data looks like
Data for petroleum refining scenarios comes from DCS/APC control systems, LIMS laboratory systems, tank farm metering systems, crude oil evaluation databases, equipment management systems, and energy management systems. Daily documents include operating procedures, production plans, material balance sheets, distillate export quality reports, interlock ledgers, shift handover records, and change management documents. Data update rhythms cover real-time, per-shift, and daily updates. Most documents contain fields such as process parameters, material data, and equipment status. Some documents require association with multi-dimensional information such as crude oil properties and unit load.
What constraints these characteristics impose on vector models and indexing
Multi-source heterogeneous data from petroleum refining creates differences in text formats and field structures. This requires vector models and indexes to adapt to the professional fields of different documents. The layered update rhythm of real-time, per-shift, and daily updates requires indexes to support incremental updates. This avoids resource consumption from full reconstruction. The business logic of coupled multiple units creates complex associations between texts. This requires retaining contextual association information during text chunking. The professional terminology differences between abnormal reports and process documents require vector models to accurately match scenario-specific process expressions. This avoids semantic confusion. Data volume continues to grow with the number of units and update frequency. This requires balancing index retrieval efficiency and storage costs.
How to set configurations
This page’s configurations are based on FastGPT v4.17.0.
| Configuration Item | Recommended Value Range | Rationale |
|---|---|---|
| Chunk Length | 800–1200 characters | Adapts to the length of single-segment process descriptions in process documents and abnormal reports, avoiding chunking that breaks complete business units such as interlock logic and material balances |
| Overlap Length | 100–150 characters | Retains contextual associations across chunks, ensuring that the process logic of coupled multiple units is not truncated during retrieval |
| Similarity Threshold | 0.75–0.85 | Filters low-match irrelevant data, adapting to the semantic similarity range of refining process terminology |
| Single Search Limit | Top 10 results | Controls the volume of retrieval results, avoiding exceeding the context window limit |
maxToken | Set to the selected embedding model/service maximum supported input tokens | maxToken is the input-token maximum supported by the selected embedding model/service; verify it in that provider configuration. See the v4.17.2 model-field reference. It is not retrieval/LLM context or a FastGPT-wide fixed cap. |
defaultToken | Calibrated based on actual measurements | Adapts to the input length limit of locally deployed embedding models |
Three common misconfigurations
- When calling a locally deployed embedding model, a connection error is returned. The interface displays model response timeout or invalid parameters. The cause is failure to correctly map the VLLM service port and access path, or the input text length exceeding the
defaultTokenlimit. - When troubleshooting knowledge base disk usage, all storage directories are mistakenly merged for statistics, making it impossible to locate the source of abnormal usage. The cause is failure to distinguish the independent storage paths of original files, split text chunks, and embedded vectors, confusing the disk usage proportion of the three types of data.
- After configuring the index, the number of retrieval results does not match expectations, with too many low-match irrelevant documents. The cause is failure to set a reasonable similarity threshold, or failure to adjust retrieval matching rules for professional refining process terminology.
How to confirm the configuration is correct
- Send a test request to the locally deployed embedding model, check whether the returned embedding vector format meets expectations, and confirm that the
defaultTokenconfiguration adapts to the model’s input requirements. - View the knowledge base storage directories, separately verify the storage paths of original files, split text chunks, and vector indexes, and confirm that the storage logic of the three types of data complies with configuration requirements.
- Retrieve documents related to specific process parameters, verify the number and matching degree of returned results, and confirm that the similarity threshold and single search limit configuration adapt to the business scenario.
- Test associated retrieval of chunked long texts, confirm that the overlap length configuration retains process contextual associations across chunks.
The values provided here are common starting points and should be measured against the reader's own samples.