What the Data for This Category Looks Like
Data for process procedures and technical standards in petroleum refining scenarios primarily comes from production operations, process, scheduling, equipment, and laboratory departments. Daily documents include operating procedures, production plans, material balances, distillate outlet quality reports, interlock ledgers, shift handover records, and change management documents. Data sources cover DCS/APC, LIMS, tank farm metering, crude oil evaluation, equipment management systems, and energy systems. Update frequencies include real-time, per shift, and daily. Document structures vary significantly: operating procedures contain fields such as unit number, operation steps, and interlock parameters; shift handover records track on-shift parameters and abnormal conditions; distillate outlet quality reports list measured indicators with units, such as product density and sulfur content.
Constraints Imposed on Vector Models and Indexing by These Characteristics
Multi-source heterogeneous data sources require indexes to support embedding of mixed-format text and structured data, and to distinguish storage strategies for real-time DCS data streams and static procedure documents. Structured data such as real-time DCS parameters requires separate field extraction and embedding, differing from chunking strategies for pure text procedures. Different update frequencies require indexes to support incremental refresh, adapting to batch updates per shift or day and real-time parameter change synchronization, avoiding performance loss from full index reconstruction. Document length varies widely, from hundreds of characters for interlock ledger entries to thousands of characters for long production plans. Flexible chunking logic is needed to balance semantic completeness and embedding efficiency. Process parameter fields with units require retaining full text semantics during embedding, to avoid matching errors caused by missing units. The multi-unit coupled business characteristic means a single recall must cover multi-source information. The number of index return results and context length must be controlled to avoid exceeding model processing limits.
Configuration Settings
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
| Chunk Length | 800–1200 characters | Adapts to single-unit single-step operation instructions and supporting parameter text for refining procedures, avoids splitting cross-unit process logic into different chunks |
| Overlap Length | 100–150 characters | Retains cross-chunk process parameter associations, such as preceding and following explanations for interlock values, prevents key parameters from being split between two chunks |
| Similarity Threshold | 0.75–0.85 | Filters low-match non-process-related documents, avoids interference with scheduling decisions and process analysis |
| Single Search Limit | Top 8 results | Matches multi-unit coupled multi-source information requirements for refining, controls context length to meet model processing requirements |
| maxToken | Set to the selected embedding model/service maximum supported input tokens | maxToken is the input-token maximum supported by the selected embedding model/service; verify it in that provider configuration. See the v4.17.2 model-field reference. It is not retrieval/LLM context or a FastGPT-wide fixed cap. |
| defaultToken | 1024 | Unifies the token count for vector generation per chunk, ensures consistency of embedded vectors and retrieval efficiency |
Three Common Misconfigurations
- Phenomenon: Knowledge base disk usage is much higher than expected, with a large gap between the displayed original file size and actual usage. Cause: Failed to distinguish storage paths for original files, split chunks, and embedded vectors; did not enable automatic cleanup policies for vector data.
- Phenomenon: Search results return a large number of irrelevant shift handover records, with the number of returned entries exceeding the preset single search limit. Cause: Failed to correctly configure the single search limit parameter, or did not enable similarity threshold filtering, resulting in low-relevance short text entries being prioritized for recall.
- Phenomenon: Index model calls return a connection failure prompt, and the specified embedding model cannot be selected as the indexing channel. Cause: Failed to configure the corresponding vector model channel parameters in AIProxy, and did not complete model channel binding for FastGPT v4.17.0.
How to Verify Proper Configuration
- View disk usage statistics for the knowledge base storage path, confirm that three types of data are included: original files, split chunks, and embedded vectors.
- Submit a process-related query, verify that the similarity scores of returned results fall within the preset threshold range.
- Check the AIProxy configuration logs, confirm that the indexing model channel is connected normally with no errors.
- Import a new interlock ledger document, confirm that the index completes updates automatically with no abnormal errors.
- Verify the number of returned results from a single search, confirm that it matches the preset single search limit.
The values listed above are common starting points and should be evaluated against the reader's own samples.