What this category’s data looks like
Data for this category originates from production operations, process, scheduling, equipment, and laboratory departments of petroleum refining enterprises. Daily documents include operating procedures, production plans, material balances, distillation outlet quality reports, interlock ledgers, shift handover records, and change management documents. Data sources include DCS/APC, LIMS, tank farm metering, crude oil evaluation, equipment management systems, and energy systems. Update frequencies cover real-time, shift-based, and daily cycles. Some documents include clear process parameter units such as temperature, pressure, and distillation range values.
What constraints do these characteristics impose on the vector models and indexing workflow
Multi-source heterogeneous data sources require indexes to support mixed access of structured and unstructured data, adapting to formats ranging from real-time DCS data to offline ledgers. Differentiated update frequencies require a strategy combining incremental and batch indexing, to handle real-time data and periodically updated documents separately. Diverse document types demand adaptation to different text chunking rules, preventing truncation of long-text process descriptions or equipment parameters. The strong association between equipment inspection data and process parameters requires indexes to retain field association relationships, ensuring that equipment status and corresponding process indicators can be linked during Q&A.
How to set configurations
| Configuration Item | Recommended Setting | Rationale for This Setting |
|---|---|---|
| Chunk Length | 800–1200 characters | Adapts to long-text process descriptions and equipment parameters in refining documents, avoiding truncation of critical information |
| Overlap Length | 100–150 characters | Preserves cross-chunk process association logic, preventing loss of context for interlocks and load coordination |
maxToken | 4000 token | Carries process information for coupled multiple units, aligns with the context limit requirement of FastGPT v4.17.0 |
defaultToken | Set from representative sample documents | defaultToken is the default token count for direct text chunking; tune it on representative source documents. See the v4.17.2 model-field reference. It is not a vector dimension, embedding-model input cap, or retrieval/LLM context. |
| Single Search Limit | Top 6 entries | Controls the amount of returned information, avoiding redundant multi-unit data interfering with core queries |
| Similarity Threshold | 0.75–0.85, calibrated via actual testing | Matches the precision requirements for refining process parameters, filtering low-relevance non-process documents |
Three common mistakes
- Issue: The number of search results is much lower than expected after indexing, or unrelated non-process documents are returned. Cause: Chunking rules are not configured for the refining scenario, leading to truncation of long-text process parameters, making them unable to match retrieval keywords.
- Issue: Real-time DCS data fails to appear in search results after updates. Cause: An incremental indexing strategy is not configured, and only full index updates are used, which cannot adapt to the real-time data update rhythm of refining scenarios.
- Issue: Equipment information and process parameters cannot be matched in search results. Cause: Field association relationships for documents are not retained, and the corresponding logic between equipment IDs and process indicators is lost during chunking.
How to confirm configurations are properly set
- Review index task logs to confirm all configured data sources have been successfully accessed and indexed.
- Submit test queries related to equipment inspection, and verify that returned results include field information from corresponding documents.
- Adjust the similarity threshold to verify that the relevance of search results meets preset business requirements.
- Trigger an incremental indexing task to confirm that newly updated real-time process data can be retrieved normally.
The values provided here are common starting points and should be evaluated against the reader's own samples.