What This Category of Data Looks Like
Data for this category comes from production operation, process, scheduling, equipment, and laboratory departments of refining enterprises. Daily documents include operating procedures, production plans, material balances, distillate export quality reports, interlock ledgers, shift handover records, and change management documents. Data sources cover DCS/APC, LIMS, tank farm metering, crude oil evaluation, equipment management systems, and energy systems. Update frequencies include real-time, per shift, and daily. Documents often include fields such as unit numbers, process parameters, quality indicators, and operation nodes. Units use industrial standard units for temperature, pressure, flow rate, and material quantity.
Constraints Imposed on Vector Models and Indexing Workflows
The data characteristics of this category impose multiple constraints on the vector model and indexing workflow. Multi-source heterogeneous data sources and varied update rhythms require flexible switching between incremental indexing and full indexing. This avoids excessive time spent on full index reconstruction triggered by real-time data. Documents contain a large number of refining-specific terminology and process parameters with units. Vector models must adapt to professional domain semantics in the industrial sector, while retaining unit information to ensure accurate retrieval matching. The business logic of multiple continuous units coupled through a logistics network requires indexes to retain the association relationships between documents. This prevents retrieval results from splitting the collaborative connections between units.
Configuration Settings
| Configuration Item | Recommended Range | Rationale |
|---|---|---|
| Chunk Length | 800–1200 characters | Refining documents often contain long paragraphs of process descriptions and parameter tables. This length can cover complete process logic units |
| Overlap Length | 100–200 characters | Prevents cross-chunk process parameters from being split, ensuring semantic coherence during retrieval |
maxToken | 8000–12000 token | Long refining professional documents require sufficient token space to retain complete semantics, matching the model input limit of FastGPT v4.17.0 |
| Similarity Threshold | 0.75–0.85 | Filters low-match non-professional documents, retaining retrieval results strongly related to process issues. Adjust based on actual testing |
| Single Search Limit | Top 10 results | Refining business issues often relate to collaborative data from multiple units. 10 results can cover most associated documents |
| Reranking | Enabled | Semantic matching for professional terms can be disrupted by surface keywords. Reranking improves the accuracy of result relevance ranking |
Three Common Misconfigurations
- Symptom: The indexing process stops at the processing step for the final set of documents. The interface continues to display "Index Building In Progress" with no progress updates. Cause: No incremental indexing strategy is configured for multi-source heterogeneous data from refining enterprises. Full indexing encounters blockages when processing large volumes of frequently updated documents.
- Symptom: Hybrid retrieval response time exceeds 10 seconds. Single retrieval return time is significantly higher than pure vector retrieval. Cause: No adjustment is made to the weight distribution of hybrid retrieval based on the professional semantic characteristics of refining documents. The additional keyword retrieval step introduces unnecessary computational overhead.
- Symptom: Some process flow diagrams and interlock logic diagrams cannot be vectorized correctly. No corresponding retrieval results appear in the knowledge base. Cause: Only text document vectorization configurations are supported. Parsing support for CAD drawings and PDF process flow diagrams commonly used in the refining industry is not added.
How to Verify Correct Configuration
- Import a set of typical refining process documents, perform retrieval tests, and check whether returned results include fields such as unit numbers and process parameters related to the query.
- Review indexing build logs to confirm that incremental indexing only processes updated documents, and no full index reconstruction is triggered.
- Adjust configuration parameter values, compare retrieval response times and result relevance across different parameters, and determine the value range that meets business requirements.
- Test vectorization results for different types of refining documents, such as shift handover records and interlock ledgers, to confirm that all supported document types can be parsed correctly.
The values provided here are common starting points. They should be evaluated against applicable samples.