What Data for This Category Looks Like
Data for this category originates from DCS/APC, LIMS, tank farm metrology systems, crude oil evaluation systems, equipment management systems, and energy systems. Daily documents include operating procedures, production plans, material balances, distillate export quality reports, interlock ledgers, shift handover records, and change management documents. Fields cover distillation range, density, ppm concentration, shift number, unit load values, crude wax content, and other related metrics. Data updates follow real-time, per-shift, and daily schedules. Production operations, process, scheduling, equipment, and laboratory departments maintain this data on a regular basis.
Constraints Imposed on Vector Models and Indexing
The data characteristics of this category impose multiple constraints on the vector models and indexing workflow. Heterogeneous data sources create notable differences in text formats. Both structured laboratory data and long-form process documents spanning thousands of characters require adapted chunking strategies. Multiple update schedules—real-time, per-shift, and daily—require indexing to support multiple refresh modes, including real-time incremental updates and daily full updates. Raw material and product quality data have distinct feature differences. Similarity thresholds must be adjusted for specific process scenarios to avoid misjudging similar content. Multi-unit coupled business logic requires vector recall to cover associated data. The single recall limit must align with the business coverage scope.
Configuration Settings
| Configuration Item | Recommended Setting Range | Rationale |
|---|---|---|
| Chunk Length | 800–1200 characters | Matches the average length of refinery long-form process documents, and avoids exceeding model token limits for individual chunks |
| Overlap Length | 100–150 characters | Preserves contextual information between adjacent chunks, and prevents loss of associated logic when long-form process descriptions are split |
| Similarity Threshold | 0.72–0.85 | Matches the feature distinctiveness of refinery process data, and avoids misclassifying different raw material properties or product quality data as similar |
| Single Search Limit | Top 10 results | Covers multi-unit coupled business associated data, and meets information needs for supplier audits and process quality troubleshooting |
maxToken | 4096 token | Aligns with the model input limit of FastGPT v4.17.0, and supports vector calculation requirements for refinery long documents |
defaultToken | Calibrated via actual testing | Adjusts for differences in token density across different document types, and optimizes consistency of vector calculations |
Three Common Misconfigurations
- Abnormal vector calculation scores, with minimal score differences between similar content. This occurs when similarity thresholds are not adjusted for refinery process data features, and default parameters for general scenarios are used instead.
- Excessively long vectorization time after uploading PDF versions of refinery operating procedures. This happens when chunk length settings exceed the reasonable range for refinery long text, leading to too many tokens per chunk and extended calculation time.
- A reduction of thousands of records occurs after vectorizing 100,000 CSV files containing laboratory data. This stems from unescaped special unit symbols (such as ppm, ℃) in some data, which interrupts the vectorization process without creating a recorded error.
How to Verify Proper Configuration
- Randomly select 3 different types of refinery documents (such as operating procedures, distillate export quality reports, and shift handover records) and run a vector chunking test. Confirm that chunk results fall within the preset length range.
- Import 100 test records containing special unit symbols. Check that the number of vectorized records matches the source data, with no abnormal losses.
- Trigger a batch vectorization task and monitor task duration. Confirm that the duration falls within the preset reasonable range.
- Input a process-related query such as "How does crude wax content affect fraction yield?" Check that the similarity scores of returned similar documents fall within the preset threshold range.
The values provided here are common starting points, and should be evaluated against the reader's own samples.