What the data for this category looks like
Data sources for oilfield services include real-time sensor data from drilling, fracturing, and logging equipment at job sites, third-party oil and gas market quotation data, and internal job cost accounting logs. This data is primarily used for revenue analysis of the oilfield services sector and daily market trend reporting in the financial industry. There are three update cycles:
- Real-time sensor data updates at high frequency according to job schedules
- Market quotation data updates periodically per market release schedules
- Job cost logs are generated after single-well job completion
Each data document is in JSON format, containing fields such as well_id, device_sn, operation_timestamp, daily_oil_output, operation_cost, market_oil_price. The units of the fields are, in order: none (unique well identifier), none (device serial number), ISO 8601 timestamp, cubic meters, yuan, yuan per ton. The size of a single document varies based on the number of fields and data volume.
What constraints do these data characteristics impose on database and operations work
The data characteristics of oilfield services impose multiple constraints on database and operations processes. First, real-time high-frequency sensor data generates continuous write traffic, with write rates during peak periods significantly higher than regular periods. This places high demands on database write performance and connection pool configuration. Second, fields and units from multiple data sources have natural differences. Unified validation must be completed before writing to avoid data storage errors. Third, revenue analysis requires retention of long-term historical data. A hot/cold data separation strategy must be configured to prevent hot storage resources from being occupied by long-term historical data. Fourth, different data sources have inconsistent update cycles. Layered storage and scheduled synchronization mechanisms must be designed to ensure data consistency.
How to set configurations
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
ob_connection_timeout | 30 seconds | Oilfield equipment sensor data transmission has fluctuations. An overly short timeout will cause normal connections to be interrupted |
milvus_insert_batch_size | 500 records/batch | Matches the single upload scale of oilfield job data, avoiding write blocking and resource waste |
vector_dimension | 768 | Corresponds to the standard dimension of embedding vectors for multi-dimensional oilfield indicators including output, cost, and price |
db_cleanup_retention_days | 180 days | Retains 180 days of real-time job data for revenue analysis. Expired data is archived to cold storage |
pgvector_index_type | ivfflat | Adapts to medium-scale vector query demands for oilfield data, balancing query speed and resource usage |
dataset_sync_interval | 5 minutes | Matches the update frequency of market quotation data, ensuring the timeliness of data used for revenue analysis |
The parameter values provided on this page are general recommendations used as a starting point for configuration. Actual values are affected by material forms, data volume, and business rules. Specific issues require case-by-case analysis, and it is recommended to test on your own samples before finalizing settings.
Three common mistakes
- Phenomenon: Database write traffic continues to rise, and hard disk IO utilization remains high for long periods. Cause: The write batch configuration was not adjusted for oilfield high-frequency real-time data, resulting in a single write data volume exceeding the database's carrying capacity, triggering frequent log writes and chunking operations.
- Phenomenon: Milvus vector database deployment fails to start, with logs indicating that the pgvector dependency was not found or configured incorrectly. Cause: Deployment templates for different databases were mixed, and pgvector configuration items were written to a Milvus instance without PostgreSQL deployed.
- Phenomenon: Some numeric fields in imported oilfield data have format errors, leading to abnormal query results. Cause: Data unit consistency was not validated, and production data with different units was directly imported into the database, causing calculation logic errors.
How to confirm configurations are properly set
- Run a stress test that simulates peak write traffic, observe database write latency and hard disk IO utilization, and adjust configurations to meet the demands of business peak loads.
- Check the service logs of Milvus and pgvector to confirm that the specified index type, sharding configuration, and connection parameters have taken effect, with no startup or runtime errors.
- Randomly select samples of imported oilfield data, compare the database table structure and field formats, and confirm that the units and types of all fields match the preset rules.
- Check the execution logs of scheduled tasks to confirm that data cleanup, archiving, and synchronization tasks are triggered and completed normally according to the preset schedule.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-09-14.