Database and Operations for Oil and Gas Exploration

What this category of data looks like

Oil and gas exploration investment research data comes from multiple sources: real-time telemetry data during drilling, geological reports from the completion phase, structured reports from reservoir dynamic monitoring, and large reservoir simulation files.

Update frequencies vary widely: real-time telemetry data is generated every 10 seconds, completion reports are updated after a single-well project is completed, and reservoir analysis reports are released quarterly or per project cycle.

Data structure falls into two categories: Structured data includes fields such as well ID, formation depth, porosity, and permeability. Units are string, meters, decimal ratio, and millidarcy respectively. Unstructured data includes LAS-format well logging files and PDF-format exploration reports. Individual unstructured files can reach hundreds of megabytes in size.

Constraints on Database and Operations

Real-time high-frequency telemetry data creates high concurrent write pressure. Databases must support low-latency batch writes to avoid data accumulation. Large unstructured LAS and PDF files require storage engines to support large file sharding and fast retrieval. Mixed access from multiple data sources requires databases to distinguish data from different sources via unified identifiers to avoid field conflicts. Strict field validation rules are needed to prevent invalid data from being written. Data update timing requirements are strict. Write order must match collection order to avoid deviations in investment research analysis.

Configuration Settings

Configuration Item	Recommended Value	Rationale
`DB_CONNECTION_TIMEOUT`	`30000 milliseconds`	Adapts to the synchronization delay requirements of oil and gas exploration real-time data, and avoids connection interruptions during high-frequency writes
`MONGO_WRITE_CONCERN`	`majority`	Ensures strong consistency of structured exploration data, and prevents data loss caused by single-node failures
`PARSE_LAS_FILE_TIMEOUT`	`600 seconds`	Adapts to the parsing time of large LAS well logging files, and avoids forced termination of large file parsing
`DB_BATCH_INSERT_SIZE`	`500 entries`	Balances the write throughput of real-time telemetry data and container memory usage, and avoids batch write timeouts
`PG_VECTOR_DIMENSION`	`1536`	Adapts to the vector embedding dimension of investment research documents generated by general large models, and ensures retrieval accuracy
`UPLOAD_FILE_MAX_SIZE`	`2048 MB`	Supports the upload and storage of large reservoir simulation files, and covers the import requirements of all types of oil and gas exploration data

The parameter values provided on this page are common starting points for configuration. Actual values are affected by material form, data volume and business rules. Specific issues require specific analysis. It is recommended to test on your own samples before finalizing settings.

Three Common Configuration Mistakes

Phenomenon: Database connection plugin execution time in workflows increases significantly. The old version takes approximately 0.2 seconds, while the new version can take up to 1 second. Cause: The new version enables vector index verification logic, and has not been optimized for the high-dimensional professional scenario of oil and gas exploration.
Phenomenon: MongoDB logs cannot distinguish the source of drilling data and well logging data. Cause: The MONGO_DOC_SOURCE_TAG parameter is not configured, and no source identifier field is added when writing data.
Phenomenon: Database connections in workflows throw connect ETIMEDOUT errors, while the target database can be accessed normally locally. Cause: The DB_ALLOWED_IPS configuration of the FastGPT container does not add the intranet IP segment of the target database, or the container network policy restricts outbound database ports.

How to Verify Successful Configuration

Run the database connection plugin test task, confirm that the returned execution time does not exceed the DB_CONNECTION_TIMEOUT threshold, and no timeout errors occur.
Import a standard LAS-format well logging file. Confirm that after successful parsing, complete structured field data can be viewed in the corresponding database table.
Check MongoDB logs, filter write operations, confirm that each piece of data carries a source identifier field, and different data sources can be distinguished via the identifier.
Check the container's network outbound rules, confirm that the target database port and IP segment are allowed to be accessed, and no connect ETIMEDOUT errors occur.

Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-09-14.