What Data Looks Like for This Category
Data for petroleum refining scenarios covers multiple types of documents and real-time collected data from production operations, process, scheduling, equipment, and laboratory departments. Daily documents include operating procedures, production plans, material balances, distillate export quality reports, interlock ledgers, shift handover records, and change management documents. Data sources include DCS/APC systems, LIMS systems, tank area metering systems, crude oil evaluation databases, equipment management systems, and energy management systems.
Data update frequencies fall into three categories:
- Real-time collection of process parameters
- Per-shift synchronization of shift handover and material data
- Daily updates of crude oil evaluation and quality reports
Data fields include process parameters, quality indicators, equipment status, metering data, and more. Units match corresponding industry standards based on their collection source.
Constraints Imposed on Database and Operations Maintenance
Multi-source heterogeneous data sources and varied update frequencies require database operations to adapt to different storage requirements. High-frequency writes of real-time process data generate pressure on database connections and disk I/O. Appropriate configuration of connection pools and storage strategies is needed.
Ledger documents from different departments have significant differences in format and fields. Unified data mapping rules are required to avoid field conflicts. Full change and traceability records must be retained for all data. This places requirements on the transactional and archiving capabilities of the database.
The scenario uses both relational and non-relational database types. Support for multiple database connection adaptations is required to ensure collection and integration of full data sets.
How to Configure Settings
The following configurations are formulated for FastGPT v4.17.0.
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
MONGODB_URI | mongodb://fastgpt:pass@db-host:27017/refinery?authSource=admin | Adapts to storage of unstructured ledgers and bulk historical data, uses authentication mode to secure data access |
PG_URL | postgresql://fastgpt:pass@db-host:5432/refinery_struct?sslmode=require | Used to store structured shift handover records and material balance data, supports complex queries and transaction operations |
DB_MAX_LINK | 20–30 | Matches concurrent write requirements for real-time data, avoids excessive connections consuming system resources |
REDIS_URL | redis://:fastgpt-pass@redis-host:6379/0 | Caches frequently accessed process parameters and scheduling instructions, reduces repeated read/write pressure on databases |
| Data Archiving Strategy | Hot data stored on SSD media, cold data regularly archived to low-cost storage | Adapts to fast read/write requirements for real-time data and long-term storage needs for historical data |
Three Common Misconfigurations
- Symptom: Database connection count exceeds limit, returns
503 Service Unavailablestatus code. Cause: TheDB_MAX_LINKconfiguration is not adjusted based on concurrent writes of real-time data, leading to exhaustion of connection resources. - Symptom: Databases show sustained high disk read/write activity after Docker deployment, leading to exhaustion of storage resources. Cause: No data hot/cold tiered storage strategy is configured. All high-frequency write real-time data is stored on standard disks without proper archiving.
- Symptom: Heterogeneous database connection fails, unable to pull quality report data from LIMS systems. Cause: No standard connection string configuration for the corresponding database is used, or access port permissions for the database are not opened.
How to Verify Configurations Are Correctly Set
- Run the built-in database connection test tool to verify the connection validity of
MONGODB_URIandPG_URL, confirm no authentication or network errors occur. - View the system monitoring panel to confirm the current database connection count does not exceed the configured
DB_MAX_LINKthreshold. Adjust the value based on actual data write volume. - Import a simulated shift handover record to verify that the data write and query process operates normally, with no missing fields or format errors.
- Check the operation logs of data archiving tasks to confirm that automatic migration of cold data to the specified storage media runs normally.
The values provided are common starting points and should be measured against the reader's own samples.