What the data for this category looks like
This category's data covers the entire production process of liquid milk, yogurt, milk powder, and dairy ingredients. Daily documents include raw milk acceptance records, standardized batching sheets, pasteurization/UHT heat treatment records, CIP cleaning records, filling ledgers, sample retention and microbial test reports. Data sources include milk source testing equipment, MES systems, homogenization and heat treatment equipment, online temperature and flow sensors, CIP systems, and LIMS laboratory management systems. Updates are performed per milk tank and production batch. Key parameters such as heat treatment time and temperature, and online monitoring data are synchronized in real time. Each record includes fields such as raw material batch number, test value, equipment number, and operation time.
What constraints do these data characteristics impose on database and operations workflows
The data characteristics of this category impose multiple constraints on the database and operations stage. Multi-source heterogeneous data access requirements demand compatibility with data sources using different protocols such as equipment sensors, MES, and LIMS. Real-time synchronized online temperature and flow data, as well as heat treatment parameters, require support for high-concurrency writes. The batch update logic based on milk tanks and production batches requires the creation of hierarchical indexes by batch number and equipment number to accelerate ledger summary and cross-record associated queries. The production process has no redundant buffer. Data disconnection directly affects key decisions such as raw milk rejection and heat treatment verification, so the database must have high-availability fault recovery capabilities. At the same time, compliance records require data traceability, so a data backup and version retention mechanism must be configured.
Configuration Guidelines
This page applies to FastGPT v4.17.0. The following is the database and operations configuration plan:
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
MONGODB_URI | mongodb://dairy_prod:pass@prod-mongo-1:27017,prod-mongo-2:27017,prod-mongo-3:27017/?replicaSet=dairyRs&maxPoolSize=60 | Adapts to the replica set deployment for dairy processing, supports automatic reconnection when the primary node drifts, and meets real-time data write requirements |
DB_MAX_LINK | 60–100 | Matches concurrent write requests from multi-source equipment, covering access to multiple types of data such as raw milk testing and online monitoring |
REDIS_URL | redis://:cachePass@prod-redis:6379/1 | Caches batch summary data from frequent queries to reduce query pressure on the primary database |
PG_URL | postgresql://dairy_compliance:pass@prod-pg:5432/compliance_db?sslmode=require | Stores highly consistent compliance records such as sample retention microbial test reports and CIP verification logs |
| Database connection timeout threshold | 900 seconds | Adapts to long-cycle heat treatment and filling data write scenarios, avoiding data collection interruptions due to timeouts |
Three Common Misconfigurations
- Symptom: MongoDB Change Streams shows disconnection errors, with logs displaying
connection closed by remote host, and automatic recovery fails. Cause: Automatic reconnection parameters for the replica set are not configured. When the primary node drifts, existing connections cannot automatically switch to the new primary node. - Symptom: Cross-batch queries return empty results or incorrect row counts. Cause: Hierarchical indexes are not created by batch number and equipment number, leading to low query efficiency or failure to match corresponding records.
- Symptom: Database connection pool is exhausted, with a
too many connectionserror. Cause: TheDB_MAX_LINKparameter is not set appropriately. Concurrent write requests exceed the connection pool limit, causing new write requests to be rejected.
How to Verify Successful Configuration
- Connect to the database instance, verify the replica set member status, and confirm that connections can automatically switch when the primary node drifts.
- Insert a simulated raw milk acceptance record, query by batch number, and confirm that the corresponding ledger data can be retrieved normally.
- Initiate multiple concurrent query requests, observe the database connection count, and confirm that it does not exceed the configured connection limit.
- Trigger a data backup operation, confirm that the backup process executes normally and the data is complete.
The values provided are common starting points and should be evaluated against the user's own samples.