What this type of data looks like
Water industry research data primarily comes from real-time monitoring systems of municipal water utilities, water quality and pipeline network data from water conservancy stations, bidding and operation archives of water projects, and official reports from water quality testing laboratories. Update frequency falls into three categories: minute-level updates for water pressure and water quality indicators at real-time monitoring points, monthly updates for monthly operational reports, and irregular updates for bidding and project archives.
Documents include structured monitoring fields and unstructured reports. Fields include point number, longitude and latitude, monitoring time, device number, and others. Units include standard fixed values such as mg/L (water quality indicators), kPa (pipe network pressure), m³/d (water supply), and more.
What constraints these characteristics impose on database and operations work
High concurrent write requirements for real-time monitoring data mean databases must support low-latency batch write operations to avoid excessive IO overhead from single-row writes.
Data tiering is distinct. Cold data such as historical monthly reports and old project archives must be stored separately from hot data such as real-time monitoring data to reduce hot storage load.
Fixed fields and units require databases to include built-in field validation and unit conversion rules to prevent incorrect data entry.
Irregularly updated unstructured documents require long-text parsing support, which imposes higher requirements for file parsing timeout settings.
Additionally, water data involves municipal public information, so strict backup and compliant storage strategies are needed to ensure data security.
How to set configuration parameters
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
vector_search_topk | Top 20-30 entries | Water industry research needs to retrieve multi-dimensional historical monitoring data and industry reports. Sufficient retrieval entries ensure coverage of relevant results. |
db_write_batch_size | 500-1000 entries per batch | Adapts to batch write scenarios for water industry real-time monitoring data, reducing IO resource consumption from single-row writes. |
PARSE_FILE_TIMEOUT_SECONDS | 300 seconds | Water quality testing reports often contain complex tables and long text. The default timeout duration is insufficient to complete parsing. |
milvus_collection_shards | 4-8 shards | Meets high concurrent write and query requirements for real-time monitoring data, improving overall vector database performance. |
OB_DB_POOL_MAX_CONN | 100-200 | Adapts to scenarios where multiple users initiate research queries simultaneously, preventing database connection pool exhaustion. |
retrieval_similarity_threshold | 0.75-0.85 | Distinguishes effective relevance for water monitoring indicators, preventing retrieval of irrelevant historical data. |
The parameter values provided on this page are general recommendations used as a starting point for configuration. Actual values are affected by material format, data volume, and business rules. Specific issues require case-by-case analysis. It is recommended to test on available samples before finalizing configuration.
Three common configuration mistakes
- Phenomenon: Sustained high disk read/write activity occurs after Docker deployment. Single-node daily write volume exceeds the warning threshold. Cause: Batch write parameters are not configured. Single-row data write mode is used, leading to excessive IO resource consumption and ultimately high disk load.
- Phenomenon: Milvus Compose deployment fails, with an error indicating incorrect storage engine configuration. Cause: Configuration items for vector databases and business databases are mixed. pgvector is incorrectly set as the default storage engine for Milvus, and correct dependent configurations are not matched.
- Phenomenon: 504 timeout error occurs when parsing water project documents. Cause: The file parsing timeout parameter is not adjusted. The default timeout duration is shorter than the time required for long document parsing, leading to task interruption.
How to confirm the configuration is correct
- Check the database connection pool monitoring dashboard. Confirm that the current active connection count does not reach the configured maximum connection limit. Adjust the value of
OB_DB_POOL_MAX_CONNbased on business peak loads. - Upload a typical water quality testing report. Check the completion status and duration of the parsing task. Verify that the
PARSE_FILE_TIMEOUT_SECONDSconfiguration is reasonable. - Write a batch of simulated real-time monitoring data. Check the query latency and write throughput via the vector database monitoring dashboard. Confirm that the
milvus_collection_shardsconfiguration meets business requirements. - Check the disk IO load monitoring. Confirm that the IO peak during batch write mode falls within a reasonable range. Verify the configuration effect of
db_write_batch_size.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-09-14.