What the data for this category looks like
This data is used by financial institutions to conduct due diligence on water treatment projects. It primarily comes from water utility operation logs, water quality monitoring reports, pipeline inspection records, environmental impact assessment approval documents, and water treatment process parameter documents. There are two update schedules: routine monitoring data is updated monthly, and emergency operation and maintenance records are synced in real time. Process parameter documents are statically archived and only updated when process modifications occur. Most documents are structured tables paired with attached PDF files, containing fields such as monitoring points, water quality indicators, treatment capacity, operating duration, etc. Water quality indicators use the unit mg/L, treatment capacity uses m³/h, and inspection records include timestamps and equipment numbers.
What constraints do these characteristics impose on the vector models and indexing workflow
The data characteristics of this category impose three constraints on the vector models and indexing workflow. First, structured tables contain numerical fields with units and text descriptions, so the vector model must be able to distinguish unit semantics to avoid confusing different meanings of similar numerical values during embedding. Second, there are two types of data update scenarios: real-time operation and maintenance records and static archived documents. This requires support for mixed scheduling of incremental indexing and full indexing to adapt to data sources with different update frequencies. Third, the documents include a large number of process flowcharts and PDF attachments of test reports, so it is necessary to adapt to long text segmentation and chart text extraction logic to avoid losing key process parameter information. At the same time, the scattered monitoring points result in a large number of data entries, so the processing scale of a single batch indexing must be limited.
How to set the configurations
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
chunk_size | 800–1200 characters | Water treatment due diligence documents mostly contain long-text process descriptions and table paragraphs. This range can retain complete parameter context and avoid losing associated information after splitting |
chunk_overlap | 100–150 characters | Retain overlapping content after long text segmentation to ensure semantic coherence between adjacent segments and cover cross-segment process parameter association logic |
similarity_threshold | 0.72–0.85 | The semantic similarity of water quality indicators and process parameters must be kept within a reasonable range to avoid recalling irrelevant monitoring data or process documents |
recall_top_k | Top 8–12 entries | A single due diligence report involves multiple monitoring points and process links. An appropriate number of recalls can cover all key information while reducing the subsequent reranking calculation load |
incremental_index_schedule | Once per hour | Adapt to the real-time synchronization requirements of emergency operation and maintenance records while balancing resource occupation for routine monitoring data updates |
max_index_batch_size | 200–300 entries/batch | Scattered monitoring points result in a large number of data entries. Limiting the batch size can avoid indexing timeouts and adapt to the computing resource limits of a single node |
The parameter values provided on this page are general recommendations used as a starting point for configuration. Actual values are affected by material form, data volume, and business rules. Specific issues require specific analysis, and it is recommended to test on your own samples before finalizing the configuration.
Three common mistakes
- Phenomenon: Indexing tasks remain in the "processing" state for a long time with no progress updates. Cause: The number of indexing entries in a single batch exceeds the configured limit, and no timeout termination parameter is set, causing the task to occupy resources continuously without completing.
- Phenomenon: After migrating indexing documents across environments, a large number of irrelevant contents appear in recall results. Cause: The vector model embedding dimensions used locally and on the server are inconsistent, or text preprocessing rules are not unified, resulting in mismatched embedding vector spaces.
- Phenomenon: Water quality indicator information with units cannot be recalled after indexing. Cause: The vector model does not perform differentiated embedding processing for numerical fields with units, leading to semantic confusion of similar numerical values after embedding.
How to confirm the configuration is correct
- Upload a single water treatment due diligence document, check the progress logs of the indexing task to confirm that the segment length matches the configured parameters.
- Initiate a retrieval request for water quality indicators, and verify whether the similarity scores of the recall results fall within the configured range.
- After configuring the incremental index, upload a new operation and maintenance record to confirm that the indexing task is automatically triggered according to the set scheduling rules.
- View the detailed information of the indexed documents to confirm that numerical fields with units and text descriptions are correctly embedded, with no field loss.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-09-14.