What the data for this category looks like
This category covers continuous production scenarios at basic chemical enterprises including chlor-alkali, synthetic ammonia, methanol, and soda ash. Daily documents include operating procedures, process indicator cards, DCS trends, laboratory reports, interlock bypass records, special work permits, and shift logs. Data sources include DCS/SIS systems, online analyzers, LIMS systems, equipment condition monitoring, energy metering devices, and raw material storage tank monitoring nodes. Data update frequencies cover real-time (process parameters), per shift (shift logs, work permits), and daily (laboratory reports). Documents include structured fields and unstructured text. Structured fields include timestamps, process parameter values such as pressure in MPa, temperature in ℃, and qualified judgment results.
Constraints on Vector Models and Indexes
The mixed data types and layered update rhythms of this category require index strategies that adapt to both structured process parameters and semantic matching for non-compliant records. Real-time updated DCS trend data requires indexes to support incremental synchronization, to avoid performance losses from full reconstruction. Differences in field units across multi-source data such as MPa and bar for pressure require vector models to retain unit information during encoding, to prevent semantic matching deviations. Splitting key compliance documents such as interlock bypass records must not be too fine-grained. Operation context must be preserved to avoid loss of critical associated information during recall.
Configuration Settings
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
| Chunk Length | 800–1200 characters | Adapts to the context length of single records such as process indicator cards and DCS trends, and avoids truncation of critical process parameters |
| Overlap Length | 100–150 characters | Preserves the association between adjacent process parameter sequences, and prevents loss of logical links for continuous production after chunking |
| Similarity Threshold | 0.7–0.9 | Filters low-match irrelevant documents, and focuses on matching results for critical process indicators and compliance records |
| Single Search Limit | Top 8 entries | Covers the recall range for on-shift critical alarms and operation records, and avoids redundant results interfering with audit judgments |
defaultToken | Set from representative sample documents | defaultToken is the default token count for direct text chunking; tune it on representative source documents. See the v4.17.2 model-field reference. It is not a vector dimension, embedding-model input cap, or retrieval/LLM context. |
maxToken | Set to the selected embedding model/service maximum supported input tokens | maxToken is the input-token maximum supported by the selected embedding model/service; verify it in that provider configuration. See the v4.17.2 model-field reference. It is not retrieval/LLM context or a FastGPT-wide fixed cap. |
This configuration was validated on FastGPT v4.17.0.
Three Common Configuration Mistakes
- Phenomenon: Custom-split document chunks are automatically deleted after being stored in the knowledge base, and index order does not match expectations. Cause: The default index strategy performs deduplication on semantically duplicate text chunks, and does not retain user-defined chunk sequences.
- Phenomenon: Custom-added vector models do not take effect, and search requests are always forwarded to the large language model. Cause: The vector model channel configuration is not correctly associated with the knowledge base index link, and requests are routed to the general large language model by default.
- Phenomenon: Similarity values returned by knowledge base searches exceed the 0-1 range, and valid results cannot be filtered using conventional rules. Cause: The vector model in use does not perform normalization processing on outputs, leading to similarity calculation results outside the standard range.
How to Verify Correct Configuration
- Upload a single DCS trend document, and check if the chunked text retains the complete process parameter sequence, with no critical values or timestamps truncated.
- Initiate a search containing specific process parameter keywords, confirm that the similarity values of returned results fall within the configurable range, and adjust the filtering threshold according to business needs.
- View the index refresh log, confirm that real-time updated DCS data can complete index synchronization within the set cycle, with no delays or data loss.
- Test index access under multi-replica deployment, confirm that search requests are evenly distributed to each replica node, with no single-point overload issues.
The values provided here are common starting points and should be measured against the reader's own samples.