This page is compatible with FastGPT v4.17.0.
What this category of data looks like
Data sources for this category include DCS/SIS systems, online analyzers, LIMS systems, equipment condition monitoring modules, energy metering devices, and raw material tank data collection nodes. Data update frequencies cover real-time (such as DCS parameters), shift-based (such as shift reports, special work permits), and daily (such as laboratory reports, process indicator cards). Daily documents include operating procedures, process indicator cards, DCS trend charts, laboratory reports, interlock bypass records, special work permits, and shift reports. Fields include equipment numbers, timestamps, process parameter values, and operation records. Units include common industrial units such as MPa, ℃, m³, hours, and others.
Constraints on vector models and indexing
Multi-source heterogeneous data sources require indexes to support mixed update modes. These modes accommodate real-time streaming data and scheduled batch data synchronization. Differences in document structure across types require vector chunking strategies to adapt to varying lengths of structured parameter text and unstructured operation records. Differences in update frequencies require distinguishing between incremental indexing for real-time parameters and scheduled full indexing for shift reports and laboratory reports. The defaultToken configuration must match the average encoding requirements of basic chemical manufacturing documents. This ensures core process content is fully encoded, and prevents key interlock logic from being truncated.
Configuration Settings
| Configuration Item | Recommended Value Range | Rationale |
|---|---|---|
| Chunk Length | 800–1200 characters | Adapts to the average content length of documents such as DCS trends and laboratory reports, avoiding overloading single chunks |
| Overlap Length | 50–80 characters | Retains cross-chunk process parameter association information, preventing chunking from cutting off key interlock logic descriptions |
defaultToken | 4096 | Matches the average encoding requirements of basic chemical manufacturing documents, covering the core content of a single batch of ledgers |
maxToken | Set to the selected embedding model/service maximum supported input tokens | maxToken is the input-token maximum supported by the selected embedding model/service; verify it in that provider configuration. See the v4.17.2 model-field reference. It is not retrieval/LLM context or a FastGPT-wide fixed cap. |
| Similarity Threshold | 0.72–0.80, calibrated via actual testing | Filters low-relevance non-process documents, accurately matching process indicators and abnormal records |
| Single Search Limit | Top 10 results | Controls the scale of retrieval results, avoiding context overflow during report generation |
Three Common Misconfigurations
- The number of retrieval results is far lower than expected. The index status shows as normal in the interface, but no matching data is returned. This occurs when associated storage for original documents and vector data is not configured. Only vector embeddings are stored in the vector database, and metadata identifiers for original text are lost. This prevents correct association with source documents during retrieval.
- Semantic retrieval returns similarity scores generally higher than 0.9, which exceeds a reasonable range. This occurs when process parameter text with units is not standardized. The vector model encodes similar parameters with different units into highly similar vectors, leading to abnormally high scores.
- Large model calls for report generation do not trigger the retrieval process. Normal question-and-answer workflows work without issues, but no retrieval data is available during report generation. This occurs when retrieval configurations are not bound to the production ledger summarization prompt template. The system does not trigger the vector index retrieval process.
How to Verify Correct Configuration
- View associated metadata in the vector database. Confirm that original document identification information is bound one-to-one with embedded vectors.
- Select a typical process indicator card as a test input. Initiate retrieval, and verify that the similarity score distribution of returned results matches the preset range.
- Trigger a production ledger summarization task. Check system logs to confirm the retrieval link is normally called and matching documents are returned.
- Check running records for the mixed update task. Confirm that synchronization of real-time streaming data and scheduled batch data is completed normally.
The values provided here are common starting points, and should be evaluated against the reader's own samples.