What the data for this category looks like
Typical data sources for this category include cell banks and seed lots, process analysis, cleanroom monitoring, MES, LIMS, and equipment integrity testing. Daily documentation covers cell bank or seed lot records, culture and purification records, aseptic filling records, environmental monitoring, media fill simulation, sterility testing, and batch release documentation.
Data update frequencies fall into three categories: per batch, real-time, and per monitoring plan. Document fields include batch identifiers, process parameters, monitoring times, test conclusions, and corresponding units such as degrees Celsius, colony-forming units per square meter, and hours.
What constraints these characteristics impose on vector models and indexing
Multi-source, decentralized data sources require indexes to support unified access and correlation of cross-system data. Data with different update frequencies need mixed configuration of incremental and full indexes. This prevents lag in real-time monitoring data or delayed synchronization of batch records.
Data structures tied to batches require indexes to be partitioned by batch. This ensures recall results only relate to records associated with the current problem batch. Business characteristics that rely on process and environmental data require vector models to correlate different types of parameters and test results. This improves the accuracy of traceability.
How to set configurations
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
| Chunk Length | 800–1200 characters | Matches the average length of single process records for this category, and aligns with input limits of mainstream vector models |
| Overlap Length | 100–150 characters | Prevents loss of contextual association for cross-chunk critical parameters such as culture temperature ranges |
maxToken | 4096 token | Aligns with input upper limits of mainstream vector models, and matches total content volume of single batch records |
| Similarity Threshold | 0.75–0.85 | Filters low-correlation non-target batch data, and accurately recalls abnormal records from the same batch or process |
| Single Search Limit | Top 10 entries | Balances recall efficiency and result completeness, and avoids redundant data interfering with defect analysis |
defaultToken | Set from representative sample documents | defaultToken is the default token count for direct text chunking; tune it on representative source documents. See the v4.17.2 model-field reference. It is not a vector dimension, embedding-model input cap, or retrieval/LLM context. |
Three common mistakes
- Issue: Vector recall results include environmental monitoring data from non-current batches. Cause: No partition filtering of index data by batch number, leading to incorrect recall of cross-batch data.
- Issue: Ollama-deployed BGE-M3 model fails to connect. Console returns call timeout error. Cause: No correct port mapping configured for the ollama vector model, or calls are initiated before the vector model service fully starts.
- Issue: Tencent Hunyuan vector model returns 401 unauthorized error after configuration. Cause: No correct API key and region parameters entered in the FastGPT provider configuration.
How to confirm proper configuration
- Import a single batch record sample. Verify data field integrity after vector ingestion, confirm that batch numbers and critical parameters are correctly encoded.
- Submit a search request for a known abnormal batch. Verify that recall results include process and environmental data from the same batch. Adjust the similarity threshold to a range that meets business requirements.
- View the index monitoring panel in FastGPT v4.17.0. Confirm that incremental data synchronization frequency matches the configured update schedule.
- Test the vector model call chain. Confirm that the returned embedding vector dimensions match the configured vector model dimensions.
The configuration values listed above are common starting points. Adjust them based on the reader's own specific samples and use cases.