What This Category of Data Looks Like
The main entities in this category are vaccine, antibody, blood product, and aseptic injectable manufacturing enterprises. Daily data covers cell bank and seed lot records, culture and purification records, aseptic filling records, clean zone environmental monitoring data, media fill simulation results, sterility test reports, and batch release documents. Data sources include cell bank management systems, process analysis instruments, clean zone monitoring equipment, MES manufacturing execution systems, LIMS laboratory information management systems, and equipment integrity testing tools. Data update rhythms fall into three categories: per production batch, real-time monitoring, and fixed periodic planning. Individual records include fields such as batch number, operation time, process parameter values, test results, and operator information. Some documents include multi-dimensional structured tables and unstructured descriptive text.
What Constraints These Characteristics Impose on Vector Models and Indexing
Data sources for this category are scattered and have mixed formats. These formats include structured tables, instrument-derived raw data, and unstructured process descriptions. The indexing system must support unified chunking and metadata binding for multi-source heterogeneous data. Data is strongly bound to production batches. The chunking process must retain batch numbers as core associated metadata to avoid cross-batch retrieval confusion. Data update frequencies vary widely. Real-time monitoring data requires incremental indexing support. Batch release documents require full reindexing support. Data anomalies may trigger serious safety risks. The vectorization and indexing process must ensure no abnormal chunk generation, to avoid low-quality vectors affecting retrieval results.
How to Set Configurations
| Configuration Item | Recommended Setting | Rationale for This Setting |
|---|---|---|
| Chunk Length | 800–1200 characters | Aligns with the average text length of single process records for this category, avoids splitting the context of critical process parameters |
| Chunk Overlap Length | 100–150 characters | Retains cross-chunk batch numbers and key operation node information, ensures contextual coherence during retrieval |
defaultToken | Set from representative sample documents | defaultToken is the default token count for direct text chunking; tune it on representative source documents. See the v4.17.2 model-field reference. It is not a vector dimension, embedding-model input cap, or retrieval/LLM context. |
maxToken | Set to the selected embedding model/service maximum supported input tokens | maxToken is the input-token maximum supported by the selected embedding model/service; verify it in that provider configuration. See the v4.17.2 model-field reference. It is not retrieval/LLM context or a FastGPT-wide fixed cap. |
| Single Search Limit | Top 5 results | Matches the number of core process records that batch release reports need to associate with, filters redundant retrieval results |
| Similarity Threshold | 0.75–0.85, calibrated via actual testing | Filters low-match irrelevant process records, ensures retrieval results are strongly associated with the target production batch |
Three Common Mistakes
- When uploading batch release documents larger than 10MB, vectorization exceptions occur after more than 1000 chunks are generated. Partial recovery occurs after repeated retries. The root cause is that batch metadata is not bound during chunking, leading to broken contextual association for some chunks, which triggers embedding model format verification failures.
- After multiple files are uploaded in bulk and the server crashes and restarts, the knowledge base status remains stuck at "not ready" with no automatic recovery action. The root cause is that no state persistence mechanism for bulk indexing is configured. Only single-file manual index triggering is supported, and no association is made with upload task execution records.
- After updating to FastGPT v4.17.0, previously indexed knowledge bases cannot return retrieval results. The root cause is that the embedding model’s vector dimension changes after the version upgrade, and the original vector library for indexing is not adapted to the new dimension, causing matching logic to fail.
How to Confirm Proper Configuration
- Upload a single cell bank seed lot record document. Check if the chunked metadata includes batch number and operation time fields, to verify that the chunking logic correctly binds business identifiers.
- Initiate a release report retrieval request for a specific production batch. Confirm that the number of returned results matches the preset single search limit, and that similarity scores fall within the set range.
- After restarting the server, check if the knowledge base list status automatically changes to "ready". Normal retrieval can proceed without manually triggering index operations.
- Call the embedding model interface, upload a text fragment of a culture and purification record, verify that the vector output has no abnormal error reports, and confirm that the vectorization process is working correctly.
The values provided are common starting points and should be measured against the reader's own samples.