What the data for this category looks like
The typical entities in this category are vaccine, antibody, blood product, and sterile injectable manufacturing enterprises. Daily documents include cell bank or seed lot records, culture and purification records, aseptic filling records, environmental monitoring data, media fill simulation results, sterility test reports, and batch release documents. Data sources cover cell bank and seed lot systems, process analysis platforms, cleanroom monitoring systems, MES, LIMS, and equipment integrity testing tools. Update cycles include per-batch production cycles, real-time monitoring data synchronization, and updates per fixed monitoring plans.
What constraints these characteristics impose on vector models and indexing
Data sources for this category are scattered and have significant format differences. Fields and units of records exported from different systems are inconsistent, requiring standardized processing before input to the vector model. High update frequency requires the indexing link to support incremental synchronization, avoiding full re-generation of the vector database. Long professional text records account for a high proportion, and chunking must avoid splitting key professional terms and associated parameters. At the same time, bid documents require rapid recall of content related to compliance and process control, which imposes clear constraints on the recall accuracy of the index.
How to configure the settings
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
| Chunk Length | 800–1200 characters | Adapts to long professional text records of this category, avoids splitting key parameters and terms |
| Overlap Length | 100–150 characters | Preserves contextual association between chunks, prevents professional logic from being truncated |
maxToken | 8192 | Adapts to the input upper limit of mainstream open-source embedding models, compatible with vector generation after long document chunking |
defaultToken | 512 | Used for vector generation of short text fields, balances computing resources and recall accuracy |
| Similarity Threshold | 0.72–0.80, calibrated via actual testing | Filters low-correlation results, meets the high recall requirements for compliance documents |
| Single Search Limit | Top 6 results | Controls the number of returned results, avoids information overload during bid document preparation |
Three common mistakes
- Scenario: The system only supports manual vector model updates triggered per single file, and cannot perform batch retraining on batch-uploaded compliance records. Cause: Batch vector update configuration is not enabled, and the system by default only responds to vector generation requests after single file upload.
- Scenario: After uploading a single batch release document, the interface shows 8 chunks, but the actual number of chunks becomes 13 after indexing is complete, with duplicate chunks. Cause: The chunk overlap length is set beyond the proportion of a single chunk content, and the chunk deduplication function is not enabled, resulting in overlapping content between adjacent chunks and repeated indexing.
- Scenario: After upgrading from an older version to FastGPT v4.17.0, original aseptic monitoring records exported from the LIMS system cannot be recalled. Cause: The default
defaultTokenparameter of the new version's embedding model has been adjusted, resulting in a mismatch with the original vector generation logic for short fields, preventing correct index generation.
How to confirm the configuration is correct
- Upload a typical batch release document, check the chunk preview interface, confirm that the number of chunks matches expectations, and there are no duplicate chunks.
- Trigger a batch vector update task, check the task logs for incremental synchronization records, confirm that incremental synchronization was performed, and no full retraining was triggered.
- Enter keywords from an aseptic filling record, check the correlation of the recalled results, and adjust the similarity threshold to meet business requirements.
- Import a medium fill simulation record in Excel format, verify that the indexed content after vectorization matches the original document fields.
The values provided are common starting points and should be evaluated against the reader's own samples.