What data looks like for this category
Core data for active pharmaceutical ingredient and pharmaceutical intermediate production originates from registered process documents, raw material batch ledgers, DCS systems, weighing systems, LIMS, chromatography data systems, and QMS. Daily generated documents include batch production records, material balance sheets, reaction and purification step records, cleaning validation records, impurity profile reports, and deviation and change documents. Data updates occur per production batch and each process step, with fields including batch identification, reaction parameters, yield data, impurity content, solvent residue levels, deviation causes, and rectification measures. Data formats across different sources have heterogeneity.
What constraints these characteristics impose on vector models and indexing
Multi-source heterogeneous data sources require indexes to support unified embedding and cross-system data retrieval. High-frequency updates per batch and process step require indexes to support incremental update configurations, avoiding computing resource waste from full index reconstruction. Specialized fields such as impurity profiles and reaction steps in professional documents require vector models to accurately capture professional term associations in the pharmaceutical and chemical industries. The batch-level update rhythm and strict compliance requirements mean that index recall accuracy directly impacts the efficiency of subsequent process review and exception handling. Irrelevant batch data must also be prevented from being incorrectly recalled.
How to set configurations
This guide applies to FastGPT v4.17.0.
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
| Chunk Length | 800–1200 characters | Matches the single-page content length of batch production records and impurity profile reports, avoiding splitting specialized reaction steps or impurity association paragraphs |
| Overlap Length | 100–150 characters | Preserves contextual continuity for reaction steps and impurity tracing, preventing chunking from breaking professional logical associations |
| Similarity Threshold | 0.75–0.85, calibrated via actual testing | Filters low-relevance historical batch data, accurately recalling documents matching current exceptions or process requirements |
| Single Search Limit | Top 8 entries | Controls context window usage, adapting to the information density requirements of single-batch process analysis |
maxToken | 4096 token | Meets long-text embedding requirements for professional documents, matching the input limit of most pharmaceutical industry vector models |
defaultToken | Set from representative sample documents | defaultToken is the default token count for direct text chunking; tune it on representative source documents. See the v4.17.2 model-field reference. It is not a vector dimension, embedding-model input cap, or retrieval/LLM context. |
Three common mistakes
- Phenomenon: Search results include auxiliary material data that is not associated with the current production batch. Cause: No data range filtering rules are configured, and non-core documents such as auxiliary ledgers are included in the index scope.
- Phenomenon: Different fields of the same batch production record (such as reaction parameters and impurity profiles) are merged into a single vector, making it impossible to accurately match process steps and impurity issues during recall. Cause: Field-level vectorization configuration is not enabled, or independent chunking rules are not set for specialized fields.
- Phenomenon: After uploading a vector file generated by a local model, the server-side search returns empty results. Cause: The embedding dimensions of the local model and server-side model are not aligned, or the vector file format does not match the storage requirements of FastGPT.
How to confirm proper configuration
- Upload a single complete batch production record, check the embedding task logs to confirm that the chunk length and overlap length match the configured values.
- Initiate a search for a specific impurity type, verify that the recalled results include content related to the target impurity, and adjust the similarity threshold to meet business accuracy requirements.
- Add a new deviation record and upload it, check the index update status to confirm that the incremental update process is triggered normally.
- Switch between different vector models, verify that the embedded vector dimensions are consistent, and confirm that the model compatibility configuration takes effect.
The values provided here are common starting points, and should be evaluated against the reader's own samples.