What the data for this category looks like
Data sources for this category include registered process documents, raw material batch ledgers, DCS system operation data, weighing system records, LIMS test reports, chromatography data system files, and QMS deviation and change documents. Data updates occur for every production batch and process step. Daily documents include batch production records, material balance sheets, reaction and purification logs, cleaning validation records, impurity profile reports, and deviation and change application documents. Fields include batch number, reaction temperature, yield value, impurity content, solvent residue amount, and more. The documents carry large volumes of pharmaceutical and chemical professional terms and precise process parameters.
What Constraints These Characteristics Impose on Vector Models and Indexes
This category’s data updates frequently per production batch and process step. Vector indexes must support incremental embedding to avoid redundant processing from full re-embedding. Documents contain large volumes of pharmaceutical and chemical professional terms and structured process parameters. Select a domain-adapted embedding model to ensure accurate recall of professional text. Single-batch document lengths vary widely, from short cleaning records to complete impurity profile reports. Adjust the index splitting strategy to accommodate different text lengths. Data links to batch metadata. Indexes must support filtering retrieval by batch dimension to improve the accuracy of ledger summary.
This configuration is adapted for FastGPT v4.17.0.
How to Configure the Settings
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
| Chunk Length | 800–1200 characters | Adapts to the typical text length of batch production records and impurity profile reports, avoids truncation of professional terms |
| Overlap Length | 100–150 characters | Retains cross-chunk reaction step and impurity association information, improves coherence of recall results |
defaultToken | 4096 token | Matches the average token density of pharmaceutical professional text, adapts to splitting requirements for most documents |
maxToken | 8192 token | Covers the full text of complete reaction and purification records, avoids truncation of key process parameters |
| Similarity Threshold | Calibrated based on actual testing | Adapts to the similarity matching accuracy of professional text such as impurity profiles and deviation records |
| Single Search Limit | Top 10 results | Balances recall coverage and retrieval efficiency, adapts to rapid summary of multi-batch ledgers |
Three Common Configuration Errors
- Phenomenon: After replacing the embedding model, index recall results for old ledger documents become abnormal, and batch re-embedding takes too long. Cause: No incremental index update strategy is configured, triggering full re-embedding of all historical documents.
- Phenomenon: Manually inserted batch production record documents show normal index status, then disappear automatically after several hours. Cause: Index persistent storage is not enabled, or batch metadata is not correctly associated with index entries.
- Phenomenon: After connecting the qwen3-embedding-8b model, the file status of impurity profile reports consistently shows "indexing". Cause: The input token limit of the embedding model does not match the document chunk length, leading to processing timeout.
How to Verify Successful Configuration
- Upload a single complete impurity profile report, and confirm that index generation time meets expectations.
- After replacing the embedding model, manually trigger single-document re-embedding, and verify consistency of recall results for multilingual text.
- View the index storage directory, and confirm that newly generated index files synchronize with document updates.
- Retrieve process parameters for a specific batch, and verify that recall results include metadata information for the corresponding batch.
The values provided are common starting points and should be measured against the reader's own samples.