What this category of data looks like
This category of data originates from the synthesis, fermentation, purification, quality assurance, and regulatory support departments of active pharmaceutical ingredient manufacturers. Daily documents include batch production records, material balance records, reaction and purification records, cleaning records, impurity profile reports, deviation and change documents. Data sources include registered processes, raw material batches, DCS systems, weighing systems, LIMS, chromatography data systems, and QMS. Updates occur per batch and per process step. The core data focuses on multi-step chemical reaction routes and intermediate impurity profiles. Fields include reaction parameters, batch numbers, impurity content, yield indicators, and similar metrics. It does not include dosage and packaging-related fields that are standard for general dosage form product categories.
What constraints these characteristics impose on the vector model and indexing link
The data in this category has a high update frequency and strong interconnections. Each production batch corresponds to multiple documents, requiring vector indexes to support fast incremental synchronization and avoid excessive resource usage from full index rebuilding. Documents related to multi-step reactions and impurity profiles contain long-text reaction parameters and complex numerical fields. Vector chunking must preserve cross-chunk contextual connections to prevent loss of process logic after splitting. Multi-source heterogeneous data from different systems requires unified field formatting before vector generation, otherwise retrieval matching deviations will occur. Errors directly relate to batch scrap and compliance risks, requiring the confidence level of retrieval results to match process validation standards, and irrelevant results with low matching scores must not appear.
How to set the configurations
The following configurations are adapted for FastGPT v4.17.0.
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
| Chunk Length | 800–1200 characters | Adapted for long-text content such as batch production records and impurity profile reports, preserves complete process step logic |
| Overlap Length | 100–150 characters | Connects reaction parameters and impurity association information between adjacent chunks, prevents contextual breaks |
defaultToken | 8192 | Adapted for vector generation of long documents, covers the complete content of a single impurity profile report |
maxToken | Set to the selected embedding model/service maximum supported input tokens | maxToken is the input-token maximum supported by the selected embedding model/service; verify it in that provider configuration. See the v4.17.2 model-field reference. It is not retrieval/LLM context or a FastGPT-wide fixed cap. |
| Similarity Threshold | 0.75–0.85, calibrated via actual testing | Filters irrelevant process documents with low matching scores, avoids misleading process judgments |
| Single Search Limit | Top 8 entries | Covers related documents for multi-step reactions, ensures retrieval results include the complete process chain |
Three common mistakes
- Symptom: After executing a full index rebuild operation, the knowledge base retrieval results include a large number of irrelevant historical batch documents. Cause: Incremental index synchronization rules are not configured for each production batch’s data. Full index rebuilding does not filter expired batch files, resulting in redundant data being included in retrieval results.
- Symptom: Calling the interface to create vectors returns a 400 status code with no response body, and the indexing function cannot operate normally. Cause: The long text length of active pharmaceutical ingredient documents is not adapted. The submitted vector generation request exceeds the token limit supported by the model, triggering an interface verification failure.
- Symptom: After local deployment, server read-write resources are exhausted daily, retrieval or file uploads lag, and some upload tasks get stuck at 1 or 2 index group progress. Cause: Chunking parameters are not adjusted for long-text impurity profile reports. Excessively large chunks cause vector generation timeouts. Additionally, concurrent generation counts are not limited, and processing multiple documents simultaneously occupies excessive disk and memory resources.
How to confirm configurations are correct
- Randomly select 3 batch production records and impurity profile reports from different batches, perform retrieval tests, verify that the returned document types and batch matching levels meet requirements, and adjust the similarity threshold to align with process validation standards.
- Upload one complete impurity profile report, check the vector generation progress and chunking results, and confirm that the configured chunk length and overlap length adapt to the document length.
- Simulate a scenario where multiple documents are uploaded simultaneously, monitor server resource usage, and adjust concurrent configurations to keep resource usage within a reasonable range.
- Perform one incremental index synchronization operation, confirm that only newly uploaded batch documents are included in the index, and that a full index rebuild process is not triggered.
The values provided are common starting points and should be evaluated against the reader's own samples.