What This Category of Data Looks Like
Data related to pharmaceutical dosage form manufacturing comes from approved formulation processes, material batch information, MES electronic batch records, equipment and environmental monitoring data, LIMS systems, and quality management systems. Daily documents include master production files, batch production records, batch packaging records, cleaning records, inspection records, deviation/CAPA/change documents, and batch release review forms. Data updates occur per production batch, for each key process step, and in accordance with environmental and equipment monitoring plans. Fields include batch identification, material parameters, process execution data, inspection results, and other related fields.
Constraints on Vector Models and Indexing
This category’s data is updated independently per batch and closely linked. Indexes must support batch-level isolation and associated retrieval to prevent cross-batch data mixing. Documents contain both structured fields and long-text process descriptions. Chunking operations must avoid damaging field integrity, or retrieval matching accuracy will decline. The high update frequency requires indexes to support incremental updates, eliminating resource consumption from full index reconstruction. Bid documents require aggregation of compliance data across multiple batches. Vector indexes must quickly associate corresponding batch information across different documents to support rapid review and quotation preparation.
How to Set Configurations
| Configuration Item | Suggested Value | Rationale |
|---|---|---|
| Chunk Length | 800–1200 characters | Adapt to the length of a single process or inspection record in batch records, avoid destroying field integrity |
| Overlap Length | 100–200 characters | Connect cross-block process step descriptions, ensure complete contextual association |
defaultToken | 4096 token | Adapt to the text length of core documents for a single batch, balance vector generation quality and call costs |
maxToken | 8192 token | Support long-text processing requirements for aggregated bid documents, avoid truncating key compliance information |
| Single Retrieval Limit | Top 5 entries | Focus on the most relevant batch data, avoid redundant information interfering with bid quotation preparation |
| Index Update Trigger Rule | Triggered after a single batch is completed | Match the rhythm of batch-based data updates, reduce invalid index operations |
Three Common Mistakes
- The knowledge base upload status remains in indexing for a long time. The cause is failure to adapt to batch-triggered incremental indexing logic, leading to thread resource exhaustion when uploading multiple batches of data in bulk.
- An error indicating rate limit exceeded occurs during vectorization. The cause is failure to adjust the concurrent request configuration corresponding to
defaultToken, causing vector model call request frequency to exceed limits. - Text chunks are lost when chunk length is set to 3000 characters. The cause is the presence of long fields such as complete process steps in batch records, with chunking truncating field boundaries leading to parsing failures.
How to Confirm Proper Configuration
- Upload the complete batch production record for a single batch. Check whether chunked text blocks retain complete material weighing and process parameter fields, with no truncation or loss.
- View vector model call logs in FastGPT v4.17.0. Confirm that request rates do not trigger limit exceeded errors, and that the
maxTokensetting adapts to the maximum length of uploaded documents. - Initiate a retrieval request related to bid documents. Verify that retrieval results include relevant data such as inspection records and deviation handling records for the corresponding batch.
- Simulate bulk upload of 3–5 batches of documents. Confirm that index status can be updated within a reasonable time frame with no long-term stagnation.
The values provided are common starting points and should be measured against the reader's own samples.