Data Characteristics
Process validation pharmacovigilance data originates from clinical trial reports, production batch records, quality control reports, and early post-market surveillance data. This data exists as both structured and unstructured documents. Examples include PDF validation protocols, CSV batch analysis results, XML adverse event reports (like E2B files), and Word or text investigation reports. Data updates frequently during validation, potentially daily or weekly for batch data. Clinical reports update at specific milestones or when anomalies are found. Fields include batch number, production date, product number, subject ID, adverse event description, severity, occurrence date, treatment measures, related drug batch numbers, and test indicators (e.g., content, purity, dissolution) with corresponding values and units.
Constraints Imposed by Data Characteristics on Deployment and Upgrade
The diversity and high update frequency of process validation data place specific demands on FastGPT's deployment and upgrade. First, FastGPT must support parsing and extraction from various file formats, especially industry-standard formats like E2B. Second, frequent data updates require FastGPT's knowledge base index to have an efficient incremental update mechanism. This ensures new data is promptly included in the retrieval scope, avoiding lengthy full rebuilds. Accurate matching and retrieval capabilities for key fields like batch number and subject ID are crucial. Appropriate recall strategies must be configured. Furthermore, a large volume of numerical test indicator data means vectorization must standardize numerical ranges and units. This prevents dimensional differences from affecting similarity calculations. Non-structured text adverse event descriptions require more refined text segmentation and entity recognition capabilities to improve question-answering accuracy.
Configuration Guidelines
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
UPLOAD_FILE_MAX_SIZE | 500 MB | Accommodates large clinical trial reports or batch record files. |
maxContext | 800–1200 characters | Balances understanding lengthy validation reports with model processing efficiency. |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Ensures complex PDF or XML files have sufficient time to parse. |
Chunk size (Segment Length) | 300–450 words | Balances completeness of adverse event descriptions with vector retrieval precision. |
Recall count (Recall Count) | Top 8 | Increases the probability of recalling relevant information from numerous batch data and adverse event reports. |
Similarity threshold (Similarity Threshold) | 0.78 | Ensures recalled results are highly relevant to process validation or adverse event queries. |
Common Pitfalls
- New batch records or adverse event reports are not retrieved after a knowledge base update. This occurs because the knowledge base index did not trigger an incremental update or the update failed, preventing new data from being included in the vector database.
- Queries for adverse event information for a specific batch number return empty or irrelevant results. This happens because the batch number field was not correctly extracted during document parsing, or field mapping configuration was incorrect, leading to inaccurate retrieval.
- After deploying FastGPT, model configurations and applications are lost after a device restart. This indicates that persistent storage for the container was not configured correctly, resulting in data volumes not being mounted or data not being saved to the host during container restarts.
Verification Steps
- Upload a test document containing new batch data and adverse event information. Check the knowledge base index status to confirm the document was successfully parsed and vectorized.
- Use a query with a specific batch number and adverse event keywords. Verify that FastGPT accurately recalls relevant document snippets and data.
- Check the
docker logs <container_id>output. Confirm there are no critical error messages, especially regarding file parsing, vectorization, or database connections. - Restart the FastGPT service. Log in to the system and verify that previously configured models, applications, and knowledge bases are fully retained, confirming persistent storage configuration is effective.
Note: The values provided are common starting points. Always measure against your own samples to determine optimal settings.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-09-21.