Data Characteristics
Pharmacovigilance data primarily originates from reports submitted by Marketing Authorization Holders (MAH). These include Individual Case Safety Reports (ICSRs), Periodic Safety Update Reports (PSURs), Risk Management Plans (RMPs), product inserts, and post-marketing study reports. Documents typically exist as PDFs, Word files, or structured text (e.g., XML). Update frequency is high; for example, ICSRs might update daily, while PSURs have semi-annual, annual, or triennial cycles. Document structures are complex, containing extensive medical terminology, clinical trial data, adverse event classification codes (e.g., MedDRA), drug information (ATC codes, CAS numbers), and regulatory content. Data fields are numerous. For instance, ICSRs include patient demographics, drug information, adverse event descriptions, and management actions. Units involve dosage (mg, g), frequency (times/day), and time (days, months, years).
Constraints on Vector Models and Indexing
The complexity and high update frequency of pharmacovigilance documents impose specific requirements on vector model and indexing construction. First, documents contain extensive specialized terminology and abbreviations. Vector models require strong domain-specific semantic understanding to differentiate similar terms in varying contexts. Second, documents combine structured and unstructured data. Indexing strategies must parse and embed different data forms. For example, tabular data might require special processing to preserve structural information. Third, high update frequency demands efficient incremental update capabilities from the indexing system. This ensures new data is quickly incorporated into the knowledge base and queries remain current. Finally, regulatory compliance requires accurate and complete information recall. Critical information omissions or misunderstandings are not permissible. This directly impacts the rigor of chunking strategies and similarity calculations.
Configuration Guidelines
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
Chunk size | 800–1200 characters | Balances semantic completeness and vector model processing efficiency; avoids splitting critical information. |
Chunk Overlap Length | 150–250 characters | Ensures contextual continuity; improves the likelihood of recalling information across chunks. |
Vector Model | bge-large-zh-1.5 | A pre-trained model for Chinese medical texts with strong semantic understanding capabilities. |
Recall count | Top 5–8 entries | Balances recall efficiency and relevance; ensures coverage of potentially relevant information. |
Similarity threshold | Calibrate by measurement | Adjusts based on specific business requirements for accuracy and recall. |
PARSE_FILE_TIMEOUT_SECONDS | 300 seconds | Accommodates parsing time for large PDF documents; prevents indexing failures due to timeouts. |
Common Pitfalls
- Document parsing remains in an "indexing" state for extended periods. This occurs when some pharmacovigilance report files are too large or complex, leading to file parsing timeouts.
- During knowledge Q&A, the system fails to accurately recall detailed descriptions of specific drug adverse reactions. This happens when the vector model does not fully understand the deep semantics of medical professional terms, or the chunking strategy splits critical information.
- When querying across multiple knowledge bases, recall results lack prioritization. This is because weights or order between knowledge bases are not explicitly configured in the query logic.
Validation Steps
- Select a batch of test documents containing typical pharmacovigilance information. Check if their indexing status is normal, with no timeouts or failures recorded.
- Use query statements containing specific medical terms and adverse events. Verify the system's ability to accurately recall relevant document snippets. Compare recall results with expected answers.
- Simulate high-frequency data update scenarios. Observe the efficiency of incremental indexing and the query availability of new data. Ensure knowledge base timeliness.
The values provided are common starting points and should be measured against the reader's own samples.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-09-21.