Data Characteristics
Neurodegenerative disease pharmacovigilance data originates from clinical trial reports, real-world evidence (RWE) studies, case reports, medical literature, and regulatory adverse event databases. This data updates frequently, especially post-market, with new adverse reaction reports continuously arriving. Document structures vary, including unstructured free-text descriptions, semi-structured case report forms (e.g., CIOMS I forms), and structured adverse drug event (ADE) database records. Field and unit specificities include descriptions of disease progression scales (e.g., MMSE, ADAS-Cog scores), specific neuroimaging metrics (e.g., brain atrophy, amyloid plaque burden), and detailed records of drug dosage, administration route, duration of use, and adverse event onset time and duration. Units commonly include milligrams (mg), milliliters (ml), days (day), and months (month).
Constraints Imposed by Data Characteristics on Vector Models and Indexing
The multimodal nature of neurodegenerative pharmacovigilance data (coexistence of structured and unstructured information) requires vector models to effectively process different information types. Frequent data updates demand high real-time and incremental update capabilities for indexing to ensure retrieval timeliness. Documents contain extensive specialized medical terminology, abbreviations, and disease-specific descriptions, requiring vector models to possess strong semantic understanding to distinguish subtle medical concept differences, such as various cognitive impairment descriptions. Numerical data like disease progression scales and imaging metrics require proper handling of their ordinal or interval properties during vectorization, avoiding information loss from simple discretization. Additionally, narrative text in adverse event reports often includes negative descriptions and expressions of uncertainty, which is crucial for models to capture the probability and severity of events.
Configuration Guidelines
| Configuration Item | Recommended Approach | Rationale |
|---|---|---|
Chunk size (Segment Length) | 512–768 characters (characters) | Balances semantic completeness with vector model processing efficiency, avoiding noise from overly long paragraphs. |
Chunk overlap (Segment Overlap) | 10%–15% | Ensures contextual continuity, especially when critical information spans across segments. |
Vector Model (Vector Model) | text-embedding-v3 or BGE-large-zh | Possesses strong Chinese medical semantic understanding, suitable for complex professional terminology. |
Recall count (Recall Count) | Top 10–20 entries (top 10–20) | Improves initial screening recall rate, covering more potentially relevant information for subsequent re-ranking. |
Similarity threshold (Similarity Threshold) | Calibrate with actual measurements | Determined experimentally based on specific business requirements for accuracy and recall. |
Rerank result count (Re-ranking Return Count) | Top 5 entries (top 5) | Streamlines the final results presented to the user while maintaining accuracy. |
Common Pitfalls
- Symptom: Retrieval results show high relevance scores, but the actual returned content does not match the query intent. Reason: The vector model has biases in understanding medical terminology, failing to accurately capture disease-specific or subtle adverse event semantics.
- Symptom: After data updates, newly entered adverse event reports cannot be retrieved. Reason: The indexing update strategy is misconfigured, failing to implement incremental or real-time indexing, leading to desynchronization between the index and the data source.
- Symptom: After uploading documents, some structured fields (e.g.,
ADE_TERM) cannot be effectively utilized during retrieval. Reason: The document preprocessing stage failed to correctly identify and extract critical structured fields, or failed to effectively integrate them with unstructured text during vectorization.
Validation Steps
- Select a batch of test queries covering different neurodegenerative diseases, adverse reaction types, and severity levels. Check the relevance of the recall results.
- Simulate adding new adverse event reports and immediately perform a retrieval. Verify if the new data is recalled promptly and accurately.
- For queries containing specific disease scale scores (e.g.,
MMSEscore < 20) or neuroimaging metrics, check if the retrieval results accurately reflect the semantics of this numerical information.
The values provided are common starting points and should be measured against the reader's own samples.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-09-21.