Data Characteristics
Pharmacovigilance data originates from safety reports, risk management plans, and drug label revision records. These documents are generated during drug development, clinical trials, and post-market surveillance. Documents update frequently, especially post-market, as adverse event reports accumulate and analysis occurs. Document structures often include extensive semi-structured and unstructured text, such as adverse event descriptions, medical terminology, dosage information, and patient characteristics. Tabular data is also common. Fields and units vary in standardization and include medical terms (e.g., ICD-10 codes), drug names, active ingredients, dosage units (mg, ml), and time units (days, months, years).
Constraints on Document Parsing and Chunking
High update frequency in pharmacovigilance documents requires efficient incremental updates and version management for timely knowledge bases. The mix of semi-structured, unstructured content, and tabular data challenges parser accuracy in identification and extraction. Specialized and diverse medical terminology, along with mixed dosage and time units, demand that document chunking maintains contextual integrity. This prevents loss of critical information from overly fine-grained segmentation. Documents may contain extensive clinical details and patient privacy information, requiring sensitive information identification and anonymization during parsing to ensure compliance. Document length is often substantial, requiring high processing capability and stability from parsing services.
Configuration Settings
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
UPLOAD_FILE_MAX_SIZE | 500 MB | Pharmacovigilance documents may contain many images and charts, leading to large file sizes. Sufficient upload space is necessary. |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Processing multi-hundred-page PDF documents takes significant time. Extending the timeout prevents parsing interruptions. |
Chunk size (Chunk Length) | 800–1200 characters | This balances contextual integrity for medical terminology with retrieval efficiency, avoiding overly long or short chunks. |
Chunk Overlap Length (Chunk Overlap Length) | 100 characters | This ensures information continuity at chunk boundaries, improving recall accuracy. |
Max Concurrent Parse Tasks | Benchmark against actual usage | Adjust this based on server resources and expected file upload frequency to prevent resource exhaustion. |
Parsing Engine Type | pdf-marker or doc2x | Choose a parsing engine with broader support for various pharmacovigilance document formats like PDF and Word. |
Common Pitfalls
- A
504 Gateway Timeouterror occurs when parsing large PDF files. This usually means the parsing time exceeded the default timeout settings of the gateway or application server. - Loss of critical medical terms or dosage information after document parsing indicates the chunk length was set too short. This leads to truncation of key information or incomplete context.
- A
Cannot read properties of undefinederror appears on the page after a locally deployed parsing service starts. This may be due to missing essential environment variables in the configuration file or incorrect installation of service dependencies.
Verification Steps
- Upload a typical large pharmacovigilance document (e.g., a PDF over 200 pages). Observe if the parsing task completes successfully without timeout errors.
- Randomly select parsed document segments. Check for complete medical terminology, drug dosages, and time units, ensuring contextual coherence.
- In the knowledge base, search for unique, long safety descriptions from the document. Verify that recall results accurately hit complete segments containing these descriptions.
- Review parsing service logs. Confirm the absence of numerous parsing failures or resource warning messages.
The values provided are common starting points. Measure them against your own samples.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-09-21.