Data Characteristics
Retail chain pharmacovigilance data primarily originates from daily operations. This includes sales records, customer inquiries, medication feedback, and internal or external adverse event reports. Data updates frequently. Sales records generate in real-time, and adverse event reports submit as events occur. Document structures typically include standardized adverse event report forms, customer medication inquiry records (often free-text or mixed structured fields), and drug batch traceability information. Fields may include drug generic name, batch number, manufacturer, anonymized patient basic information, adverse reaction description, occurrence time, and handling measures. Units often involve milligrams (mg), grams (g), milliliters (ml) for dosage, and dates and specific times for time units.
Constraints Imposed by Data Characteristics on Document Parsing and Chunking
The high update frequency of retail chain pharmacy data requires an efficient, real-time document parsing system. This ensures new data integrates quickly into pharmacovigilance analysis. Free-text content in adverse reaction descriptions presents challenges for parsing accuracy and robustness due to its variability and unstructured nature. This is particularly true for identifying key entities like drug names, symptoms, and event times. Anonymized patient information requires the parsing process to effectively identify and protect sensitive data. Drug batch numbers and manufacturer information within documents are crucial for subsequent drug traceability and risk assessment. Parsing must ensure precise extraction of these fields. The integration of multi-source data, such as sales records and adverse event reports, also requires the parser to handle different document formats and structures and establish effective correlations.
Configuration Settings
| Configuration Item | Suggested Value | Rationale |
|---|---|---|
Chunk size (Chunk Length) | 500-800 characters | Balances the completeness of adverse reaction descriptions with relevance during recall. |
Chunk Overlap Length (Overlap Length) | 100-150 characters | Ensures contextual continuity and prevents critical information from being cut off. |
OCR Language | zh | Primarily processes Chinese documents, improving recognition accuracy. |
PARSE_FILE_TIMEOUT_SECONDS | 300 seconds | Accommodates the parsing time for large PDF reports or documents with many images. |
maxContext | 4000 tokens | Adapts to longer free-text descriptions in adverse reaction reports. |
Recall count (Recall Count) | Top 8 | Controls the computational load for subsequent processing while ensuring recall rate. |
Common Pitfalls
ocr errorin logs and file parsing failures usually indicate poor image quality or handwritten fonts in the document, preventing the OCR engine from recognition.- File parsing functionality failing while normal chat functions work may be due to an internal error or resource limitation in a specific model or service (e.g.,
claude 3.7) that the file parsing relies on. - Errors when parsing JSON-formatted internet search results often occur because the returned JSON structure does not match expectations, preventing the parser from correctly matching fields.
Verification Steps
- Upload typical adverse event report PDFs and image files. Check if the parsed text content is complete and free of garbled characters.
- Perform keyword searches on the parsed chunks. Confirm that key entities (e.g., drug names, symptoms, times) are accurately identified and located.
- Compare original documents with parsing results. Check if the extraction of key structured fields (e.g., batch number, manufacturer) is correct and verify data type consistency.
- Simulate user questions about specific adverse event incidents. Observe if the retrieved chunks are highly relevant to the question and contain supporting original information.
The values provided are common starting points and should be measured against your own samples.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-09-21.