Data Characteristics
Recombinant protein pharmacovigilance data primarily comes from clinical trial reports, post-market real-world evidence (RWE) analysis reports, case documents submitted to spontaneous adverse event reporting systems, and scientific literature on drug safety. Document update frequencies vary: clinical trial reports typically release after trial completion, RWE reports may update periodically, and spontaneous reporting systems continuously receive new data. Document structures are diverse. Clinical trial reports often have strict section divisions, such as study design, subject characteristics, and adverse event incidence. Spontaneous reports are often free-text descriptions, which may include patient demographics, medication history, adverse event descriptions, management actions, and outcomes. Regarding fields and units, recombinant protein drug dosages often use milligrams (mg), International Units (IU), or specific activity units. Adverse event frequencies are often percentages, and severity may use standardized grading like CTCAE (Common Terminology Criteria for Adverse Events).
Constraints Imposed by These Characteristics on "Document Parsing and Chunking"
The diversity of recombinant protein pharmacovigilance documents challenges document parsing. The structured nature of clinical trial reports requires parsers to effectively identify and extract specific section content, such as adverse event lists and laboratory abnormalities. The free-text nature of spontaneous reports demands stronger Named Entity Recognition (NER) and event extraction capabilities to accurately capture drug names, adverse events, dosages, and temporal relationships from unstructured descriptions. Specialized terminology and units for recombinant proteins require the parsing module to correctly handle expressions like "IU/kg" and "µg/ml" to avoid misinterpretation. Additionally, potential table data in documents, such as adverse event incidence statistics, requires table parsing capabilities to extract structured information. The varying update frequencies across different data sources mean the knowledge base needs to support incremental updates and version management to ensure retrieved information is current.
Configuration Settings
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
Chunk size | 800–1200 characters | Balances contextual completeness and retrieval efficiency, accommodating long text descriptions and table content. |
Chunk Overlap Length | 100 characters | Ensures contextual continuity and prevents critical information from being split by chunk boundaries. |
embeddingModel | text-embedding-ada-002 | Broad applicability with good understanding of biomedical terminology. |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Addresses the time required to parse large clinical trial reports or PDFs containing complex tables. |
maxContext | 4000 token | Ensures sufficient context when processing spontaneous reports that include multiple adverse event descriptions. |
chunk_strategy | By Paragraph And Title chunks，And Table Recognition | Effectively processes chapter titles in structured documents while also handling unstructured paragraphs and table data parsing. |
Three Common Mistakes
- Parsing logs show "OCR recognition failed" or garbled output because the OCR language pack or model is not configured correctly, leading to an inability to recognize specific characters or special symbols in the document.
- Uploading large PDF documents results in a long wait or "request timeout" error because the
PARSE_FILE_TIMEOUT_SECONDSparameter is not adjusted, causing the parsing process to exceed the default time limit. - Retrieval results lack critical drug dosage or adverse event incidence information because table data was not effectively identified and extracted during document chunking, or chunks were too short, leading to context loss.
How to Confirm Correct Configuration
- Select a recombinant protein pharmacovigilance document with various data types (e.g., a clinical trial report), upload it, and review the parsed knowledge base chunk preview. Confirm that key information (e.g., adverse event names, dosages, frequencies) is extracted completely and correctly.
- Perform retrieval tests on the knowledge base. Input queries containing recombinant protein drug names, adverse event symptoms, or specific dosage units. Evaluate the relevance and accuracy of retrieval results, ensuring that returned chunks effectively answer the questions.
- Upload a PDF document containing complex tables. Check if the parsed chunks structure the table content or present it in a readable way, and if key data from the tables can be found through retrieval.
- Simulate uploading multiple large documents concurrently. Observe system response times and parsing success rates to confirm that parameters like
PARSE_FILE_TIMEOUT_SECONDScan handle the actual workload.
Note: The values provided are common starting points. Measure against your own samples for optimal results.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-09-21.