Data Characteristics
Patient Assistance Program (PAP) clinical trial pre-screening data comes from two main sources: clinical trial protocol documents and patient medical records. Protocol documents include detailed inclusion/exclusion criteria, study designs, and medication regimens. Patient medical records include diagnostic reports, examination results, medical history, and medication records. These documents are typically in PDF, Word, or structured text formats (e.g., HL7 CDA).
Clinical trial protocol documents are updated infrequently, usually only when a trial starts or undergoes major revisions. Patient medical records, especially for hospitalized or currently treated patients, are updated frequently. They represent a large volume of data in diverse formats, often containing extensive unstructured or semi-structured text. Fields cover medical terminology, laboratory indicators with units, diagnostic codes (e.g., ICD-10), drug names, and dosages. Significant amounts of free-text descriptions are also present.
Constraints from "Document Parsing and Chunking"
The complexity of clinical trial protocol documents requires document parsing to handle intricate hierarchical structures and graphical information, ensuring the completeness of inclusion/exclusion criteria. The diversity of patient medical records demands that the parser handles various medical text formats and accurately extracts key entities such as diagnoses, medications, and test results with their values.
The high update frequency of patient data means that the chunking strategy must support incremental updates and rapid re-indexing. Extensive medical terminology and abbreviations require chunking to maintain contextual integrity, preventing semantic loss due to sentence breaks. The standardization of fields and units requires accurate identification of values and corresponding units after parsing for subsequent numerical comparison and filtering. Free-text descriptions impose higher demands on semantic understanding and entity recognition to avoid missing critical information.
Configuration Guidelines
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
Chunk size (Chunk Length) | 500–800 characters | Balances semantic integrity with vector model processing capabilities, avoiding chunks that are too long or too short. |
Overlap Length | 100–150 characters | Ensures contextual continuity at chunk boundaries, reducing semantic fragmentation. |
Parsing Mode | Smart Parsing | Identifies document structures such as headings, paragraphs, and lists for structured chunking. |
Max File Size | 200 MB | Accommodates large clinical trial protocol documents and medical records containing imaging reports. |
Vector Model (Vector Model) | text-embedding-ada-002 or bge-large-zh-v1.5 | Strong semantic understanding for Chinese medical text, supports longer contexts. |
Entity Recognition Model | Calibrated by actual measurement | High-precision recognition for medical entities like diagnoses, medications, and lab indicators. |
Common Pitfalls
- After uploading large files to the knowledge base, some chunk vectorizations fail, showing abnormal status. This occurs because certain complex tables or embedded objects are not correctly extracted during document parsing, leading to empty or abnormal content chunks, which then affects vectorization.
- Queries about a patient's eligibility for specific inclusion/exclusion criteria return inaccurate or missing results. This happens when medical terms or numerical contexts are truncated during chunking, resulting in incomplete semantics that impact recall or matching accuracy.
- Processing JSON results from database nodes requires writing extensive code for parsing. This is because database query results are highly structured but lack automated parsing capabilities directly linked to the knowledge base chunking mechanism, necessitating manual extraction of key fields.
Verification Steps
- Upload typical clinical trial protocol documents and patient medical records. Check the number and content of chunks in the knowledge base to ensure that key information (e.g., inclusion/exclusion criteria, diagnoses, medications) is fully retained.
- Perform queries based on core inclusion/exclusion criteria. Verify that the recalled chunks accurately cover the relevant information points and assess the relevance of the recalled entries.
- Examine document parsing logs. Confirm that no significant errors or warnings occurred during parsing, especially regarding text extraction from complex tables and images.
- Simulate patient data to test the pre-screening process. Verify that the system can accurately determine a patient's inclusion/exclusion status based on chunk content and provide corresponding reasons.
The values provided are common starting points and should be measured against specific samples.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-09-21.