Data Characteristics in this Category
Dermatology pharmacovigilance data comes primarily from clinical trial reports, real-world studies, case reports, patient feedback, and drug inserts. This data updates frequently. Adverse event reports emerge continuously, especially after new drugs launch. Document structures vary. Examples include structured Case Report Forms (CRFs), semi-structured medical texts (e.g., handwritten doctor's notes, clinical records), and unstructured free text (e.g., patient self-reports, social media discussions). Fields and units are highly specific. For instance, skin lesion descriptions involve morphological features (macules, papules, vesicles), size (millimeters, centimeters), color (erythema, purpura), distribution (localized, generalized), and specific medical terminology and scoring scales (e.g., SCORAD score, EASI score).
Constraints Imposed by these Characteristics on "Document Parsing and Chunking"
Dermatology data originates from diverse sources with varying degrees of structure. This demands robust document parsing, requiring handling of embedded image text and handwriting recognition. High-frequency updates mean the knowledge base needs to support incremental updates and rapid indexing to capture the latest adverse event information. Documents contain extensive medical jargon, abbreviations, and multilingual content. This challenges the tokenizer's vocabulary coverage and multilingual processing capabilities. Highly detailed skin lesion descriptions and the presence of scoring scales require chunking strategies that preserve descriptive integrity, avoid truncation at critical information points, and identify and extract numerical data, ensuring correct unit association.
Configuration Settings
| Configuration Item | Recommended Value | Rationale for this Value |
|---|---|---|
chunk_size | 800–1200 characters | Balances the completeness of skin lesion descriptions with retrieval efficiency, preventing excessive fragmentation and loss of context. |
overlap_size | 100–200 characters | Ensures sufficient contextual overlap between adjacent chunks, improving retrieval coherence. |
max_tokens | 4000 tokens | Accommodates the verbose nature of medical texts, ensuring a single request can process longer medical records or reports. |
parse_image_text | enabled | Dermatology documents often contain skin lesion images and captions; text information within images needs to be identified and extracted. |
custom_stopwords | add medical jargon | Excludes common medical stopwords that do not contribute to RAG, improving semantic matching accuracy. |
timeout_seconds | 600 seconds | Provides sufficient parsing time for large PDF reports or OCR recognition tasks, preventing interruptions. |
Three Common Mistakes
- Uploading large PDF documents results in low question-answering accuracy. Logs show
chunk_sizeis too small or chunk content is missing. This occurs because the default chunking strategy fails to effectively handle complex multi-column layouts or mixed text-and-image dermatology reports, leading to truncation of critical information. - Database connection configuration is correct, but system-generated SQL queries error out. This happens because parsing results include punctuation or non-standard characters, causing SQL syntax errors.
- Uploading Excel spreadsheets results in the system failing to understand table content, and question-answering results are empty. This occurs because
table_parsing_strategyis not specified or an unsuitable automatic segmentation strategy is chosen for multi-column medical data tables, disrupting the logical relationships between rows.
How to Confirm Proper Configuration
- Upload a typical case report or drug insert. Check the parsed chunk content to confirm that key symptoms, diagnoses, treatment plans, and adverse event descriptions are complete and logically coherent.
- Compare the original document with the parsing results. Verify that text from images and table data are accurately extracted and included in the chunks, paying special attention to morphological details, size, and color of skin lesion descriptions.
- Select multiple random chunks. Perform keyword searches and question-answering tests. Evaluate whether retrieval results contain the expected information and check if the retrieved content has sufficient context.
The values provided are common starting points and should be measured against the reader's own samples.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-09-21.