Data Characteristics
Solid tumor treatment protocols and SOP documents originate from national health commissions, drug administrations, local medical insurance bureaus, hospital ethics committees, and internal management guidelines of oncology hospitals. These documents update quarterly or semi-annually, covering new drug approvals, clinical guideline revisions, and medical insurance policy adjustments. Documents are primarily PDFs, containing numerous tables, images, flowcharts, and nested directories. Text content is highly specialized, involving tumor types (e.g., lung cancer, breast cancer), staging (e.g., TNM staging), treatment plans (e.g., chemotherapy, radiotherapy, targeted therapy), drug names (e.g., PD-1 inhibitors), dosage units (e.g., mg/kg), administration routes, and side effect management. Disease staging and treatment pathways often feature multi-level nesting.
Constraints from these Characteristics on Document Parsing and Chunking
The characteristics of solid tumor protocol documents impose specific requirements on document parsing and chunking. First, frequent updates necessitate efficient incremental parsing capabilities to quickly synchronize the latest policies and guidelines. Second, the presence of numerous tables and flowcharts means that text extraction alone is insufficient to capture all information, requiring image recognition and structured table parsing capabilities. Multi-level descriptions of disease staging and treatment pathways demand a chunking strategy that recognizes and preserves hierarchical context, avoiding information loss from flattening. Furthermore, common specialized terms and abbreviations in documents require chunking to maintain term integrity, preventing splitting within critical phrases. Precise numerical information like drug dosages requires the parser to accurately identify units and their associations.
Configuration Guidelines
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
UPLOAD_FILE_MAX_SIZE | 500 MB | Solid tumor guidelines often contain many charts and figures, leading to larger file sizes. |
Chunk size (Chunk Length) | 800–1200 characters (characters) | Balances contextual completeness and retrieval efficiency, avoiding splitting critical treatment pathway descriptions. |
Chunk Overlap | 100–150 characters (characters) | Ensures contextual continuity at chunk boundaries, especially for specialized terms and multi-level descriptions. |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds (seconds) | Processing charts and tables in large PDF documents can be time-consuming. |
Enable Image OCR | True | Solid tumor documents contain many flowcharts and table images; OCR is needed to extract text. |
Table Parsing Mode (Table Parsing Mode) | Structured | Identifies and extracts table rows, columns, and cell content, preserving semantic structure. |
Three Common Mistakes
- Parsing results show garbled text or missing critical information because OCR was not enabled or improperly configured, leading to unrecognised text in images.
- Q&A results misunderstand disease staging and treatment plans because chunk length was too short or hierarchical relationships were not considered, causing important context to be severed.
- Processing large PDF documents frequently times out or fails because file size limits or parsing timeout settings are too low to handle complex documents.
How to Verify Configuration
- Select a solid tumor SOP document containing complex tables and flowcharts, upload it, and review the parsed raw text to verify if it completely includes information from images and tables.
- Choose paragraphs from the document that describe disease staging and treatment pathways. Observe if the chunking results maintain logical continuity and hierarchical relationships, without splitting critical information.
- Search the knowledge base for unique specialized terms or drug names from the document. Check if the recalled chunks are accurate and have complete context.
- Test with solid tumor documents of varying sizes and complexities. Observe if parsing times complete within the set timeout threshold and check for error logs.
Note: The values provided are common starting points. Measure them against your own samples.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-09-21.