Document Parsing and Chunking for Medical Information (MI)

Data Characteristics for MI Response Tracing

MI response tracing data primarily consists of reply records from medical information departments to medical inquiries from doctors, patients, or the public. These records are typically unstructured text, including inquiry questions, MI specialist replies, and references to medical literature or product information. Data updates frequently, with new inquiries and replies generated continuously. Document structures often include fields such as date, inquirer type, inquiry content, MI specialist ID, reply content, and reference citations. Reply content may contain extensive medical terminology, drug names, dosage units (e.g., mg, ml), frequencies (e.g., once daily, bid), and may involve mixed languages.

Constraints Imposed by These Characteristics on Document Parsing and Chunking

High update frequency demands efficient and automated document parsing and chunking to ingest new response records quickly. Unstructured text content requires robust natural language processing capabilities to accurately identify and extract key information, such as diseases, drugs, and symptoms. The specialized nature of medical terminology and units requires the tokenizer and entity recognition models to possess domain-specific medical knowledge, preventing incorrect splitting or omission of professional vocabulary. Mixed language content requires the parser to identify and process different language texts, ensuring semantic integrity of chunks. Furthermore, due to the traceability requirements of response records, original document metadata (e.g., inquiry ID, timestamp) must be properly preserved and associated during the chunking process.

Configuration Guidelines

Configuration Item	Recommended Value	Rationale
`Chunk size` (Chunk Length)	`500–800 characters` (characters)	Balances semantic completeness and recall efficiency, preventing excessively long or short chunks from losing context or introducing too much noise.
`Chunk Overlap Length` (Chunk Overlap Length)	`50–100 characters` (characters)	Ensures contextual continuity at chunk boundaries, improving retrieval recall, especially when processing question-answer pairs.
`UPLOAD_FILE_MAX_SIZE`	`200 MB`	Accommodates the size of individual response records or bulk import files, preventing upload failures due to oversized files.
`PARSE_FILE_TIMEOUT_SECONDS`	`600 seconds` (seconds)	Provides sufficient time to process documents with large amounts of text or complex structures, preventing parsing timeouts.
`Knowledge Base Chunking Strategy`	`By Title, By Text Length`	Combines with the semi-structured nature of response records, prioritizing semantic structure for chunking, supplemented by length control.
`Max Concurrent Parsing Tasks`	`Based on server resources`	Configured according to server CPU, memory, and other resources to prevent system crashes or parsing interruptions due to excessive concurrency.

Three Common Pitfalls

After uploading a batch of files, some documents fail to parse, and backend logs show File Parsing Timeout (File Parsing Timeout). This typically occurs when the PARSE_FILE_TIMEOUT_SECONDS parameter is set too short, not allowing enough time for large or complex documents to parse.
In the parsed knowledge base, medical terms are incorrectly tokenized, leading to inaccurate retrieval results. This usually happens when the tokenizer is not optimized for the medical domain or lacks relevant domain-specific dictionaries.
After importing a large number of documents, system memory usage becomes excessively high, sometimes leading to OOM (Out Of Memory) errors. This can be due to setting Max Concurrent Parsing Tasks too high, or individual document parsing consuming too much memory.

Verification of Configuration

Upload a typical response record document containing various medical terms and question-answer structures. Examine the parsed chunks to ensure the completeness of key information and specialized vocabulary.
Batch upload a certain number of response record files. Monitor task status through the system backend to confirm all documents complete parsing successfully, with no significant error logs.
Retrieve specific medical terms or inquiry cases from the knowledge base. Evaluate the accuracy and relevance of the recall results. If recall results are not satisfactory, adjust Chunk size (Chunk Length) and Chunk Overlap Length (Chunk Overlap Length).
Monitor server resource usage, especially during batch parsing, to ensure CPU, memory, and other resources remain within controllable limits, thereby assessing the reasonableness of Max Concurrent Parsing Tasks.

Note: The values provided are common starting points. It is recommended to measure them against your own samples for optimal performance.

Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-09-21.