Data Characteristics
Patient assistance program documents in the biopharmaceutical sector originate from official pharmaceutical company announcements, project manuals, implementation rules, and government policy interpretations. These documents are updated infrequently, typically quarterly or semi-annually, coinciding with policy adjustments, project cycle conclusions, or major treatment plan updates. Document structures commonly include project background, eligibility criteria, assistance plans (drug names, dosing cycles, assistance ratios), application procedures, required materials lists, contact information, and disclaimers. Common fields include drug batch numbers, patient medical record numbers, ID numbers, medical insurance payment ratios, out-of-pocket expenses, assistance amounts, and assistance periods. Units frequently involve currency (Yuan), time (months, years), quantity (boxes, units), and percentages (%).
Constraints Imposed by Data Characteristics on Document Parsing and Chunking
Infrequent document updates mean initial knowledge base creation requires high-quality parsing. Subsequent incremental update pressure is low. Documents are highly structured, especially eligibility criteria, assistance plans, and material lists, which often appear as tables or clear paragraphs. This requires the parser to effectively identify table boundaries and column content, avoiding flattening table data and losing intrinsic structural relationships. Fields involve sensitive personal information and precise numerical calculations. Chunking must ensure complete numerical values and units are included, preventing truncation or unit loss due to overly granular chunking. For example, if a condition like "out-of-pocket expenses exceeding 5000 Yuan" is chunked into "out-of-pocket expenses exceeding" and "5000 Yuan," semantic integrity is compromised. Furthermore, the rigor of policy terms requires chunks to retain complete legal or institutional provisions to support accurate compliance Q&A.
Configuration Settings
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
chunk_size | 800–1200 characters | Patient assistance program documents often contain lengthy policy provisions and detailed descriptions, requiring sufficient length to maintain semantic integrity. |
chunk_overlap | 100–200 characters | Ensures sufficient contextual overlap between adjacent chunks, preventing critical information from being split at chunk edges and leading to incomplete retrieval. |
chunk_strategy | Smart splitting by title, table, paragraph | Tables and titles are critical structural elements in patient assistance documents; smart splitting effectively preserves structural information. |
max_tokens | 4096 | Given the complexity and length of policy documents, a larger token limit is set to handle large PDF files. |
table_extraction | Enabled | Assistance plans and material lists are often presented in tabular form; enabling table extraction improves understanding and utilization of tabular data. |
min_char_length | 50 characters | Filters out overly short, meaningless chunks, ensuring each chunk contains useful information. |
Common Pitfalls
- Truncation or omission of critical numerical values or codes, such as assistance amounts or drug batch numbers, in parsing results. This occurs when the document parser fails to correctly identify data boundaries when processing tables or specific text formats, leading to important information being split during chunking.
- The system fails to correctly identify multi-column data in uploaded Excel-format assistance plans, preventing the association of information across different columns during Q&A. This happens when table parsing is not enabled or incorrectly configured, causing the system to treat table content as plain text and flatten it.
- When a user asks about specific eligibility criteria, the system returns multiple irrelevant policy provisions. This is due to an overly coarse chunking strategy that merges several independent conditions into a single chunk, introducing noise during retrieval.
Verification of Configuration
- Select 5 random patient assistance documents, upload them to the knowledge base, and ask questions about key eligibility criteria, assistance scope, and required materials. Check if the Q&A results are accurate and complete.
- Examine the chunking of documents containing tabular content in the knowledge base. Ensure that table row and column relationships are effectively preserved, for example, by previewing chunk content to confirm that table data has not been flattened.
- For clauses containing numbers and units, such as "out-of-pocket expenses exceeding 5000 Yuan," verify that the system can accurately identify and return complete numerical and unit information by asking relevant questions.
- Simulate a user asking "What materials are needed to apply for assistance?" and check if the returned chunks precisely correspond to the "Required Materials List" section in the document, without including other irrelevant information.
Note: The values provided are common starting points. It is recommended to measure against your own samples for optimal performance.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-09-21.