Document Parsing and Chunking for Preclinical Safety

Data Characteristics

Preclinical safety assessment data primarily originates from early drug development safety evaluation reports. These include in vivo and in vitro toxicology studies, pharmacokinetic (PK) reports, and pharmacodynamic (PD) reports. Documents are mainly in PDF format, with some Word documents and Excel spreadsheets. Data update frequency is relatively low, with generation concentrated after each preclinical study phase. Document structure is highly standardized, adhering to GLP (Good Laboratory Practice) guidelines. They contain fixed sections such as objectives, materials and methods, results, discussion, and conclusions. Fields within reports include dosage, administration route, test substance information, animal strain, observation indicators (e.g., body weight, organ coefficients, blood routine, urine routine, pathological examination results), adverse reaction descriptions, and toxicity grades. A wide variety of units are present, such as mg/kg, g/kg, mmol/L, U/L, ℃, and %, often presented in tabular form.

Constraints on Document Parsing and Chunking

The standardized structure and extensive tabular data in preclinical safety assessment reports require the document parser to accurately identify sections and extract table content. Fixed sections enable more effective structured parsing strategies, prioritizing key sections to improve information extraction efficiency. Raw Excel data or embedded tables in reports require the parser to accurately identify headers, rows, and columns, and associate cell content with corresponding field semantics, preventing data misalignment or loss. Strict professional terminology and diverse measurement units in reports demand finer granularity for chunking. This ensures each chunk contains complete semantic information, avoiding truncation that separates critical data or units from values. The low update frequency means initial parsing quality is paramount, with less need for subsequent re-parsing. Therefore, parsing stability and accuracy take precedence over parsing speed.

Configuration Recommendations

Configuration Item	Recommended Value	Rationale
`UPLOAD_FILE_MAX_SIZE`	`500 MB`	Preclinical safety assessment reports often contain numerous charts and images, leading to large file sizes.
`Chunk size` (Chunk Length)	`800–1200 characters` (characters)	Ensures inclusion of complete observation indicators, dosages, units, and adverse reaction descriptions, maintaining semantic integrity.
`Chunk Overlap`	`100–200 characters` (characters)	Connects different observation periods or related descriptions, reducing information loss.
`PARSE_FILE_TIMEOUT_SECONDS`	`600 seconds` (seconds)	Accommodates parsing time for large PDF files and complex tables, preventing timeouts.
`Table Content Extraction Mode`	`Structured Extraction`	Improves extraction accuracy for large amounts of standardized tabular data.
`Named Entity Recognition`	`Enable and configure specific entities`	Identifies key information such as drug names, dosages, units, and toxicity grades.

Common Pitfalls

After uploading large PDF or Excel files, the system becomes unresponsive for an extended period or returns a 504 Gateway Timeout error. This typically indicates the file size is too large or the PARSE_FILE_TIMEOUT_SECONDS parameter is set too short, causing the parsing process to time out.
After knowledge base training, query results show values separated from units, or tabular data is disorganized. This occurs when the document parser's table content extraction mode is inappropriate, failing to correctly identify table structures, leading to incomplete semantics during chunking.
After importing an Excel file, some row data is not correctly recognized or fields are empty. This may be due to the Excel file structure not meeting expectations, or the parser incorrectly matching headers with data columns.

Verification Steps

Upload a typical safety assessment report PDF containing complex tables and multi-page charts. Observe if the parsing status is successful and check the number of chunks and their content in the knowledge base.
Randomly select 5-10 chunks from the knowledge base. Verify that each chunk contains complete dosage, unit, observation indicators, and adverse reaction descriptions, ensuring semantic integrity.
Import an Excel report containing various measurement units and professional terminology. Check if the data fields in the knowledge base after import match the original file, and if units and values are correctly associated.

The values provided are common starting points and should be measured against the reader's own samples.

Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-09-21.