Document Parsing and Chunking for Cleaning Validation

Data Characteristics

Cleaning validation quality documents in the biopharmaceutical sector originate from cleaning protocols, validation plans, validation reports, deviation records, and change control documents. These documents are typically in PDF, Word, or scanned image formats. Update frequency correlates with production batches, equipment maintenance, and regulatory requirements, leading to weekly, monthly, or quarterly revisions.

Document structure is highly standardized, including clear titles, sections, tables, figures, and attachments. Key fields cover equipment names, cleaning agents, cleaning process parameters (e.g., temperature, time, concentration), sampling points, residue limits, test methods, test results, and conclusions. Units are precise, such as temperature (°C), time (min), concentration (ppm), and residue (µg/cm²). Documents often include specific batch numbers, report numbers, and signatures.

Constraints on Document Parsing and Chunking

The standardized structure and precise key fields of cleaning validation documents demand high accuracy and structured extraction capabilities from document parsing. Extensive tabular data and figures challenge traditional text-based parsing methods. The parsing system must identify and process complex tables and understand captions accompanying figures.

Unique identifiers like batch numbers and report numbers require chunking to maintain information integrity, preventing context loss from cross-chunk splitting. Frequent updates and revisions mean the parsing system needs version management capabilities to quickly identify added, modified, or deleted content. Strict units and field formats require parsing results to accurately retain original numerical values and unit information, providing reliable data for subsequent retrieval and inference.

Configuration Settings

Configuration Item	Recommended Value	Rationale
`Chunk size` (Chunk Length)	`800–1200 characters`	Paragraphs in cleaning validation documents often contain complete process descriptions or test results. This length helps maintain contextual integrity.
`Chunk Overlap Length` (Chunk Overlap Length)	`100–250 characters`	Ensures sufficient overlap between adjacent chunks to capture key information relationships across paragraphs.
`Enhanced File Parsing`	Enable `Marker` or `LayoutParser`	Handles PDFs/scanned documents with numerous tables, figures, and complex layouts, improving structured data extraction accuracy.
`PARSE_FILE_TIMEOUT_SECONDS`	`600 seconds`	Cleaning validation reports can be large files containing multiple pages of high-resolution images, requiring longer parsing times.
`maxContext`	`32000 tokens`	Ensures that necessary technical details and related information from cleaning validation reports are covered during question answering.
`Recall count` (Retrieval Count)	`Top 5-8 entries`	Considering the professional and correlative nature of the document content, appropriately increasing the retrieval count improves relevance coverage.

Common Pitfalls

Loss or misalignment of critical tabular data after file parsing. This occurs when parsing tools inaccurately recognize complex table structures or when Enhanced File Parsing is not enabled.
Failure to invoke the file parsing tool during a conversation, returning a generic error. This might be due to incompatible marker_images versions in the Docker environment or incorrect GPU driver configuration.
Retrieval results failing to include specific batch numbers or equipment names explicitly mentioned in the document. This happens when chunk granularity is too fine, splitting key identifiers across different chunks, or when Chunk Overlap Length (Chunk Overlap Length) is set too low.

Verification Steps

Upload a cleaning validation report containing complex tables and figures. Check if the parsed text fully retains table content and figure descriptions.
Retrieve key fields (e.g., batch number, residue limit, test results). Verify the accuracy and completeness of these fields in the retrieval results.
Ask a question about a specific cleaning process step in the document. Verify if the answer accurately references the equipment, cleaning agents, and parameters involved in that step.
Attempt to upload a recently revised cleaning validation protocol. Confirm if the system can identify and process changes, such as new or modified cleaning parameters.

The values provided are common starting points. Measure them against specific samples to determine optimal settings.

Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-09-21.