Data Characteristics
Cleaning validation data originates from production batch records, cleaning validation protocols, analytical method validation reports, risk assessment reports, and deviation handling reports. Document update frequency is typically low, tied to specific batch production cycles or validation activity execution cycles (e.g., semi-annual or annual retrospective validations). Document structures often include standardized chapter titles, tabular data, experimental result charts, and signature pages. Key fields include equipment number, batch number, product name, residue limit, cleaning agent information, sampling points, analytical methods, test results, and acceptance criteria. Units involve mg/cm², ppm, µg/swab, requiring high precision.
Constraints on Document Parsing and Chunking
The standardized chapters and tabular data in cleaning validation documents require parsers to accurately identify and maintain structural integrity, especially data relationships within tables. Critical numerical fields like residue limits and test results need precise extraction. Correct unit identification is vital for subsequent numerical comparisons and logical judgments. Since document update frequency is low, there is less pressure on the timeliness of document parsing and chunking, but accuracy and stability are paramount. Experimental charts and signature pages, commonly found in these documents, require special handling during chunking. Mark them as non-text content or provide summary descriptions to avoid interfering with core information extraction. Identifying identifiers like batch numbers and equipment numbers helps establish document relationships, forming a more complete knowledge graph.
Configuration Settings
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
Chunk size (Chunk Length) | 800–1200 characters | Balances contextual completeness and retrieval efficiency. Avoids excessively long chunks diluting key information and excessively short chunks losing semantic connections. |
Chunk Overlap Length (Chunk Overlap Length) | 150–200 characters | Ensures contextual continuity across chunks, especially at the boundaries of tables or critical descriptive paragraphs. |
Parsing Mode | Smart Parsing | Suitable for documents with mixed content including tables, chart descriptions, and text. Better at identifying structure. |
Extract Table Content | Enabled (Enable) | Cleaning validation documents contain a large amount of critical tabular data that needs accurate extraction and structuring. |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Cleaning validation documents can have many pages and complex structures; allows sufficient parsing time. |
UPLOAD_FILE_MAX_SIZE | 100 MB | Accommodates validation reports that may contain many images and scanned documents. |
Common Pitfalls
- After uploading a PDF file, the system displays "File parsing failed" or "Request error." This may be due to the file being too large or containing encrypted content, causing the parser to time out or lack permissions.
- In the parsed knowledge base, tabular data is incorrectly parsed as continuous text, leading to the loss of critical numerical values and units. This occurs because the parsing mode did not correctly identify the table structure.
- In retrieval results, batch data for the same equipment is confused. This happens when document chunking does not sufficiently use identifiers like batch numbers for semantic isolation.
Verification Steps
- Upload a typical cleaning validation report PDF file to the knowledge base. Check the parsing logs for any exceptions or timeout messages.
- Randomly select parsed document chunks and compare them against the original document. Verify that key fields (e.g., residue limits, test results) and their units are extracted accurately.
- For documents containing complex tables, verify that table content is correctly identified and structured. For example, query test values for a specific equipment and batch number to confirm result accuracy.
- Perform retrievals using different keywords. Observe whether the recalled chunks fully contain the context required for the query, especially for information spanning multiple pages or paragraphs.
Note: The values provided are common starting points. Measure them against specific samples.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-09-21.