Data Characteristics
Supplier audit data primarily consists of audit reports from pharmaceutical companies, supplier qualification documents (e.g., GMP certificates, production licenses, product standards, inspection reports), change notifications, and historical audit rectification records. Data update frequencies vary. Qualification documents typically update annually or upon changes. Audit reports update according to the audit cycle.
Audit reports are often structured or semi-structured, containing fixed fields such as audit findings, defect classifications, and recommended corrective actions. Qualification documents come in various formats, including PDFs, scanned images, and Word documents. These may embed images of stamps, signatures, or charts. Fields and units involve production batch numbers, expiry dates, inspection results (e.g., percentage content, impurity ppm values), and equipment calibration data. Units are highly standardized, but expressions may differ.
Constraints on Document Parsing and Chunking
The diversity of supplier audit data imposes multiple constraints on document parsing. First, numerous scanned images and embedded images in qualification documents require robust OCR capabilities. The parsing tool must accurately recognize text, including complex tables and handwritten annotations, within images.
Second, extracting structured data from audit reports requires a fine-grained chunking strategy. This ensures critical information, such as audit findings, defect descriptions, and corrective actions, remains intact and properly linked.
Third, standardizing fields and units, such as unifying different expressions for percentages and ppm values, is fundamental for accurate knowledge base retrieval.
Finally, version management and incremental updates for historical audit data require the parsing process to identify document differences, preventing duplicate entries or overlooking important changes.
Configuration Settings
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
Chunk size (Chunk Length) | 500-800 characters | Balances paragraph integrity in audit reports with retrieval efficiency, preventing excessive splitting of key information. |
Overlap Length | 50-100 characters | Ensures contextual continuity, especially for linking audit findings and corrective actions. |
OCR_ENABLED | True | Supplier qualification documents often contain scanned images and embedded image information, requiring OCR. |
IMAGE_EMBEDDING_ENABLED | True | Identifies visual information like stamps and charts within embedded images, enhancing semantic understanding. |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Provides sufficient parsing time for large audit reports and qualification documents with complex charts. |
CHUNK_STRATEGY | By Paragraph | Audit reports are often organized by paragraphs; chunking by paragraph maintains semantic integrity. |
Common Pitfalls
- Image content is missing or incorrectly recognized in parsing results, appearing as empty or garbled fields. This occurs if OCR is not enabled or the OCR engine's recognition capability is insufficient.
- Critical information in audit reports (e.g., defect descriptions and corrective actions) is split into multiple disconnected chunks. This leads to missing context during retrieval. The
Chunk size(Chunk Length) setting is too small. - The knowledge base fails to recognize certain supplier qualification document links, with logs showing "unsupported link type." This happens if only specific formats of public sharing links, such as
https://docs.yuque.com/type public documents, are supported.
Verification Steps
- Select a supplier qualification document containing scanned images and embedded images. After parsing, check if the text content is complete and accurate, especially key information within images.
- Choose a typical audit report. After parsing, check if key paragraphs, such as audit findings, defect classifications, and corrective actions, are chunked completely and maintain logical coherence.
- Use a document containing specific fields and units (e.g., percentage content, ppm values) for parsing. Verify that field extraction results are correct and units are standardized.
The values provided are common starting points and should be measured against the reader's own samples.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-09-21.