Data Characteristics
Monoclonal antibody (mAb) related regulation and Standard Operating Procedure (SOP) documents typically originate from internal quality management systems of pharmaceutical companies, or regulatory guidelines published by agencies like the National Medical Products Administration (NMPA) and the U.S. Food and Drug Administration (FDA). These documents update infrequently. However, updates often involve significant changes to core production and quality control processes. Document structures are primarily PDF and Word formats, containing numerous charts, flowcharts, batch record templates, and detailed operational steps. Common fields include batch number, production stage, quality control (QC) indicators, testing methods, limits, and equipment numbers. Units involve concentration (mg/mL), purity (%), titer, time (h), and temperature (℃). Numerical precision requirements are extremely high.
Constraints on Document Parsing and Chunking
The characteristics of mAb regulation documents impose specific requirements on parsing and chunking. First, complex charts and flowcharts in PDFs and Word documents require the parser to have Optical Character Recognition (OCR) capability. It must also understand mixed text and image layouts to avoid missing critical information. Second, regulatory guidelines and SOP texts are often logically rigorous and hierarchically structured. However, single paragraphs can be long, containing multiple steps or conditions. Automatic segmentation can easily lead to semantic discontinuity. Third, tabular data in batch record templates requires structured extraction by row or cell, ensuring the integrity of each QC point or operational step. Finally, accurate identification of key numerical values and units, such as concentration and purity, directly impacts the precision of question answering. Therefore, chunking should maintain a close association between values and units.
Configuration Guidelines
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
chunk_size | 500–800 characters | Balances paragraph completeness and retrieval efficiency. Avoids excessively long chunks that disperse semantics or overly short chunks that lose context. |
chunk_overlap | 50–100 characters | Ensures sufficient overlap between adjacent chunks. Mitigates semantic discontinuity caused by segmentation boundaries, especially for process step descriptions. |
split_by_title | True | Regulation and SOP documents often have clear chapter titles. Splitting by title effectively maintains semantic integrity and improves RAG performance. |
enable_ocr | True | Addresses image information embedded in documents, such as flowcharts and batch record screenshots. Ensures text within images is also parsed. |
table_parsing_strategy | auto | Automatically identifies and structurally parses tables in documents, especially batch record templates. Ensures tabular data is retrievable. |
max_file_size_mb | 100 MB | Allows for larger file sizes, considering SOP documents may contain many images or appendices. |
Common Mistakes
- After document parsing, critical batch record table data is not recalled in question answering. The
table_parsing_strategyis not enabled or does not correctly identify the table structure. This causes table content to be treated as plain text and chunked without order. - When a user asks about an operational step, the recalled answer only contains part of the step, lacking prerequisites or subsequent impacts. The
chunk_sizeis set too small, causing logically related sentences to be split into different chunks. - After uploading a PDF document, some text information embedded in images is not retrieved. Error logs show OCR processing failed or took too long. The
enable_ocris not activated, or the OCR engine's ability to recognize complex image text is insufficient, leading to a timeout.
Verification Steps
- Upload a mAb SOP document containing complex charts and tables. Review the parsed chunk preview. Confirm whether table content is structurally extracted and if image and text information is complete.
- Ask questions about specific operational steps or QC limits within the document. Observe the recalled chunk content. Ensure each chunk contains a complete logical unit without semantic loss.
- Use text information embedded in images within the document to ask questions. Verify if the OCR function successfully recognized and indexed the text in the images.
- Compare fields involving numerical values and units, such as concentration and purity, before and after parsing. Ensure these key pieces of information are not truncated or incorrectly identified after parsing.
The values provided are common starting points and should be measured against specific samples.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-09-21.