Document Parsing and Chunking for Stability Study

Data Characteristics

Stability study data originates from long-term, accelerated, and intermediate stability test reports. These reports are typically in PDF, DOCX, or scanned image formats. Data updates align with batch production and national drug regulatory annual reporting requirements, usually quarterly or annually. Document structure is generally fixed, including test batch information, storage conditions (temperature, humidity, light), test items (e.g., content, dissolution, related substances, microbial limits), test methods, raw data tables, trend charts, and conclusions. Fields include batch number, test date, time points (e.g., T=0, 3M, 6M, 12M), test results (numerical, with units like %, μg/mL, CFU/g), and limit standards. Units and test methods must strictly follow pharmacopoeia or registration standards.

Constraints on Document Parsing and Chunking

The fixed structure and extensive tabular data in stability study reports require high-accuracy table recognition from the document parser. Trend charts and graphs within reports, if not effectively extracted as text, can lead to critical data loss. Update frequency is relatively low, but each update may involve comparing large amounts of historical data, demanding robust knowledge base version management and incremental updates. Accurate identification of numerical fields and units is critical; any parsing error can affect subsequent pharmacovigilance decisions. Time point field recognition must support various expressions, such as 3Units months or 3M. If internal Confluence pages contain stability study procedures or standards, their parsing capability directly impacts knowledge base completeness. The linear flow of text necessitates more structured processing for front-end display to enhance user experience.

Configuration Settings

Configuration Item	Recommended Value	Rationale
`Chunk size`	`800–1200 characters`	Ensures each chunk completely contains all relevant descriptions, data rows, and conclusions for a single test item, preventing key information truncation.
`Chunk overlap`	`100–200 characters`	Preserves contextual relevance, especially where tables span pages or paragraphs connect, aiding more comprehensive information retrieval during RAG.
`File Type Whitelist`	`['.pdf', '.docx', '.csv', '.xlsx']`	Covers primary formats for stability study reports, compatible with raw data and summary reports.
`ENABLE_TABLE_EXTRACTION`	`true`	Stability study reports contain extensive tabular data; enabling table extraction is crucial for obtaining core data.
`maxContext`	`4000 characters`	Ensures retrieval results can include comparative data from multiple batches or time points, supporting more complex queries.
`Recall count`	`5 entries`	Provides sufficient comparative samples for multi-dimensional analysis while maintaining accuracy.

Common Pitfalls

Misaligned table data rows or merged cell content in parsed output. This occurs when document table lines are blurry or complex nested headers exist, leading to inaccurate structural recognition by the parser.
Text descriptions or conclusions below stability trend charts are not extracted. This may be due to disabled Optical Character Recognition (OCR) or insufficient compatibility with specific fonts or layouts.
Internal Confluence page content fails to parse, showing error codes 401 Unauthorized or 403 Forbidden. This happens when FastGPT is not configured with the correct internal proxy or authentication information, preventing access to restricted resources.

Verification Steps

Upload a typical stability study report PDF file. Check if all tabular data is correctly parsed into the knowledge base, specifically verifying that batch numbers, test items, time points, and numerical result fields are complete and correctly aligned.
For reports containing trend charts and conclusions, retrieve relevant keywords. Verify that the recall results include text descriptions below or near the charts.
Attempt to parse an internal Confluence page containing stability study procedures. Confirm that content is successfully fetched and added to the knowledge base.
Use queries containing specific units (e.g., %, μg/mL). Verify that the system accurately recalls test results containing these units and that numerical values and units match.

The values provided are common starting points and should be measured against the reader's own samples.

Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-09-21.