What Data for This Category Looks Like
Core data for pharmaceutical dosage form manufacturing comes from approved formulation processes, material batches, MES systems, electronic batch records, LIMS systems, and quality management systems. This data is generated daily by production, quality assurance, quality control, and warehouse departments. Daily documents include master production files, batch production records, batch packaging records, cleaning records, inspection records, deviation/CAPA/change documents, and batch release review forms. Most have fixed field structures, including fields with clear units such as dosage amounts, process parameters, and test results. Data update frequency follows per batch, each key step, or per monitoring plan. Single documents often contain associated data for multiple batches and steps.
What Constraints These Characteristics Impose on Document Parsing and Chunking
Documents for this category are mostly multi-column structured tables or fixed-format PDFs. Single documents contain associated data for multiple batches, so batch and step associations for data must be strictly preserved. Frequently updated documents require efficient parsing processes to avoid loading redundant historical data. Documents under GMP compliance requirements must retain original fields and units in full. Automatic chunking may disrupt column structures or split content across batches, so parsing rules need targeted configuration. Multi-source data is exported in diverse formats, so parsing logic for different document types must be supported. Sufficient processing time must also be reserved to avoid timeouts.
Configuration Settings
The following configurations are based on FastGPT v4.17.0.
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Matches the length of core data for a single process step or single batch, preventing content from multiple batches or steps from being merged into the same chunk |
chunkSplitter | Separate by line (for Excel), Separate by chapter (for PDF) | Preserves the integrity of individual material feeding entries, inspection records, or complete process chapters, preventing multi-column data from being disrupted |
customPdfParse | Enable | Strips fixed headers and footers from GMP documents such as batch numbers, production workshops, and other irrelevant content, retaining only core business data blocks |
PARSE_FILE_TIMEOUT_SECONDS | 300–600 seconds | Adapts to parsing duration for multi-batch batch record documents, preventing parsing timeouts due to large data volumes |
chunkSettingMode | Manual configuration | Adjust chunking rules for different document types (Excel batch records, PDF deviation reports) to adapt to the fixed field structure of GMP documents |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Adapts to upload requirements for multi-batch batch record compressed packages, preventing large file uploads from being blocked |
Three Common Mistakes
- Phenomenon: After uploading an Excel-format batch record, the automatic chunking result splits the material name and dosage amount from the same row into different segments, destroying field integrity. Cause: The line-based chunking basis was not specified, and the default semantic chunking rule was used, resulting in cross-row and cross-column splitting of structured data.
- Phenomenon: When parsing the currently uploaded document, the backend also loads previously uploaded documents of the same type, causing a significant increase in parsing time. Cause: The historical document association parsing switch for the knowledge base was not disabled, and the default setting loads all knowledge base documents for the current parsing process.
- Phenomenon: After uploading a batch record compressed package larger than 1000 MB, the parsing task fails directly and returns an upload block prompt. Cause: The
UPLOAD_FILE_MAX_SIZEconfiguration was not adjusted, and the upload request was blocked due to exceeding the default limit.
How to Verify Correct Configuration
- Upload a single Excel batch record with multiple columns of data. Verify each row of data corresponds to an independent segment, with no cross-row or cross-column splitting.
- Upload a single PDF deviation report with headers and footers. Verify parsed chunks have stripped headers and footers, retaining only core deviation descriptions and handling measures.
- Upload a single 500 MB multi-batch batch record document. Confirm no timeout prompt appears during parsing, and chunked results fully cover all batch data.
- Upload the current attachment. Confirm the knowledge base only loads the currently uploaded file, with no historical documents included in the parsing process.
The values provided are common starting points and should be evaluated against relevant samples.