What the data for this category looks like
The data for this category mainly comes from registered process documents, raw material batch ledgers, DCS production control systems, weighing systems, LIMS laboratory information management systems, chromatography data systems, and QMS quality management systems. Daily generated documents include batch production records, material balance sheets, reaction and purification step records, cleaning validation records, impurity profile reports, and deviation and change control documents. Data updates follow the rhythm of each production batch and each process step. A single document contains clear process parameter fields such as feeding amount, reaction temperature, impurity content, and solvent residue content, corresponding to complete process records for a single production link or batch.
What constraints these characteristics impose on the "document parsing and chunking" link
Documents in this category are mostly multi-step process records and professional test reports. Most fields include process parameters and quality indicators, and single-batch documents are associated with multi-link data. The high-frequency update feature requires chunking to adapt to independent business units of a single batch or single step, avoiding invalid chunking across batches and steps. The dense professional terminology feature requires the parsing link to retain chemical and process proper nouns, preventing damage to term integrity during splitting. At the same time, there is a strong correlation between documents, such as batch production records and corresponding impurity profile reports. Chunking must retain context associations to ensure that the complete process chain can be traced during cross-review. Some documents are exported from different industrial systems, with non-standard formats. The parsing link must support files in different formats to avoid parsing failures.
How to set configurations
The configurations on this page are based on FastGPT v4.17.0.
| Configuration Item | Recommended Approach | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Documents in this category contain long paragraphs of process steps and test data. 800–1200 characters can retain the complete context of a single step or single test item, avoiding splitting of professional terms and associated data |
chunkSplitter | Split by document headings and paragraph boundaries | Documents in this category have a clear structure. Splitting by headings can match the step chapters of batch production records, ensuring that chunks correspond to independent business units |
customPdfParse | Enabled | Some documents in this category are non-standard PDFs exported from industrial systems. Enabling custom parsing can support extraction of tables and professional symbols, improving parsing accuracy |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Associated documents for a single batch, such as batch records plus impurity profiles, may occupy large storage space, requiring allowance for large file uploads |
PARSE_FILE_TIMEOUT_SECONDS | 300–600 seconds | Parsing large or multi-page process documents requires a longer time, avoiding parsing failures due to timeout |
chunkSettingMode | Custom mode | Documents in this category have fixed structural and field characteristics. Custom mode allows flexible adjustment of chunking rules to adapt to the splitting needs of professional documents |
Three common mistakes
- Symptom: After uploading a PDF or DOCX file in a server environment, the document parsing node returns a 404 error, and parsing works normally in a local environment. Cause: The server environment has not correctly configured read and write permissions for the file storage path, or the front-end upload path does not align with the back-end parsing path.
- Symptom: After deploying a specified large model, professional fields such as impurity content and batch number cannot be extracted from the parsing results. Cause: The
customPdfParseconfiguration is not enabled, or the chunking rules do not match the professional term context of the document, resulting in loss of key fields during parsing. - Symptom: Chunk results include cross-batch associated data, making it impossible to locate single-step process parameters during cross-review. Cause: The
chunkSplitterconfiguration is set to split by fixed character count, rather than by document headings and step boundaries, destroying the structural integrity of single-batch documents.
How to confirm configurations are correctly set
- Upload a single-batch batch production record PDF, and check whether the parsing result retains complete reaction steps and parameter fields, with no missing content or garbled text.
- View the file upload log to confirm that the uploaded file has been stored in the specified directory, with no path errors or permission errors.
- Adjust the
chunkSizeparameter, compare the context integrity of the chunk results, and confirm that single-step process records are not split into multiple chunks. - Upload multiple associated documents, such as batch records and impurity profile reports, and check whether the chunk results retain the association identifiers between documents.
The values provided here are common starting points and should be evaluated against the reader's own samples.