What this category of data looks like
The data for this category comes from the synthesis, purification, quality control, and registration support departments of active pharmaceutical ingredient manufacturers. Core documents include batch production records, material balance sheets, reaction and purification logs, impurity profile reports, deviation and change documents. Data sources include registered process documents, DCS systems, weighing records, LIMS systems, chromatography data systems, and quality management systems. Updates occur once per production batch and per process step. Documents contain fields such as batch numbers, reaction parameters, material consumption, yield values, impurity content, and process change records. There is no fixed unified format, and some tables have relatively deep nested levels.
What constraints do these characteristics impose on document parsing and chunking
Multi-source and heterogeneous document sources lead to significant format differences between documents related to a single batch. There are many nested tables and long text paragraphs. When chunking, it is necessary to avoid splitting process-related content that spans pages, otherwise, the process consistency explanation during bidding will be affected. The high-frequency update requirement for each batch means that the parsing and chunking process must adapt to batch upload scenarios, avoiding resource waste caused by repeated parsing of the same fields. Tender document and quotation preparation requires linking process parameters and material data. Chunking must retain the contextual association between fields, preventing the inability to match core business information such as yield and impurity after splitting, which affects the accuracy of quotations. Some documents contain long sections of chromatography data and impurity profiles, so the chunking logic for long texts must be adapted to avoid truncating key test results.
How to set the configuration
This configuration is based on FastGPT v4.17.0. The specific parameters are as follows:
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 1000–1500 characters | Adapts to the length of a single process step in batch production records, avoiding splitting of cross-step related content |
chunkSplitter | Split by paragraph + table | Retains the contextual association between tables and corresponding paragraphs in the document, adapting to nested table structures |
customPdfParse | Enabled | Adapts to format parsing of multi-source heterogeneous documents, optimizing nested table and long text recognition effects |
PARSE_FILE_TIMEOUT_SECONDS | 120–180 seconds | Covers parsing time for long and batch documents, avoiding timeout interruptions |
UPLOAD_FILE_MAX_SIZE | 500 MB | Adapts to the scenario requirements of combined upload of multiple documents per batch |
Three Common Mistakes
- Phenomenon: Some nested tables cannot be chunked properly, and the parsed table content is scattered. Cause: The paragraph + table chunking rule is not configured, and the default chunking logic splits table rows and associated headers.
- Phenomenon: Out-of-memory error occurs after deployment, and GPU information cannot be detected. Cause: GPU drivers and CUDA environments are not correctly mounted during Docker deployment, or the configured
chunkSizeexceeds the available video memory range of the graphics card. - Phenomenon: Image domain names are lost after importing a Word document, and images cannot load properly after converting to Markdown for upload. Cause: The configuration to retain external image links during document parsing is not enabled, only local image paths are retained.
How to Confirm the Configuration Is Correct
- Upload a single batch production record document, check whether nested tables are fully retained in the parsed chunk results without scattered splitting.
- Check the system GPU recognition log, confirm that the CUDA version matches the graphics card driver, and there are no out-of-memory errors.
- Upload a Word document containing images, check whether the image links in the parsed Markdown content retain the original domain names.
- Upload batch documents for a single batch, check whether the parsing time is within the set
PARSE_FILE_TIMEOUT_SECONDSrange, with no timeout interruptions.
The values provided here are common starting points and should be evaluated against local samples.