What the data for this category looks like
This category of data primarily originates from wafer fab MES systems, equipment FDC modules, defect detection and metrology equipment, process recipe libraries, and experiment records. Daily generated documents include process flow cards, batch run records, equipment recipes, SPC control charts, exception handling sheets, defect maps, yield reports, and more. Some documents contain nanoscale parameter fields, wafer coordinate points, and batch association information. Data update cycles cover each batch of wafer output, real-time equipment parameter uploads, and daily yield summaries. Most documents are multi-page PDF format, containing large amounts of structured tables and bitmap defect maps.
What constraints these characteristics impose on document parsing and chunking
Nanoscale parameters and batch-bound fields require retaining contextual associations during chunking, to avoid splitting cross-batch parameter combinations. Mixed documents of multi-page structured tables and bitmap defect maps are prone to cross-page table splitting or failed bitmap parsing. Real-time generated batch run records require parsing timeout settings to match data update cycles, to avoid truncating incomplete batch logs. Most documents originate from scanned copies or controlled printed materials, which may have insufficient resolution leading to OCR recognition errors, so low-quality text parsing logic must be adapted. The document structure of ultra-long process chains requires retaining inter-process association logic during chunking, to avoid breaking the complete chain for process traceability.
How to configure the settings
Configuration item explanations under FastGPT v4.17.0 are as follows:
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Matches the information density of single-process parameter groups and single-page tables in wafer manufacturing documents, to avoid splitting cross-batch associated fields |
chunkSplitter | Prioritize separation by header/footer | Adapts to the chapter structure of multi-page process flow cards, retaining complete context for individual processes |
customPdfParse | Enable OCR enhancement mode | Adapts to low-quality text and bitmap content in scanned exception handling sheets and defect maps |
PARSE_FILE_TIMEOUT_SECONDS | 180–240 seconds | Matches the parsing duration of daily batch-uploaded yield reports and multi-page batch records, to avoid timeout truncation |
chunkSettingMode | Manual configuration | For special fields such as nanoscale parameters and batch IDs, manually adjust chunk boundaries to retain association logic |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Supports uploading complete batch equipment recipe library documents, to avoid splitting overly large process files |
Three common configuration mistakes
- Structured table documents fail to chunk properly: Manifested as table content split into multiple discontinuous chunk units. The cause is that default chunking rules do not adapt to header anchors for cross-page tables, leading to forced splitting of tables by character length.
- GPU fails to be detected after deployment, and out-of-vRAM error is reported: Manifested as container startup logs prompting no available GPU, or insufficient vRAM errors during parsing. The cause is that the Docker container did not correctly mount GPU drivers and the CUDA environment, and the
chunkSizesetting exceeds the vRAM threshold available to the graphics card. - Image domain names are lost after Markdown import: Manifested as generated Markdown text only retaining local relative paths for images, without the original external link domain names. The cause is that original external link configuration information for images was not retained during document parsing.
How to confirm the configuration is correctly set
- Upload a single-page SPC control chart document, and verify that the chunking result retains the complete control limit and sampling point data combination, with no splitting abnormalities.
- Review container startup logs to confirm that the GPU driver and CUDA version match the deployment environment, with no out-of-vRAM error reports.
- Import a Word document containing external link images, and verify that the generated Markdown text retains the complete external link domain names of images.
- Upload multi-page batch run records, and confirm that the parsing logs show no timeout truncation prompts, matching the preset
PARSE_FILE_TIMEOUT_SECONDSduration.
The values provided are common starting points and should be evaluated against the samples used in actual deployments.