What this category of data looks like
This data typically originates from MES systems, equipment FDC collection modules, defect detection metrology devices, process recipe libraries, and similar sources across process integration, module, equipment, yield, and manufacturing departments in wafer fabs. Daily documents include process flow cards, batch run records, equipment recipes, SPC charts, exception handling sheets, defect maps, yield reports, and more. Data is generated in real time alongside wafer batches. Some statistical reports are updated daily. Most documents contain fields such as process parameters, batch numbers, equipment identifiers, defect coordinates, and yield values. Some content includes nanometer-precision values, batch traceability codes, and other exclusive formats and identifiers.
What constraints these characteristics impose on document parsing and chunking
Multi-source heterogeneous document formats (including tables, charts, and numerical reports) require the parsing link to support extraction of different structured content. High-frequency real-time batch data requires the chunking process to have low-latency processing capabilities. This avoids impacting the timeliness of inspection and audit. Documents tied to ultra-long process chains have large content spans. Incorrect cross-process chunking must be avoided to prevent key process steps from being split. Exclusive fields such as nanometer-level parameters and batch codes require retaining context associations during chunking. This prevents key parameters from being split into different chunks. Documents in industrial controlled formats often use non-standard layouts. Parsing support for non-general PDF, CSV, and other formats is needed.
How to set configurations
These configurations are adapted for FastGPT v4.17.0.
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Adapts to the length of process descriptions and parameter groups in wafer manufacturing documents, avoiding splitting of key process steps |
chunkSplitter | Split by paragraph + table boundaries | Adapts to documents with structured tables such as SPC charts and defect maps, preventing table content from being split |
customPdfParse | Enable industrial PDF parsing mode | Adapts to non-standard controlled PDF documents in wafer fabs, retaining parameter fields and chart structures |
PARSE_FILE_TIMEOUT_SECONDS | 300–600 seconds | Adapts to the parsing duration of large batch run records and yield reports, avoiding timeout failures |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Adapts to the upload requirements of complete run records and defect map files for large wafer batches |
chunkSettingMode | Manual configuration mode | Adjust chunking parameters for different document types (such as recipe libraries, defect maps) to adapt to exclusive field formats |
Three common errors
- The issue: SPC charts and table documents uploaded to the knowledge base only display text content in search results, with table structures lost. The cause is that the
customPdfParseindustrial PDF parsing mode is not enabled. The default parsing logic cannot retain the structured layout of tables. - The issue: Adjacent process parameters in chunking results are split into different chunks, making complete process step association impossible during retrieval. The cause is that the
chunkSizesetting is too small, failing to cover the complete parameter group length of a single process. - The issue: A timeout error is returned when parsing large yield reports. The cause is that the
PARSE_FILE_TIMEOUT_SECONDSsetting value is too low, failing to match the parsing time of large industrial documents.
How to confirm the configuration is properly set
- Upload a process flow card document containing SPC tables, and check whether the complete table row and column structure is retained in the parsing result.
- Select a segment of process description text containing three consecutive processes, and verify that the chunking result does not split process parameters into different chunks.
- Upload a batch run record file of approximately 100 MB, and check whether the parsing task completes within the preset
PARSE_FILE_TIMEOUT_SECONDS. - Switch between different document types (such as defect maps, equipment recipes) to confirm that the chunking rules automatically adjust based on the configuration.
The values provided are common starting points and should be measured against the reader's own samples.