What Data for This Category Looks Like
Typical sources include silicon wafer incoming material records, workshop MES systems, operation logs of diffusion furnaces, coating equipment, and printing presses, EL and IV test reports, and laboratory quality inspection data. Updates occur after each production batch, after each product inspection, and during daily summaries. Daily documents include process cards, diffusion and coating parameter tables, printing and sintering operation records, EL defect detection images, IV performance grading data, and yield statistics documents. Most document fields are process parameter values, equipment operation metrics, test result identifiers, and batch numbers. Corresponding units include degrees Celsius, micrometers, watts, and other standard industrial units.
Constraints for Document Parsing and Chunking
Multi-source, heterogeneous document formats including plain text, tables, and charts require the parsing pipeline to adapt to all file types. This avoids separating image-text or table data. High-frequency batch and wafer-level data require chunked content to accurately associate batch and wafer number identifiers. This avoids mixing data across different batches. Fine-grained single-process parameter and test data require that chunks not be overly large. Overly large chunks will lose associated details of defects and performance for individual wafers. Document structures vary significantly across processes. Process cards follow linear step sequences, while equipment logs use time-series data. Chunking logic must adapt to these different structures to ensure full data for the corresponding process can be located during retrieval.
Configuration Settings
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Photovoltaic cell manufacturing documents contain continuous process parameter groups and batch data. This range preserves contextual associations between parameters and avoids splitting critical process steps. |
chunkSplitter | Chunk by paragraph + batch identifier | Daily documents are divided into paragraphs by process batch. Chunking this way ensures each chunk corresponds to complete single-batch operation data, facilitating subsequent retrieval and association. |
customPdfParse | Enable pdf-marker to adapt to mixed image-text documents | PDF documents containing EL detection images and numerical tables will extract chart annotations and corresponding parameters when enabled, preventing information loss caused by separated image-text content. |
PARSE_FILE_TIMEOUT_SECONDS | 1200 seconds | Multi-process documents for a single batch may contain large amounts of time-series data. A longer timeout ensures complete parsing of long files and avoids mid-process interruptions. |
chunkSettingMode | Custom mode | Photovoltaic cell manufacturing documents have large structural differences. Custom mode allows flexible adjustment of chunking rules to adapt to different file types such as process cards and equipment logs. |
Three Common Misconfigurations
- Phenomenon: Insufficient table data entries are returned during retrieval. For example, only some entries of a single batch's IV grading data are returned. Cause: The
chunkSizesetting is not adjusted for the fine-grained batch data of photovoltaic cells. Overly large chunks split the complete table for a single batch, preventing full entries from being matched during retrieval. - Phenomenon: Annotation information for EL detection images is lost after PDF parsing. Cause: The
customPdfParseconfiguration is not enabled. The default parsing logic only extracts plain text content and cannot adapt to mixed image-text EL detection reports. - Phenomenon: Timeout errors occur when parsing long documents containing multiple processes. Cause: The
PARSE_FILE_TIMEOUT_SECONDSsetting is too small, and insufficient time is reserved for parsing long files containing large amounts of time-series equipment logs.
How to Verify Proper Configuration
- Upload a typical PDF document containing EL detection images and parameter tables, and check if the parsing result based on FastGPT v4.17.0 includes chart annotations and corresponding parameter values to confirm that the
customPdfParseconfiguration is active. - Import a single batch's equipment log document, and check if chunked content is divided by batch identifiers and paragraphs to confirm that the
chunkSplitterandchunkSettingModeconfigurations perform as expected. - Upload the largest-sized process document, wait for parsing to complete, and confirm that no timeout errors occur to verify that the
PARSE_FILE_TIMEOUT_SECONDSsetting is reasonable. - Retrieve process parameter data for a single batch, confirm that all associated entries for that batch are included in the returned results to verify that the
chunkSizeconfiguration does not overly split data.
The values provided here are common starting points and should be evaluated against the reader's own samples.