What the data for this category looks like
Daily data sources include customer BOMs and coordinate files, line change logs, stencil and reflow oven temperature records, first article confirmation reports, SPI/AOI inspection reports, maintenance records, and traceability labels. Data is collected from MES systems, chip mounters, SPI/AOI equipment, reflow oven temperature sensors, and similar devices. Update frequencies cover per work order, each line change, and real-time scenarios. Document structures include structured tables, inspection reports with embedded defect images, time-stamped oven temperature logs, and more. Core fields include part numbers, reference designators, component batches, oven temperature parameters, and coordinate positions. Some documents include physical units such as temperature (℃) and coordinates (mm).
Constraints Imposed on Document Parsing and Chunking
The multi-source heterogeneous documents and high-frequency update characteristics of this category impose multiple constraints on the document parsing and chunking workflow. First, documents include inspection reports with embedded defect images and structured tables. The system must support both text extraction and image association, otherwise visual traceability of quality defects cannot be supported. Second, real-time work order data and batch line change documents require the parsing and chunking process to deliver high response speeds, to avoid reduced retrieval efficiency from overly large chunks. Third, the integrity of core fields such as part numbers and reference designators directly impacts traceability accuracy. Chunking must retain field associations to avoid splitting a single record across multiple chunks. Additionally, format differences across document types are substantial. Parsing rules must adapt to multiple structures including tables, logs, and reports.
The following configuration guidance is based on FastGPT v4.17.0’s document parsing and chunking functionality.
How to Set Configurations
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | SMT documents include structured tables, coordinate data, and defect descriptions. This value range balances text integrity and retrieval efficiency, and avoids splitting core field associations |
chunkSplitter | Split by heading + paragraph | Daily documents include hierarchical headings for first article confirmation reports and AOI reports. Splitting by headings preserves the logical integrity of report chapters |
customPdfParse | Enabled | AOI and SPI reports include embedded defect images and stencil photos. When enabled, the system extracts images and associates them with corresponding text content, adapting to the visual retrieval needs of quality traceability |
UPLOAD_FILE_MAX_SIZE | 200 MB | Single work order scenarios often involve packaged uploads of stencil files, multiple AOI reports, and other documents. This value covers most batch upload scenarios |
PARSE_FILE_TIMEOUT_SECONDS | 300–600 seconds | Large BOM files and batch AOI reports take longer to parse. This value range avoids parsing task interruptions due to timeout |
chunkSettingMode | Manual configuration | SMT document formats vary significantly. Manually adjusting chunking parameters allows adaptation to different types of quality traceability documents |
Three Common Misconfigurations
- Symptom: Embedded component images in parsed Excel documents are not associated with corresponding text, and cannot be retrieved. Cause:
customPdfParseis not enabled, or Excel embedded image extraction rules are not configured. The default parsing only extracts plain text and ignores embedded media assets. - Symptom: Uploading a document package larger than 200 MB triggers a 413 status code error for file size limit exceeded. Cause: The
UPLOAD_FILE_MAX_SIZEparameter is not adjusted, and the default threshold is used, which cannot cover the upload needs of batch work order documents. - Symptom: Single coordinate record is split across two chunks in the chunking result. Cause:
chunkSizeis set too small, exceeding the length of a single structured record, leading to broken field associations.
How to Verify Proper Configuration
- Upload an AOI test report with embedded defect images, and check if the parsing result includes image links or corresponding text descriptions.
- Upload a single packaged document package of 200 MB, and confirm the upload and parsing task does not trigger a 413 status code file size limit exceeded prompt.
- Review chunking results, confirm each chunk contains complete report chapters, with no splitting of headings and their associated content.
- Upload an Excel-format BOM test file, and confirm the parsed result retains complete mapping of core fields including part numbers and reference designators.
The values provided are common starting points and should be validated against local sample datasets.