What this category of data looks like
Data sources for this category include MES systems, testing equipment, laboratory records, process parameter documents, customer standards, and historical defect databases. Daily generated documents cover control plans, patrol inspection records, SPC reports, final product inspection reports, nonconforming product review forms, release forms, and quality monthly reports. Data updates align with each production batch, each shift’s patrol inspections, and real-time testing data collected continuously. Documents include fixed fields and specific units: millimeters for inspection dimensions, megapascals for pressure, and parts per million (ppm) for nonconformity rates. Some documents are structured spreadsheets, while others are non-standard review texts.
What constraints do these characteristics impose on document parsing and chunking?
Documents in this category combine structured spreadsheet and non-standard review text features, with high data update frequency. Parsing and chunking workflows must balance efficiency and accuracy. Format differences across multi-source data require parsing rules to adapt to multiple document types including HTML, PDF, and Excel. Fixed fields and specific units require chunking processes to retain contextual associations, and prevent splitting critical release judgment information such as inspection thresholds and tolerance ranges. Large-volume spreadsheets uploaded in batches increase demands on parsing duration and memory usage. Unreasonable chunking rules may cause loss of critical information, which must be avoided.
How to configure settings
The following configurations follow official rules for FastGPT v4.17.0.
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Adapts to the per-page field density of control plans and SPC reports, and retains complete context for inspection thresholds and associated descriptions |
chunkSplitter | Split by paragraph + heading | Matches the hierarchical structure of patrol inspection records and release forms, and avoids splitting independent inspection items and judgment rules |
customPdfParse | Enabled | Supports parsing table and image text embedded in PDFs, and adapts to structured data and inspection data images in SPC reports |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Adapts to the parsing duration of batch final product inspection reports, and avoids parsing timeout failures for large files |
chunkSettingMode | Manual configuration | Adjusts chunking rules for different document types, and ensures that critical approval fields in release forms are not split |
Three common configuration errors
- Symptom: No parsed content appears in the knowledge base after uploading HTML interface documents generated by javadoc. Cause: The default parsing rule does not filter script and style tags embedded in HTML, leading to valid text being incorrectly filtered out.
- Symptom: Inspection data images embedded in PDF documents are not recognized, and recall results only include plain text. Cause: The
customPdfParseconfiguration is not enabled, and the PDF image text extraction function is not activated. - Symptom: Parsing tasks time out and fail when multiple SPC reports are uploaded in batches. Cause: The value of
PARSE_FILE_TIMEOUT_SECONDSis too low, and does not align with the parsing duration of large-volume reports.
How to verify correct configuration
- Upload a control plan PDF, and check that the parsed text includes complete context for inspection parameters and tolerance ranges, with no split or broken inspection items.
- Upload an SPC report that includes embedded tables, and confirm that the parsed result retains the row and column structure of the table and numerical units.
- Upload HTML documents generated by javadoc, and check that the parsed result includes valid content for interface descriptions and parameter descriptions, with no blank or invalid tag text.
- Upload a single final product inspection report with a volume not exceeding 1000MB, and confirm that the parsing task completes within 600 seconds with no timeout errors.
The values provided are common starting points and should be measured against applicable samples.