What the data for this category looks like
The documents for this category mainly include winding and core assembly records, vacuum drying records, insulating oil reports, partial discharge and withstand voltage test reports, and nameplate parameter tables. Data sources include design BOM, silicon steel sheet and insulating material data, vacuum and drying equipment collection data, oil test results, and test station test data. Documents are updated per single product or each batch of materials, and include fields such as insulation water content, partial discharge amplitude, withstand voltage level, lead connection position, etc. Units are mostly ppm, pC, kV, mm, and similar units.
Constraints on document parsing and chunking
Documents for single products require accurate traceability of corresponding batch materials and test data. Therefore, the association between fields must be retained during parsing, and content across associated fields must not be split arbitrarily. Insulating oil reports and test reports contain highly sensitive insulation parameters. The parsing process must ensure that data is not leaked. A large number of tabular data exist in documents, such as test data tables and nameplate parameter tables. Chunking must avoid splitting complete rows or columns of tables, otherwise the corresponding relationship between parameters will be disrupted. The update frequency of documents for each product is independent. Chunking must adapt to the small-batch parsing scenario of single documents to avoid excessive system resource consumption.
Configuration settings
This configuration is based on FastGPT v4.17.0
| Configuration Item | Recommended Setting/Value | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Adapts to the length of single-paragraph parameter descriptions in test reports and assembly records for this category of documents, avoiding splitting complete test data groups |
chunkSplitter | Split by paragraph + table boundaries | Matches the characteristics of this category of documents, which contain a large number of structured tables and coherent process descriptions, and retains table integrity |
customPdfParse | Enabled | Uses custom parsing rules to extract structured fields for complex layouts in insulating oil reports and withstand voltage test reports |
UPLOAD_FILE_MAX_SIZE | 500 MB | Adapts to the total size limit of full batch documents for a single product, avoiding parsing interruptions caused by oversized files |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Reserves sufficient time to process long documents containing multiple sets of test data, avoiding timeout failures |
chunkSettingMode | Manual configuration | Adjusts chunking rules manually for the personalized document structure of single products to match field association relationships |
Three frequently made mistakes
- The tables in uploaded test report PDFs appear misaligned, and the corresponding relationship between fields is disrupted after parsing. The cause is that the table boundary splitting rule of
chunkSplitteris not enabled, and direct character-based chunking causes table rows to be split. - Document parsing fails, and the interface displays a parsing timeout status. The cause is that
PARSE_FILE_TIMEOUT_SECONDSis not adjusted to a value suitable for long documents, and the parsing time of the complete test report for a single product exceeds the default threshold. - The parsed fields of the uploaded nameplate parameter table are empty. The cause is that
customPdfParseis not enabled, and the default parsing rule cannot recognize compactly arranged parameters in nameplates.
How to confirm correct configuration
- Upload a typical insulating oil report PDF, and check whether the parsed text retains complete test data tables without row or column splitting.
- Upload a long document containing multiple sets of test data, and confirm that no timeout error is triggered after parsing is complete.
- Manually adjust the
chunkSizeparameter, upload the same document to compare chunking results, and confirm that chunks do not split paragraphs associated with key parameters. - View parsing logs to confirm that no abnormal records related to file content leakage appear, verifying that the parsing process is performed locally.
The values provided are common starting points and should be measured against the reader's own samples.