What data for this category looks like
Data for this category comes from design BOMs, silicon steel sheet and insulating material inventory data, vacuum drying equipment operation logs, insulating oil test reports, and high-voltage test station test records. Daily documents include winding and core assembly records, vacuum drying records, insulating oil test reports, partial discharge and withstand voltage test reports, and equipment nameplate parameter tables. Data is updated per single product or per batch of raw materials. Most documents are structured tables and precise parameter records, with fields including insulation resistance, partial discharge magnitude, withstand voltage value, and more. Corresponding units are megaohms, picocoulombs, and kilovolts.
What constraints do these characteristics impose on the document parsing and chunking stage?
Full-process documents for a single product have high relevance. Chunking must retain cross-document parameter associations to avoid breaking the correspondence between test data and assembly records. Structured table documents have fixed fields but minor differences in units. Chunking must retain the binding relationship between table headers and data to prevent field misalignment after parsing. Frequently updated raw material and equipment log data requires adaptation to fast upload chunking efficiency, while ensuring that single chunk content does not exceed the processing limit of the vectorization model. The single chunk length of key test data such as partial discharge and withstand voltage must be controlled to avoid reduced retrieval accuracy caused by too many scattered parameters in a single chunk.
How to set the configurations
The following configurations are formulated based on the document parsing and chunking feature of FastGPT v4.17.0:
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Adapts to the single-chunk carrying capacity of structured tables and long parameter records in this category of documents, avoiding loss of details during vectorization caused by excessive single-chunk content |
chunkSplitter | Split by paragraph and table boundaries | Retains semantic integrity of documents such as winding records and test reports, preventing splitting from breaking the association between tables and preceding or following explanations |
customPdfParse | Enable structured parsing mode | Accurately extracts fixed fields from insulating oil reports and withstand voltage test reports, avoiding parameter misalignment caused by plain text parsing |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Adapts to the total size of full test records and assembly documents for a single product in this category, meeting batch upload requirements |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Addresses parsing time for large single test reports, avoiding interruptions to the parsing process due to timeout |
chunkSettingMode | Manual configuration mode | Customizes chunking rules for the structured characteristics of this category of documents, adapting to splitting needs of different document types |
Three common mistakes
- Phenomenon: After uploading an insulating oil report or withstand voltage test report larger than 10 MB, some chunks fail vectorization, and the interface shows abnormal status for some chunks. Cause: The
chunkSizeis not adjusted to fit large file chunking, resulting in single-chunk content exceeding the model's processing limit, or chunking boundaries destroying the integrity of key parameters. - Phenomenon: When attempting to import a Feishu online spreadsheet format nameplate parameter table, parsed fields are empty or formatted incorrectly. Cause: The structured parsing mode of
customPdfParseis not enabled, and the default parsing rules cannot recognize the nested header structure of Feishu spreadsheets. - Phenomenon: The dataset.csv generated using the export function cannot be directly used to create a new knowledge base Q&A set. The export format is only a basic data template, and no Q&A pair format is generated. Cause: The format option for knowledge base export is not switched, and the default export only generates an original data template.
How to confirm the configuration is correct
- Upload a full set of test record documents for a single product, check whether the chunking results retain the complete table headers and corresponding data rows
- View the parsing task run log to confirm there are no interrupt records triggered by
PARSE_FILE_TIMEOUT_SECONDS - Test uploading batch documents within 1000 MB, confirm that the number of chunks and single-chunk character count match the preset configuration
- Import a Feishu format nameplate parameter table, check whether the parsed fields completely match the parameter items in the original document
The values provided are common starting points and should be measured against the reader's own samples.