What this category of data looks like
The data for this category mainly comes from customer historical system exports, CSV or Excel templates, database export results, and product import logs. Commonly processed documents include data migration templates, field mapping tables, migration execution plans, and migration validation reports. Data updates occur per order or irregularly. Single imported datasets are mostly multi-column structured content, including fields such as business codes, field mapping relationships, and data validation rules. There are no unified fixed units, and adjustments are made based on the customer's actual business scenarios.
Constraints imposed on the document parsing and chunking link
Multi-source structured data requires the parsing module to support multiple file formats including Excel and CSV, to avoid misalignment of business fields caused by improper splitting. Irregular single-batch imports require the parsing process to only handle the currently uploaded file, preventing accidental loading of historical documents. The structured nature of documents such as field mapping tables requires chunking logic to retain the integrity of individual records, and not randomly split cell or field content. The demand for large files in batch imports also requires the parsing module to set reasonable timeout and size limits to avoid task interruptions.
Configuration Settings
The following configurations are based on FastGPT v4.17.0, and can be fine-tuned according to actual migration scenarios:
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Adapts to the length of a single migration record plus field descriptions, avoiding splitting complete business field content |
chunkSplitter | Split by line | Adapts to structured data formats such as Excel and CSV, ensuring each row is an independent segment that meets the chunking requirements of migration data |
customPdfParse | Disabled | Most migration data is in structured formats such as Excel and CSV, no custom PDF parsing is required, reducing redundant processing steps |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Adapts to the demand for large file imports in batch migrations, preventing single imports from being blocked by the system |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Covers the parsing time of batch structured data, preventing timeout errors during large file parsing |
chunkSettingMode | Manual mode | Allows custom chunking rules, adapting to the chunking requirements of special structured documents such as field mapping tables |
Three Common Misconfigurations
- Phenomenon: Chaotic automatic chunking results after uploading an Excel file, with each row unable to be used as an independent segment. Cause: The
chunkSplitterconfiguration is not set to split by line, and the default semantic paragraph chunking logic is used, which destroys the field integrity of structured data. - Phenomenon: When importing migration data in batches, the system parses previously uploaded documents simultaneously. Cause: The option to only process the current attachment is not selected during the upload step, or the global parsing cache is not cleared, causing historical documents to be loaded and parsed together.
- Phenomenon: After uploading an image via the API, the model cannot recognize the image content. Cause: The OCR function of the document parsing module is not enabled, and the API request does not carry the trigger configuration for image parsing, resulting in the image only being stored as an attachment and not parsed.
How to Verify Configurations Are Correctly Set
- Upload a single Excel test file with multiple columns, check the chunking preview result, confirm that each row corresponds to an independent segment, and verify that the
chunkSplitterconfiguration takes effect. - After uploading a batch migration file, check the parsing log to confirm that only the currently uploaded file is parsed, and no historical documents are loaded.
- Upload a test file exceeding the preset
UPLOAD_FILE_MAX_SIZE, verify that the upload process proceeds normally without blocking errors. - Wait for the parsing to complete, check the system log to confirm that no timeout prompt corresponding to
PARSE_FILE_TIMEOUT_SECONDSappears.
The values provided are common starting points and should be measured against the reader's own samples.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-10-19.