What data in this category looks like
Data for this category comes from data asset inventory results, database metadata, business system documentation, and public regulatory documents. Daily processed documents include data catalogs, classification and grading rule tables, important data identification lists, and review minutes. Update frequency is quarterly, or triggered when business systems change. Most documents are structured tables, formatted rule descriptions, or review records. Fields include asset ID, data category, sensitivity level, control requirements, and more. There is no unified fixed unit; field units adjust based on document type.
Constraints Imposed on the Document Parsing and Chunking Link
This category’s data includes structured table catalogs, rule tables, and long-paragraph review minutes. If chunking logic does not adapt to table structures, field splits will break, and data association relationships will be lost. Chunking long review minutes by fixed character counts may split the logic of complete review opinions, reducing the matching accuracy of subsequent classification and grading rules. Quarterly batch-updated documents require support for batch parsing. Single-document timeouts will slow the overall process. Confluence documents in internal network deployment scenarios need to adapt to private data source parsing rules to avoid parsing failures.
How to Set Configurations
Configuration settings below are based on FastGPT v4.17.0.
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Adapts to the structured tables and long review minutes of this category, avoiding splitting key fields or breaking review logic |
chunkSplitter | Mixed table + paragraph splitting | Adapts to structured tables for data catalogs and rule tables, as well as long paragraphs of review minutes, preserving data association relationships |
customPdfParse | Enabled | Adapts to batch PDF annual reports and rule documents, preserving the original format of tables and text |
PARSE_FILE_TIMEOUT_SECONDS | 300–600 seconds | Adapts to 1000+ page PDF annual reports and large Excel data lists, avoiding parsing timeouts |
chunkSettingMode | Manual mode | For Excel data with 10000+ rows, custom chunking rules can be defined to avoid oversized single chunks |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Adapts to the upload needs of large batch documents, meeting batch processing for quarterly data asset inventories |
Three Common Configuration Mistakes
- Internal network Confluence pages cannot extract content:
customPdfParseor private data source parsing rules are not configured, causing private format documents to fail to be recognized. - Large-volume Excel files return incomplete prompts after parsing:
UPLOAD_FILE_MAX_SIZEis set too small, orPARSE_FILE_TIMEOUT_SECONDSconfiguration is insufficient, causing large tables to terminate parsing before completion. - 10000+ row Excel data chunks are overly coarse:
chunkSettingModeis not switched to manual mode, and default chunking rules are used to split large tables, causing single chunks to contain too many rows of data, making accurate matching of classification and grading rules difficult.
How to Verify Correct Configuration
- Upload a single 1000-page PDF annual report, check if parsing completes within the time configured by
PARSE_FILE_TIMEOUT_SECONDS, with no timeout errors. - Import a structured data catalog Excel file, check if parsed chunks preserve association relationships for fields such as asset ID and data category, with no split breaks.
- Upload internal network Confluence format review minutes, confirm the parsing result contains complete review opinion content, with no missing fields.
- Adjust
chunkSettingModeto manual mode, split 10000+ row Excel data, check if the size of each chunk meets the matching requirements of classification and grading rules.
The values provided are common starting points and should be evaluated against the reader’s own samples.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-10-19.