What This Category of Data Looks Like
This category of data primarily comes from databases, data dictionaries, masking tools, and access approval systems. It corresponds to daily deliverables including masking plans, field mapping tables, masking validation reports, and approval records, updated per batch project or single task. Most documents are structured tables (for example, field mapping tables include sensitive field names, masking rules, and effective scopes) or unstructured approval text. Fields are mostly technical identifiers, compliance tags, and business values, with no unified fixed units, adjusted based on project scenarios.
How These Characteristics Constrain the Document Parsing and Chunking Link
The multiple source formats of this category of data require the parsing module to support capabilities such as reading Excel worksheet names and enhanced PDF parsing. The per-project update feature means the chunking module must adapt to document spans ranging from a few pages to dozens of pages. Structured field mapping tables contain key information such as sensitive fields and rules, so the table structure and field associations must be accurately retained, avoiding chunking that splits cross-row sensitive configurations. The high risk of error consequences requires the parsing module to have timeout protection and failure retry mechanisms to prevent compliance processes from stalling due to parsing interruptions.
Configuration Settings
This configuration applies to FastGPT v4.17.0.
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Adapts to the information density per chunk of field mapping tables and plan text in masking documents, avoiding excessive sensitive fields in a single chunk leading to redundant context after parsing |
chunkSplitter | Paragraph + table recognition | Preserves the integrity of structured tables, while splitting unstructured approval text by natural paragraphs, avoiding splitting masking rule descriptions |
customPdfParse | Enabled | Adapts to PDF format documents such as masking validation reports, retains the original document layout and table structure, preventing loss of sensitive field information |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Covers the parsing time of large masking project documents (such as Excel field mapping tables with multiple sheets), avoiding parsing failure due to timeout |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Adapts to batch-imported masking project document packages, meeting the needs of batch parsing of multiple documents |
chunkSettingMode | Manual configuration | Allows adjustment of chunking parameters for different types of masking documents (such as field mapping tables, approval records) to adapt to scenario-based requirements |
Three Common Configuration Mistakes
- When
customPdfParseis not enabled, importing non-PDF format documents will not trigger enhanced parsing, resulting in failure to accurately identify sensitive fields in the document. The phenomenon is that the imported document content is empty or key fields are missing. The reason is that this enhanced function only applies to PDF format documents. - When the chunking logic is not configured to retain Excel worksheet metadata, parsing Excel field mapping tables will lose sheet names, making it impossible to distinguish masking configurations of different projects by worksheet. The phenomenon is that the parsed knowledge base only displays table content without associated sheet names.
- When the chunking logic is not adjusted to adapt to multi-column Excel documents, importing field mapping tables with multiple columns will only recognize the first two columns of data, resulting in loss of masking configuration information. The phenomenon is that the parsed knowledge base only displays the first two columns of the table. The reason is that the default chunking logic has insufficient support for multi-column Excel documents in non-CSV formats.
How to Verify Correct Configuration
- Upload a single Excel field mapping table containing multiple sheets, check whether the parsed knowledge base displays the name and corresponding content of each sheet.
- Upload a non-PDF format office document, confirm whether
customPdfParseis enabled, and check whether the parsed content completely retains the original document structure. - Import a single large masking project document, view the parsing task's time consumption logs, and confirm that no timeout error is triggered.
- Initiate an external parsing request, check whether the returned result includes the document's metadata and chunking information.
The values provided are common starting points and should be measured against the reader's own samples.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-10-19.