What the data for this category looks like
Data for this category originates from business systems, App permission lists, privacy policies, and user request records. Update frequency is quarterly or with each product version change. Commonly processed documents include personal information inventories, data processing rules, personal information impact assessment reports, and user authorization texts. Most documents are structured tables or long text passages, with fields including information subject, processing purpose, legal basis, storage period, and other relevant fields. There is no unified fixed format, and some documents have traces of multiple version iterations.
Constraints imposed on document parsing and chunking by these characteristics
Documents contain mixed structured tables and long text content. Parsing must balance table cell splitting and the semantic integrity of paragraphs. Documents with multiple version iterations have repeated content or field shifts, so chunking must avoid splicing content across versions. Update frequencies are irregular and the number of documents processed per batch is large, so parsing tools must support batch processing and set reasonable timeout thresholds. Given the presence of identifiable natural person information, chunked content must retain information subject association fields to avoid breaking the contextual connection required for compliance review.
How to set configurations
These configurations are based on FastGPT v4.17.0's document parsing tool:
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Adapts to the semantic unit length of documents such as personal information inventories and impact assessment reports, avoiding splitting that breaks the contextual connection required for compliance review |
chunkSplitter | Split by paragraph + heading | Adapts to the feature of heading hierarchies and structured paragraphs in documents, retaining the scenario association logic required for compliance review |
customPdfParse | Enable structured table parsing | Adapts to scenarios where documents contain personal information inventory tables, avoiding incorrect splitting of table content into scattered text |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Meets batch processing needs for uploading multiple privacy policies and user authorization texts at one time |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Meets the complete parsing duration of long documents such as annual impact assessment reports, avoiding mid-parsing timeout interruptions |
chunkSettingMode | Manual configuration | Allows adjustment of chunking parameters for different types of compliance documents, adapting to diverse compliance material formats |
Three common mistakes
- After passing in an external link, only page metadata is returned, and the document body content cannot be parsed. The cause is that the
customPdfParseparameter is not configured correctly, or the page pointed to by the link has anti-crawling mechanisms that cause parsing failure. - The knowledge base page cannot copy a single chunk ID. The cause is that the front-end rendering logic does not enable the copy interaction for this field, and the metadata display switch is not turned on in the configuration.
- Custom-split document chunks are automatically deleted after being stored in the knowledge base. The cause is that the system enables document chunk deduplication logic by default, and the check is not turned off in
chunkSettingMode, resulting in a mismatch with the order of custom splits.
How to confirm configurations are properly set
- Upload a typical personal information inventory document, and check if the parsed chunks retain the structured content of the table without scattered split cell text.
- View the parsing log to confirm that the
PARSE_FILE_TIMEOUT_SECONDSparameter is in effect, and no timeout error occurs. - Copy a single chunk ID, and verify that the front-end interface enables the copy interaction function for this field.
- Upload a custom-split document, and check if all custom chunks are retained in the knowledge base, with no automatic deduplication and deletion.
The values provided are common starting points and should be measured against the reader's own samples.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-10-19.