What this category of data looks like
This category of data comes from permission-controlled internal knowledge bases of state-owned enterprises, government agencies, and public service institutions. It includes system documents, business data documents, and more. Most sources are permissioned folders from internal collaboration platforms such as Feishu and DingTalk, or locally encrypted stored documents. Updates trigger on demand when internal systems are revised or business data changes, with no fixed cycle. Document formats are mainly PDF, PPT, and Word, with some scanned copies. Document structures include fields such as chapter numbers, effective scope, and security classification. Some files come with access control tags bound to permissions.
What constraints do these characteristics impose on document parsing and chunking
This category of data’s characteristics create multiple constraints for the document parsing and chunking process. First, documents carry metadata such as permission tags and security classification. Chunking must extract and bind this metadata to matching text blocks to avoid losing permission information. Second, formats include editable documents, scanned copies, and PPT files. The parsing process must support multiple format engines. Scanned copies must trigger the OCR process, otherwise chunked content will be empty or incomplete. Third, most internal documents are long texts with clear chapter structures. Chunking must follow chapter boundaries to prevent semantic fragmentation from cross-chapter splicing. Finally, permission-controlled data sources must verify access permissions before parsing. Unauthorized files cannot start the parsing process.
How to configure these settings
Reference only; verify against the target FastGPT release and deployment documentation.
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Adapts to the chapter and paragraph length of internal system documents, prevents single block content from exceeding model context limits |
chunkSplitter | Split by chapter titles | Matches the chapter structure of internal documents, ensures complete semantic chunking, and aligns with the reading logic of internal files |
customPdfParse | Enable OCR adaptation mode | Covers scenarios such as scanned PDF and encrypted PDF, resolves issues where scanned document content cannot be synchronized |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Meets upload requirements for large business data documents or batch system files, prevents parsing failures for large files |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Reserves sufficient time for long document parsing and OCR processes, prevents parsing tasks from being interrupted by timeouts |
chunkSettingMode | Manual configuration mode | Supports adjusting chunking rules for different document formats, adapts to parsing requirements for mixed formats such as PPT and PDF |
Three common mistakes
- Issue: After syncing a Feishu knowledge base, PPT and PDF files do not display parsed content. Cause: OCR adaptation mode for
customPdfParseis not enabled. Scanned or encrypted PDF files cannot complete parsing, and PPT formats are not correctly identified. - Issue: After deploying the latest version of marker, logs show an
ocr erroralert. Cause: A reasonable duration forPARSE_FILE_TIMEOUT_SECONDSis not configured, or uploaded files exceed theUPLOAD_FILE_MAX_SIZElimit, causing the OCR process to interrupt. Or the scanned copy has too low resolution to complete text recognition. - Issue: Normal chat works normally, but model errors are triggered after adding file parsing. Cause:
chunkSizeis set too large, causing chunked content to exceed the context length supported by the model, or chunking does not follow chapter boundaries leading to semantic conflicts, making the model unable to process abnormally formatted text blocks.
How to confirm configurations are correctly set
- Upload a scanned PDF file, check if the parsed text content contains complete text information, to confirm that the
customPdfParseconfiguration is effective. - Upload a long document with more than 500 pages, verify that the parsing process does not trigger timeout errors, to confirm that the value of
PARSE_FILE_TIMEOUT_SECONDSis reasonable. - Sync PPT and PDF files from the Feishu knowledge base, confirm that parsed chunked content is displayed for all format files, to confirm that the
chunkSplitterconfiguration adapts to multi-format documents. - View chunked text blocks, confirm that each block’s length falls within the preset
chunkSizerange and that no cross-chapter splitting occurs, to confirm that the chunking rules are effective.
The values provided here are common starting points. They should be validated against the specific sample documents used in the target deployment.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-09-28.