This document is written based on the document parsing and chunking functions of FastGPT v4.17.0.
What data in this category looks like
Data in this category mainly comes from on-site resources such as customer equipment rooms, device lists, software packages, and network topologies. Documents processed daily include deployment manuals, installation records, configuration lists, and on-site check-in sheets. The update rhythm follows project phases: corresponding documents are added or updated when each deployment link is completed. Most document structures are step-by-step lists, parameter lists, or structured tables. Fields include device model, configuration parameters, operation steps, timestamps, and other items, with no unified standard format.
What constraints do these characteristics impose on the document parsing and chunking link
On-site deployment documents are mostly non-standard step-by-step or parameter-based content, with scattered sources. Parsing tools must adapt to mixed-format documents. Data updates follow project phases, so incremental parsing and chunking must be supported. Device lists, configuration lists, and similar documents contain a large number of structured numerical fields. Chunking must retain field correlation, to avoid splitting that breaks parameter correspondence. On-site deployment documents may contain customer-specific configurations. Parsing must ensure local data processing. Large deployment manuals and network topology diagrams may also be included in on-site documents, which sets clear requirements for parsing timeout and file upload limits.
How to set the configuration
| Configuration Item | Recommended Value | Rationale for This Value |
|---|---|---|
chunkSize | 800–1200 characters | Adapts to the length of step paragraphs and configuration parameters in deployment manuals, avoids splitting that breaks single operation logic |
chunkSplitter | Split by paragraph + title | Matches the step-by-step structure of on-site documents, retains the integrity of operation steps |
customPdfParse | Enabled | Adapts to non-text elements such as network topologies and device lists embedded in deployment manuals, ensures structured data extraction |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Accommodates large deployment manuals and complete network topology files, matches the document scale of on-site projects |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Prevents mid-parsing interruption for large documents, matches the processing duration requirements of on-site documents |
chunkSettingMode | Manual mode | Allows adjusting chunking rules for different document types such as configuration lists and check-in sheets, adapts to multi-document scenarios within this category |
Three common mistakes
- Phenomenon: Only page metadata is returned after passing a document link, with no actual document content. Cause: The
customPdfParseconfiguration is not enabled, so non-standard PDFs or deployment manuals with embedded formats cannot be parsed. - Phenomenon: The chunk ID cannot be copied on the knowledge base page. Cause: The chunk ID display switch is not enabled, or the front-end rendering logic does not grant copy permissions.
- Phenomenon: Duplicate chunks are automatically deleted after custom-chunked documents are stored in the knowledge base. Cause: The global deduplication logic is enabled by default, and the option is not disabled in the knowledge base configuration, causing associated content from custom chunks to be split and deleted.
How to confirm the configuration is correct
- Upload a typical deployment manual. Verify that parsed chunks retain complete operation step paragraphs without mid-paragraph truncation.
- Review parsing task logs. Confirm no timeout errors occur, matching the configured duration of
PARSE_FILE_TIMEOUT_SECONDS. - Access the document management page of the knowledge base. Attempt to copy any chunk ID, confirm the copy function operates properly.
- Upload a network topology file larger than 500 MB. Confirm the upload and parsing process is not interrupted.
The values provided are common starting points, and should be evaluated against samples specific to the target deployment scenario.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-10-19.