This document applies to FastGPT v4.17.0, and explains the document parsing and chunking steps for security and compliance material review under dataset delivery and version management scenarios.
What this category of data looks like
This category of data primarily comes from object storage, data warehouses, annotation platforms, and customer acceptance feedback. Commonly processed documents include dataset descriptions, version records, delivery lists, and acceptance reports. Data updates follow a rhythm of per-batch delivery or per-version releases. Core document fields include version number, sample difference description, delivery completeness identifier, and acceptance result. There is no unified fixed format. Some documents include digital signatures or nested complex table structures, and data volume increases with each delivery batch.
What constraints do these characteristics impose on the document parsing and chunking step
Diverse data sources require the parsing process to support file formats from different storage paths, to avoid delivery package omissions caused by path parsing errors. The version-based update feature requires chunk results to be associated with corresponding version identifiers, to prevent mixing of data from different batches. Complex tables and digital signatures in documents increase parsing difficulty. Some general parsing tools cannot directly extract content from signed PDFs. Additionally, the non-standardized structure of core document fields requires retaining contextual associations during chunking, to ensure subsequent traceability of data versions and delivery completeness information, and avoid irreproducible training results.
How to set configurations
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Adapts to the typical paragraph length of documents such as delivery lists and acceptance reports, avoiding embedding deviations caused by overloaded single chunk content |
chunkSplitter | Auto-switch based on document type | Calls corresponding parsing logic for text documents and table documents respectively, to support complex tables and signed PDF formats |
customPdfParse | Enabled | Enables custom PDF parsing to adapt to PDF files with digital signatures, resolving the issue where general parsing cannot recognize encrypted or signed document content |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Adapts to the parsing time of large delivery packages, avoiding parsing task failures caused by timeouts |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Matches the maximum file size of single-batch delivered datasets, preventing truncation of complete delivery data during upload |
chunkSettingMode | Manual configuration | Adjusts chunking parameters for delivery documents of different versions, to ensure complete retention of contextual information associated with versions |
Three common mistakes
- Failing to enable the
customPdfParseconfiguration results in failure to recognize content after uploading a PDF with a digital signature, manifested as empty text fields after parsing. - Failing to adjust the
PARSE_FILE_TIMEOUT_SECONDSvalue leads to parsing timeout when uploading delivery packages larger than the conventional size, manifested as a failed task status and an error message indicating failed file reading. - Failing to switch
chunkSplitterto adapt to table documents results in failure to correctly chunk complex nested tables, manifested as truncated table content or only retaining headers in the knowledge base.
How to confirm proper configuration
- Upload a single test PDF with a digital signature, check whether the parsing result contains complete text content.
- Upload a single test delivery package with a size close to 1000 MB, confirm that the parsing task completes within 600 seconds with no timeout errors.
- Upload a delivery list document containing complex nested tables, check whether table content in the knowledge base is completely retained and formatted correctly.
- View the metadata of chunk results, confirm that each chunk is associated with a corresponding version identifier field, enabling traceability of data batches.
The values provided are common starting points and should be evaluated against the reader's own samples.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-10-19.