What this category of data looks like
Data in this category mainly comes from local public data open platforms, integrated big data platforms, departmental business databases, and electronic certificate repositories. The update cycle is monthly. Commonly processed documents include public data resource catalogs, data open lists, data item descriptions, desensitization rules, interface documents, and update frequency descriptions. Most documents contain structured fields such as data item codes, open conditions, interface addresses, and desensitization methods. Some documents include batch data tables and permission description items.
What constraints do these characteristics impose on the document parsing and chunking link
The monthly batch update feature requires parsing and chunking to support batch task scheduling, to avoid overload during single processing. Documents include structured fields and tables, so chunking must retain field associations, and must not split cross-page table cells or complete data item descriptions. Code snippets and permission clauses in interface documents must be fully retained, to prevent critical configuration content from being truncated during chunking. Document formats submitted by different departments are inconsistent, so multiple parsing templates must be supported to avoid parsing failures due to format differences. Desensitization rule documents involve sensitive information boundaries, so chunking must ensure that the correspondence between permission descriptions and data items is not broken.
How to configure settings
Reference only; verify against the target FastGPT release and deployment documentation.
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Adapt to documents with structured fields and tables, avoid splitting complete data items or interface descriptions |
chunkSplitter | Split by heading + paragraph | Match the hierarchical structure of data items and rule descriptions in documents, retain contextual associations |
customPdfParse | Enable table recognition mode | Accurately extract batch data tables in documents, prevent incorrect splitting of table content |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Adapt to batch catalog lists and interface document packages submitted by departments |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Meet the parsing time requirements for large batch documents, avoid timeout interruptions |
chunkSettingMode | Manual configuration | Adjust chunking parameters for different document types such as catalog lists and interface documents |
Three common misconfigurations
- Symptom: After uploading a PDF or catalog document, the data processing section displays empty content, and no results appear in search tests. Cause: The table recognition mode of
customPdfParseis not enabled, so batch table content is not correctly extracted. - Symptom: Some PDF files parse to empty content, while others can be recognized normally. Cause: Document formats are inconsistent, the applicable parsing template is not switched, and encrypted documents or documents with complex headers and footers are not correctly parsed.
- Symptom: After local deployment, there is no progress feedback during data processing during training, but the conversation function operates normally. Cause: A reasonable duration for
PARSE_FILE_TIMEOUT_SECONDSis not configured, and batch document parsing times out without triggering a prompt.
How to confirm the configuration is correct
- Upload a single interface document, check if the parsed text retains complete interface addresses and parameter descriptions, and confirm that key content is not truncated during chunking.
- Upload a batch catalog list document, verify that all structured fields are included in the parsed data, with no missing or split errors.
- Adjust the
chunkSizeparameter, re-upload the test document, compare the completeness of the chunking results, and confirm they match the structural characteristics of business documents. - Upload a maximum-size document package, verify that the
UPLOAD_FILE_MAX_SIZEandPARSE_FILE_TIMEOUT_SECONDSconfigurations meet upload and parsing requirements.
The values provided here are common starting points, and should be measured against the reader’s own sample data.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-09-28.