What data falls under this category
Data for this category primarily comes from local public data open platforms, integrated big data platforms, departmental business databases, and electronic credential repositories. The update frequency is monthly. Processed documents include public data resource catalogs, data open lists, data item descriptions, desensitization rules, interface documentation, and update frequency descriptions. Some documents use structured table formats, while others are instructional documents with standardized titles. Core fields include data item name, open scope, desensitization requirements, interface address, and update cycle, among others.
How these characteristics impact document parsing and chunking
Data sources and document formats for this category are diverse. They include structured table-based catalogs and standardized instructional PDF documents. This requires the parsing process to support multi-format input. The monthly update feature requires chunking to retain context related to update cycles, to avoid mixing data across cycles. Core fields such as desensitization requirements and open scope must be fully retained within a single chunk. Arbitrary splitting of these fields will reduce the accuracy of subsequent data authorization and open operations. Additionally, the fixed structure of interface documentation must be recognized. Chunking must be divided by interface units to ensure semantic integrity after embedding.
Configuration Settings
Reference only; verify against the target FastGPT release and deployment documentation.
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | The description length of individual data items in public data catalogs is moderate, preventing chunking across data items and ensuring semantic integrity |
chunkSplitter | Split by title + paragraph | Most documents in this category have standardized titles, and splitting by title preserves complete units of data items and interface descriptions |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Parsing large interface documents or batch-exported catalog files takes a long time; this setting avoids timeout interruptions to the parsing process |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Public data catalogs may include batch-exported structured data files, so this setting accommodates large file upload requirements |
customPdfParse | Enabled | Desensitization rules and interface documents are mostly in standard PDF format; custom parsing accurately extracts structured fields and title information |
chunkSettingMode | Manual mode | Chunking rules must be adjusted according to data open units to adapt to different types of public data documents |
Three Common Misconfigurations
- Symptom: After uploading an Excel-format public data catalog file, the data processing steps display as empty. Cause: Field mapping rules for Excel parsing are not configured correctly. Excel files in this category often contain multi-header nesting or hidden authorization fields. The default parsing rules cannot recognize standard data item columns.
- Symptom: After uploading a batch data open list, vectorization takes an excessive amount of time. Cause: The
chunkSizeparameter is not adjusted. Oversized chunks lead to high embedding computing load, or chunk optimization configuration is not enabled. - Symptom: When submitting a non-document query, the system triggers the document parsing process. Cause: The
trainingTypeconfiguration is set to full parsing mode, causing non-document inputs to also trigger parsing logic.
How to Verify Correct Configuration
- Upload a PDF file containing a single data item description. Review the parsed chunk content to confirm that core fields such as desensitization requirements and open scope are not missing.
- Upload an Excel data catalog file of approximately 100 MB. Confirm that the parsing steps do not show timeout errors, and that data processing results are generated normally.
- Submit a non-document query text. Confirm that the system does not trigger the document parsing process, and directly enters the conversation logic.
- Adjust the
chunkSizeparameter to the preset range, then re-upload the interface document. Check whether the chunk length meets expected requirements.
The values listed here are common starting points, and should be evaluated against your own sample data.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-09-28.