What this category's data looks like
The data sources for unit-based system retrieval come from internal system documents of central and local state-owned enterprises, government departments, and public service institutions, including multiple formats such as PDF, Word, and CSV. Data updates occur irregularly alongside the processes of revising, repealing, and releasing new systems, with no fixed cycle. A single document usually includes structured fields such as issuing organization, system number, effective date, scope of application, and main body clauses. Document formats and field order vary across different organizations. Some documents also include attachments, tables, and hierarchical chapter structures.
What constraints these characteristics impose on the document parsing and chunking workflow
The multi-format data sources of unit-based systems require the parsing workflow to support adaptation to multiple file types, to avoid parsing failures caused by format incompatibility. Differences in document formats and fields across different organizations require chunking rules to be flexible, and cannot rely on a single fixed splitting logic. Irregularly updated data requires the parsing workflow to support incremental synchronization and fast re-parsing, to ensure the timeliness of retrieval content. Structured fields and hierarchical chapters require the chunking workflow to retain key metadata and structural information, to avoid damaging contextual associations during retrieval.
How to configure these settings
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | The main body of unit-based systems is relatively long. This range balances the completeness of single-block context and retrieval efficiency |
chunkSplitter | Split by title + paragraph | Unit-based systems generally have clear chapter titles. Retaining title-associated paragraphs can improve the semantic accuracy of retrieval results |
customPdfParse | Enabled | Adapts to the unit identifiers and document number headers/footers often found in unit-based system PDFs, and allows custom filtering of irrelevant content |
PARSE_FILE_TIMEOUT_SECONDS | 300–600 seconds | Parsing large system PDF files takes a long time. This range avoids interrupting the parsing process due to timeout |
chunkSettingMode | Manual configuration | The system formats of different organizations vary significantly. Manual adjustment can improve the adaptability of chunking rules |
Three common misconfigurations
- Phenomenon: Parsing fails after uploading CSV format system files, with format error logs returned. Cause: After upgrading to FastGPT v4.17.0, the default CSV parsing rules were adjusted, and the column separator configuration of older versions of CSV was not adapted.
- Phenomenon: When calling the knowledge base file collection creation API interface, parsing parameters are not specified, and table content is lost in PDF parsing results. Cause: No
customPdfParserelated parameters are configured in the request, and the default parsing logic does not retain structured table content. - Phenomenon: After attempting to connect a third-party document parsing tool, the parsing results are not correctly synchronized to the chunking workflow. Cause: The callback address of the third-party parsing service is not specified in the
customPdfParseconfiguration, causing the process to interrupt.
How to confirm the configuration is correct
- Upload a single typical unit-based system PDF file, view the parsed text content, and confirm that the unit identifiers in the headers and footers have been filtered out.
- View the chunked result list, confirm that the length of each chunk is within the configured range, and that each chunk is associated with the corresponding chapter title.
- Call the knowledge base file collection creation API interface, carry the
customPdfParseparameter, and verify that the returned parsing results include the expected tables or structured content. - Upload system files exceeding the default upload size, confirm that the upload and parsing processes do not trigger timeout or size limit error messages.
The values provided here are common starting points, and should be evaluated against the user's own samples.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-09-28.