What the data for this category looks like
Common data sources include crowdsourcing platform backends, annotation quality statistical reports, annotator training materials, and financial settlement systems. Daily processed documents include personnel access rule documents, annotation training courseware, monthly performance statistics tables, and settlement detail lists. Common data fields include personnel identifiers, task completion volume, quality pass rate, settlement amount, and others. Update cycles mostly involve daily synchronization of that day’s annotation task data, with monthly updates for performance and settlement data. Document formats include PDF, structured HTML, Excel export files, and other types.
What constraints these characteristics impose on the "document parsing and chunking" link
Diverse document types with structured fields require parsing and chunking logic to adapt to multiple formats including PDF, HTML, and Excel. The logic must also preserve associations between fields such as personnel identifiers and settlement amounts, to avoid data breaks after chunking. Short documents updated daily need to fit fast parsing workflows. Long monthly documents require reasonable chunk size control to retain context integrity. Some documents also include interface instructions for annotation tools. This requires precise parsing and chunking support for HTML format, otherwise subsequent Q&A will fail to obtain valid information.
How to set configurations
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Adapts to long text documents such as performance tables and settlement lists, preserves field associations and prevents chunk breaks |
chunkSplitter | Split by paragraph + title | Adapts to documents with training titles and personnel entries, ensures complete semantic integrity of chunks |
customPdfParse | Enabled | Adapts to PDF-format training materials and access rules, retains original layout and image-associated content |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Adapts to monthly batch settlement document upload requirements, prevents large file upload failures |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Adapts to long document parsing time requirements, prevents mid-parsing timeout interruptions |
chunkSettingMode | Manual configuration | Adapts to personalized chunking needs for different document types, allows flexible parameter adjustment |
Three common mistakes
- Symptom: After using
chunkmode to call the pushdata API to upload documents, the index status remains stuck with no progress for a long time. Cause: ThePARSE_FILE_TIMEOUT_SECONDSparameter is not configured, or its value is set too short, causing long document parsing to time out before completion. - Symptom: After uploading PDF-format personnel training materials, images within the documents are not extracted or displayed. Cause: The
customPdfParseconfiguration is not enabled. The default PDF parsing logic does not support extracting image-associated content. - Symptom: After uploading javadoc-generated HTML-format interface documents, no valid parsing results are obtained. Cause: The
chunkSplitteris not configured to adapt to HTML tag structures. The default chunking logic cannot recognize interface description paragraphs within HTML.
How to confirm configurations are set correctly
This verification is based on FastGPT v4.17.0:
- Upload a single PDF training material under 100 MB, check if the parsing result includes image-associated text and original title structure, to confirm the
customPdfParseconfiguration is active. - Call the pushdata API to upload test documents, monitor parsing time, to confirm the
PARSE_FILE_TIMEOUT_SECONDSvalue is greater than actual parsing time. - Upload javadoc-format HTML interface documents, check if the parsed chunks include interface description paragraphs, to confirm the
chunkSplitterconfiguration adapts to HTML structures. - View the knowledge base upload records, confirm that documents do not remain in index status for long periods, and that parsing result fields are complete with no breaks.
The values provided here are common starting points, and should be evaluated against the reader's own samples.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-10-19.