What this category of data looks like
Typical entities include emergency management bureaus, water affairs bureaus, meteorological bureaus, sub-districts, towns and townships, and emergency command centers. Daily processed documents include duty logs, early warning notices, consultation minutes, rapid disaster reports, emergency response plans, material allocation slips, and more. Data sources cover meteorological early warning systems, hydrological monitoring systems, emergency command platforms, grassroots disaster reporting systems, and video surveillance. Data is updated daily. Document fields include rainfall and water regime data, danger level, material quantity, evacuated personnel information, and other items, paired with professional units such as millimeters, cubic meters per second, and number of people.
What constraints do these characteristics impose on the document parsing and chunking step?
Multi-source heterogeneous data sources lead to large differences in document formats. There are both structured table-style material allocation slips, as well as rich-text consultation minutes and disaster rapid reports with images. This requires the parsing link to support text extraction, table structuring, and image content recognition simultaneously. Daily updated real-time data requires chunk boundaries to match document timestamps, to avoid mixing cross-batch data and affecting retrieval accuracy. The need to retain professional fields and units requires that chunking does not destroy the integrity of single records, otherwise key information such as rainfall and water regime will be split. In addition, documents in emergency scenarios may contain large-volume monitoring data files, which puts higher requirements on parsing timeout settings.
How to set the configuration
Reference only; verify against the target FastGPT release and deployment documentation.
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Emergency documents contain professional fields and units. Chunks that are too long will lose contextual association, and chunks that are too short will destroy the integrity of disaster data |
chunkSplitter | Split by paragraph + table header | Emergency documents often contain tables, early warning titles, and consultation items. Splitting by paragraph can retain complete information for a single early warning or single disaster record |
customPdfParse | Enable image text extraction + table structuring | Avoid ignoring on-site disaster images and material tables uploaded by users, matching the multimodal requirements of emergency documents |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Some large-volume CSV or PDF consultation minutes exported from hydrological monitoring take a long time to parse. Reserve sufficient time to avoid parsing failures |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Adapt to large-volume CSV and compressed package documents exported from hydrological monitoring, avoid interception of large file uploads |
chunkSettingMode | Manual configuration | Adapt to document formats from different sources (such as short-text meteorological early warnings, long tables of material allocations), avoid automatic chunking destroying professional content structure |
Three common mistakes
- When uploading a PDF document containing on-site disaster images, the parsed text blocks do not include image-related descriptions, and knowledge base retrieval fails to recall image content. The cause is that the image text extraction configuration of
customPdfParseis not enabled. The default parsing rule only extracts pure PDF text content. - When uploading HTML interface documents generated by Javadoc, knowledge base retrieval has no matching results, and the parsed text blocks are empty or only contain a small amount of irrelevant tag content. The cause is that the default parsing rule does not adapt to the tag nesting logic of structured HTML, and cannot extract parameters and description text in the interface documents.
- Chunk results contain redundant data across dates, and retrieval results include both early warning notices from 3 days ago and daily rainfall and water regime data. The cause is that chunk boundary control is not performed according to the document release time field, leading to incorrect merging of cross-period documents.
How to verify the configuration is correct
- Upload a single PDF document containing on-site disaster images, check whether the parsed text blocks include on-site descriptive text from the image.
- Upload a material allocation slip document containing a table, check whether the parsing result retains the row and column structure of the table and corresponding field values.
- Upload a 10MB hydrological monitoring CSV file, check that the parsing process does not trigger a timeout error.
- Manually adjust the
chunkSizeparameter, then upload a short-text early warning document, check that the chunk length matches the set character range.
The values provided here are common starting points, and should be evaluated against the user's own samples.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-09-28.