What the data for this category looks like
Data for this category primarily comes from SIEM, EDR, NDR, and threat intelligence sources, with update frequencies of real-time or daily. Commonly processed documents include alarm disposal sheets, duty logs, threat intelligence notifications, and monthly work reports. Most document structures include fields such as alarm ID, trigger time, associated IP, disposal steps, and threat level. Some tabular documents store standardized fields classified by event type.
What constraints these characteristics impose on the document parsing and chunking link
Real-time updated data sources require the parsing link to support fast batch processing, to avoid delays that impact alarm judgment and false positive elimination. Scenarios with mixed document types must adapt to different text and table structures, to prevent chunking from losing key associated fields. Documents such as alarm disposal sheets contain step-by-step content, so chunking must retain contextual connections, to avoid breaking disposal logic. Daily updated weekly report and notification documents must support batch parsing and time-based archiving, to enable subsequent retrieval and review.
How to set the configurations
The following configurations are based on FastGPT v4.17.0.
| Configuration Item | Recommended Value Range | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Adapts to the field length of documents such as alarm disposal sheets and duty logs, retains key contextual associations |
chunkSplitter | Chunk by paragraph + semantic meaning | Adapts to the multi-step, strongly associated content structure of SOC documents, avoids splitting disposal logic |
PARSE_FILE_TIMEOUT_SECONDS | 120–180 seconds | Prevents parsing failures due to timeout when processing batch threat intelligence notification documents |
customPdfParse | Enabled | Supports parsing PDF alarm disposal sheets with embedded tables, improves the completeness of document content extraction |
UPLOAD_FILE_MAX_SIZE | 500 MB | Supports batch import of archived files from threat intelligence sources, meets daily data processing requirements |
chunkSettingMode | Manual configuration | Adapts to the differentiated chunking needs of multiple document types, allows flexible adjustment of chunking rules |
Three common mistakes
- Symptom: File upload followed by parsing timeout, with request failure displayed. Cause: The
PARSE_FILE_TIMEOUT_SECONDSparameter was not adjusted, and the timeout period was not extended for large files or batch documents. - Symptom: Partial data is missing after uploading a tabular dataset to the knowledge base. Cause:
customPdfParsewas not enabled, or the chunking configuration was not adapted to table fields, resulting in truncated or omitted table content. - Symptom: The imported and exported knowledge base does not match the original instance. Cause: Parsing configurations and chunking rules were not synchronized, only the original files were exported without completing full configuration migration.
How to confirm that configurations are properly set
- Upload a single PDF alarm disposal sheet, check whether the parsed text contains complete core fields such as alarm ID and disposal steps.
- Submit a batch of threat intelligence notification files, verify whether the parsing task completes within the preset timeout period.
- Import a tabular dataset, check that the number of imported fields matches the original file.
- Adjust the
chunkSizeparameter, check whether the length of the chunked text falls within the preset range.
The values provided are common starting points and should be validated using samples relevant to the specific use scenario.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-10-19.