What the data for this category looks like
This category’s data primarily comes from SIEM, EDR, NDR, and threat intelligence sources, with update cycles split into real-time and daily. Daily processed documents include alert disposition sheets, duty logs, threat intelligence bulletins, daily and weekly reports. Most documents mix structured fields and free text. Fields include alert ID, disposition time, threat level, associated IP, and disposition measures. There is no unified fixed format template.
What constraints do these characteristics impose on the document parsing and chunking link?
The data characteristics of this category create multiple constraints for parsing and chunking. Documents mixing structured and free text require simultaneous extraction of structured fields and semantic-coherent free text chunking. Real-time updated alert disposition sheets require the parsing process to have low latency, to avoid timeouts that disrupt duty response. Significant format differences exist across multi-source heterogeneous documents, so parsing rules must be adapted for different source documents. Long-cycle documents such as daily and weekly reports require reasonable control of chunk length, to retain contextual association information needed for threat analysis.
How to set configurations
The following configurations are written based on the configuration logic of FastGPT v4.17.0.
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Alert disposition sheets and threat intelligence bulletins in SOC scenarios mostly mix short paragraphs and long text. This range can retain the complete semantics of a single alert or single threat intelligence entry |
chunkSplitter | By paragraph + semantic boundary | For documents mixing structured and free text, splitting by paragraph preserves the textual context linked to structured fields. Splitting by semantic boundaries avoids breaking long sentences |
PARSE_FILE_TIMEOUT_SECONDS | 30–60 seconds | Real-time updated alert data requires fast parsing. Excessive timeout will negatively impact SOC duty response efficiency |
customPdfParse | Enabled | Threat intelligence bulletins are mostly in PDF format. Enabling custom parsing adapts to the specific layout of such documents, improving structured field extraction accuracy |
UPLOAD_FILE_MAX_SIZE | 100 MB | This matches the common maximum size for long documents such as daily and weekly reports, preventing parsing failures from overly large files |
chunkSettingMode | Manual configuration | Document formats vary significantly in SOC scenarios. Manual configuration allows precise matching of document characteristics from different sources |
Three common mistakes
- Phenomenon: Some uploaded PDF files display empty content, while others are recognized normally. Cause: The
customPdfParseconfiguration is not enabled, or the PDF uses non-standard encryption or embedded fonts, preventing the parsing engine from reading text content. - Phenomenon: Import a Java interface file by changing its file suffix to .txt, and no data is parsed. Cause: Java interface documents contain specific syntax structures. Plain text parsing cannot recognize structured code blocks and interface definitions, and no custom parsing rules are used to adapt to code-type documents.
- Phenomenon: After using chunk mode to call the pushdata API for upload, the interface continuously shows indexing in progress. Cause: The
PARSE_FILE_TIMEOUT_SECONDSconfiguration value is too low, or chunk parameters are set too large, leading to excessively long parsing time and failure to complete the indexing process.
How to confirm the configuration is properly set
- Upload typical documents for this category, verify that parsed chunks retain associated text for core business fields such as alert ID and threat level.
- View parsing logs to confirm there are no timeout-related errors, and that parsing time meets business response requirements.
- Upload multi-source heterogeneous documents to verify parsing integrity across different document formats.
- After adjusting chunk parameters, confirm that no critical semantic breaks appear in chunks of long documents.
The values provided are common starting points and should be measured against the reader's own samples.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-10-19.