What this category of data looks like
This category of data primarily comes from SIEM, EDR, firewall logs, and host images. Updates occur in real time or are triggered by individual security incidents. Daily processed documents include emergency response reports, forensics records, disposal plans, and notification materials. A single document may contain structured log fragments, unstructured operation records, and hash verification fields. Some log fields require association with timestamps, attack source identifiers, and disposal action parameters. No unified fixed format template exists.
What constraints do these characteristics impose on the document parsing and chunking stage
Multi-source heterogeneous data formats require the parsing stage to support both structured logs and unstructured documents. This prevents loss of key associated fields. Real-time or incident-triggered updates require low latency in parsing and chunking workflows. Do not set overly long timeout thresholds. Different documents have highly varied structures. Adapt chunking rules for each document type. Retain the integrity of associated fields such as attack sources and disposal actions. This prevents chunking from breaking evidence chain logic. It also avoids harming subsequent attack path restoration and evidence chain organization work.
How to set configurations
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Adapts to the single-segment logical length of emergency response reports and forensics records. It avoids splitting key disposal steps. |
chunkSplitter | Switch based on document type | Use line-wise chunking for structured logs. Use paragraph-wise chunking for unstructured reports. This adapts to multi-format data. |
customPdfParse | Enabled | Adapts to non-standard PDF content such as embedded log screenshots and hash value text in emergency response reports. It improves parsing completeness. |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Meets upload requirements for single large host image logs or multi-page disposal plan documents. |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Meets parsing time requirements for large-volume log files. It avoids parsing failures due to timeouts. |
chunkSettingMode | Manual configuration | Adjust chunking rules separately for SIEM and EDR logs from different sources. This adapts to heterogeneous data characteristics. |
Three common configuration mistakes
- An "unsupported file type" prompt appears when uploading Excel-format forensics records. The root cause is that the
customPdfParseconfiguration is not enabled, or custom parsing rules are not configured for non-standard documents. - A parsing timeout error occurs after uploading a 3MB emergency response PDF. The root cause is that the
PARSE_FILE_TIMEOUT_SECONDSsetting value is lower than the file’s actual parsing time. It fails to match processing requirements for large-volume documents. - Retrieved chunk results lose associated attack source IP and disposal action metadata. The root cause is that chunking splits the associated fields of structured logs. It does not retain the binding logic between fields.
How to confirm configurations are properly set
- Upload a host image log file within the maximum 1000 MB supported by FastGPT v4.17.0. Confirm that the parsing task has no timeout errors.
- Upload a single SIEM structured log. Check whether the parsed chunk results retain core associated fields such as timestamps and attack source IPs.
- Switch the
chunkSplitterconfiguration. Test the chunking effects of unstructured emergency response reports and structured logs separately. Confirm that the chunking logic matches the document type. - Upload an Excel-format forensics record file. Confirm that upload and parsing complete normally, with no unsupported format prompts.
The values provided here are common starting points, and should be evaluated against the reader’s own samples.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-10-19.