What the data for this category looks like
This category’s data originates from SIEM, EDR, firewall logs, and host images. Updates occur in real time or are triggered by individual security incidents. Commonly processed documents include emergency response reports, forensic records, disposal plans, and notification materials. Most documents contain event identifiers, timestamps, operation log snippets, disposal steps, and compliance check items. The content length of a single document ranges from several pages to dozens of pages, with no unified fixed format template.
What constraints do these characteristics impose on the "document parsing and chunking" link
Data sources mix structured logs and unstructured reports, with no unified format template. The document parsing link must support multiple file formats to avoid losing log snippets and metadata. Updates occur in real time or are triggered by individual security incidents. The parsing process must limit execution time to avoid delays that affect emergency disposal timeliness. Single-document content length varies widely. The chunking strategy must balance the integrity of long log snippets and complete disposal steps. It must also avoid chunks that are too large to reduce retrieval accuracy, or too small to break the contextual association of the evidence chain.
How to configure the settings
The following configurations are based on the parameter logic of FastGPT v4.17.0, adapted to the document processing needs of security incident emergency response scenarios:
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Emergency response documents contain long log snippets and disposal steps. This range preserves contextual integrity and avoids breaking the associative logic of the evidence chain via chunking |
chunkSplitter | Split by paragraph + semantic chunking | Paragraphs in emergency response documents mostly correspond to independent disposal links or log categories. Semantic chunking improves retrieval relevance |
customPdfParse | Enabled | Default raw PDF parsing may lose embedded log screenshots and compliance check tables. Enabling custom parsing preserves complete visual content |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | A single emergency response document may contain dozens of pages of logs and reports. This duration covers the full parsing process to avoid timeout interruptions |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Forensic documents associated with large host images may occupy significant storage space. This upper limit covers the scale of conventional incident documents |
chunkSettingMode | Manual configuration | Emergency response documents have large format differences. Manual configuration allows flexible adjustment of chunking rules to adapt to different document types |
Three common errors
- Symptom: Embedded log screenshots cannot be extracted after uploading a PDF, and the image field in the parsing result is empty. Cause: The
customPdfParseconfiguration is not enabled, and the default basic PDF parsing logic is used, which cannot retain non-text visual content. - Symptom: Retrieval results only return isolated log snippets and cannot associate corresponding disposal steps. Cause: The
chunkSizevalue is too small, splitting complete disposal steps into multiple independent chunks and breaking contextual association. - Symptom: A parsing timeout error is triggered after uploading a large forensic document. Cause: The
PARSE_FILE_TIMEOUT_SECONDSvalue is lower than the time required for document parsing, and does not adapt to the content scale of emergency response documents.
How to confirm the configuration is valid
- A typical emergency response PDF document is uploaded. Images and tables in the parsing result are checked for completeness, to confirm the custom PDF parsing configuration is effective.
- Parsing chunks from a single document are extracted. The chunk length is verified against the preset range, to confirm the chunk size configuration is effective.
- A test document close to the preset upper limit is uploaded. The parsing process is verified to complete within the preset duration, to confirm the timeout configuration is effective.
- Specific log snippets in the test document are retrieved. The retrieval results are confirmed to associate corresponding disposal steps, to verify whether the chunking rule adapts to the document structure.
The values provided are common starting points and should be measured against the reader's own samples.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-10-19.