What the data for this category looks like
This category's data primarily comes from WAF, IDS, firewalls, and threat intelligence platforms. Daily processed documents include rule change tickets, blocking logs, false positive analysis reports, and policy rollback records. Most documents contain fields such as rule identifiers, trigger conditions, blocking thresholds, and analysis times. Data update frequency is daily or event-triggered. Individual document sizes range from a few lines to dozens of pages. Some log-type documents are generated in batches.
What constraints these characteristics impose on the document parsing and chunking link
Diverse data sources lead to significant differences in document formats. There are both structured log-type documents and unstructured rule change tickets, which requires parsing adaptability. Batch-generated log-type documents have repeated fields and formats, so chunking must avoid splitting cross-event related content. Daily or event-triggered updates require the parsing process to support high-frequency access, and timeout risks must be managed in advance. False positive analysis and policy rollback records contain sensitive protection threshold information. Chunking must ensure complete contextual association, avoiding splitting critical thresholds into different chunks.
How to set the configurations
Configuration items under FastGPT v4.17.0 are listed below:
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Adapts to the field density of blocking logs and rule change tickets, avoiding splitting the trigger conditions of a single rule into different chunks |
chunkSplitter | Split by paragraph + title | Matches the structure of rule IDs and change descriptions in documents, preserving contextual association |
customPdfParse | Enabled | Supports parsing policy documents in digitally signed PDF format, resolving the issue where signed files cannot have their content recognized |
UPLOAD_FILE_MAX_SIZE | 100 MB | Adapts to the upload requirements of batch blocking log packages, avoiding parsing failures for large files |
PARSE_FILE_TIMEOUT_SECONDS | 300 seconds | Covers the time required for parsing merged multiple log files, avoiding interrupting the parsing process due to timeout |
chunkSettingMode | Manual configuration | Adjusts chunking parameters for different document types (logs/reports) to adapt to the diverse document structures within this category |
Three common mistakes
- Symptom: An error occurs when reading a file using the Doc2x tool, with a prompt indicating only specified formats are supported. Cause: The
customPdfParseconfiguration is not enabled, or the uploaded file format exceeds the preset supported range, and the non-standard format of security product documents is not adapted to. - Symptom: Digitally signed PDF policy documents cannot have their content recognized after upload. Cause: The
customPdfParseconfiguration is not enabled. The default parsing engine cannot process signed PDF files, resulting in failed content extraction. - Symptom: The access link returned by the API knowledge base cannot be accessed normally, with a prompt indicating permission errors or missing content. Cause: The
UPLOAD_FILE_MAX_SIZEconfiguration is not set correctly, leading to incomplete file uploads, or thePARSE_FILE_TIMEOUT_SECONDSsetting is too short. The parsing process is not completed before the link is returned, resulting in incomplete content generation.
How to confirm the configurations are set correctly
- A single rule change ticket document is uploaded, and the parsed chunks are checked to retain complete rule IDs and change description paragraphs, with no critical information split across chunks.
- A batch blocking log package is uploaded, and the parsing time is confirmed to meet expectations, with no timeout errors, and a single chunk does not contain unrelated cross-event logs.
- A digitally signed PDF policy document is uploaded, and it is verified that complete protection rule content can be extracted after parsing.
- The API is called to upload a test file, and the returned parsing result is checked to have complete fields, with no missing rule thresholds or analysis time information.
The configuration values listed above are common starting points. They should be evaluated against the reader's own sample data.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-10-19.