What the data for this category looks like
Core data for this category originates from WAF, IDS, firewalls, and threat intelligence platforms. Four standard document types are produced regularly: rule change tickets, blocking logs, false positive analysis reports, and policy rollback records. Data updates occur daily or when triggered by a single event. Documents typically include fields such as unique identifier, effective time, rule content, associated business scope, and false positive/interception rate. No unified mandatory format exists, and most documents are structured tables or semi-structured text.
What constraints do these characteristics impose on document parsing and chunking?
Scattered data sources and inconsistent formats require the parsing link to support multiple non-standard text structures. Single-event triggered updates create high-frequency upload demands, so single-file parsing timeout and size limits must be controlled. Fields including false positive rate and rule effective time in documents have strict chunking accuracy requirements. Contextual association information must be retained to avoid losing the binding relationship between rules and their associated business scope after splitting. Metadata such as timestamps and rule IDs in blocking logs must be extracted alongside the main text, to enable quick location of corresponding policies during subsequent retrieval.
How to set configurations
The following configurations are designed for the document parsing module of FastGPT v4.17.0.
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunkSize | 800–1200 characters | Security product documents often contain rule text and associated metadata. This range retains the complete context of a single rule, preventing rules from becoming disconnected from their associated business scope after splitting |
chunkSplitter | Split by title + paragraph | Adapts to common chapter title and rule entry structures in documents, ensuring chunked content maintains logical coherence |
customPdfParse | Enabled | Most security product documents are exported as PDF format. Enabling this allows accurate extraction of text and metadata, avoiding format confusion |
UPLOAD_FILE_MAX_SIZE | 1000 MB | Supports batch uploading of blocking logs and false positive analysis reports, preventing upload failures for large files |
PARSE_FILE_TIMEOUT_SECONDS | 600 seconds | Reserves sufficient parsing time when processing documents with large numbers of log entries, preventing timeout interruptions |
chunkSettingMode | Manual configuration | Allows flexible adjustment of chunking parameters for the special structure of security documents, adapting to different types of operation documents |
Three common configuration mistakes
- Symptom: After uploading a doc or docx format file in a server environment, the document parsing node returns a 404 error and cannot read the file content. Cause: The local environment can access the local file path, but the server environment lacks proper file storage path mapping, so the parsing node cannot obtain the actual address of the uploaded file.
- Symptom: RAG retrieval results only contain main text content and do not include chapter title information. Cause: The chunking configuration does not enable title-associated extraction, or
chunkSplitteris not set to split by title, causing chapter titles and main text to separate. - Symptom: After batch uploading multiple blocking logs, some documents fail to parse due to timeout. Cause: The
PARSE_FILE_TIMEOUT_SECONDSparameter is not adjusted, and the default timeout duration is insufficient for processing large-capacity log documents.
How to verify correct configuration
- Upload a typical rule change ticket document, and check if the parsed text fully retains rule content and associated business scope.
- Check the parsing node’s running logs to confirm no 404 or timeout errors, and verify that
UPLOAD_FILE_MAX_SIZEandPARSE_FILE_TIMEOUT_SECONDSconfigurations match current upload requirements. - Trigger a RAG retrieval, confirm returned results include both main text and corresponding chapter titles, and verify that
chunkSplitterand chunking context configurations are active. - Upload security documents in different formats (such as PDF, docx), and confirm that the
customPdfParseconfiguration can properly parse all supported file formats.
The values provided are common starting points and should be measured against the reader's own samples.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-10-19.