What Data for This Category Looks Like
Data for this category primarily originates from WAF, IDS, firewalls, and threat intelligence platforms. Daily generated documents include rule change tickets, blocking logs, false positive analysis reports, and policy rollback records. Most individual data entries include fields such as rule ID, trigger condition, blocking threshold, and associated event ID. Data update frequencies are typically daily or triggered by security incidents. The number and format of fields per record vary slightly based on protection device type. Overall, most data consists of structured text and semi-structured logs.
Constraints Imposed on Vector Models and Indexing
Decentralized data sources and significant format differences require the indexing process to support parsing for multi-source heterogeneous data. Update frequencies fluctuate with security incidents. Incremental indexing must be supported to avoid resource costs from full reconstruction. False positive analysis and rule review rely on precise context recall. This places high requirements on the semantic matching accuracy of vector models. Individual records include strongly associated fields such as rule ID and trigger condition. Field-level metadata must be retained during indexing to enable precise retrieval. Index volume must also be controlled to avoid reducing real-time query performance.
How to Set Configurations
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
| Chunk Length | 800–1200 characters | Adapts to the single-paragraph text length of blocking logs and rule change tickets, avoids semantic breaks within chunks |
| Overlap Length | 100–200 characters | Preserves rule context across chunks, prevents missed associated trigger condition descriptions during retrieval |
| Similarity Threshold | 0.75–0.85 | Balances false positive blocking and false negative risks, matches the semantic matching accuracy requirements of security rules |
| Single Search Limit | Top 10 entries | Controls retrieval return volume, avoids overload while covering common false positive analysis scenarios |
defaultToken | 4096 | Adapts to long-text rule change tickets and threat intelligence reports, ensures complete context input |
| Reranking Switch | Enabled | Optimizes retrieval result sorting logic, improves associated recall efficiency for false positive samples |
Three Common Misconfigurations
- Symptom: When deploying FastGPT v4.17.0 locally, no image indexing model option appears after creating a knowledge base and uploading documents. Cause: This image indexing feature is preinstalled only in commercial deployment packages by default. Manual import of the corresponding model plugin is required for open-source local deployments.
- Symptom: When
defaultTokenis set to 2048, long rule change tickets cannot be fully loaded into context. Cause: The configuration does not match the actual document length, resulting in truncation of critical rule descriptions and reduced associated retrieval accuracy. - Symptom: An error occurs when connecting an Embedding model to OneAPI, and vectors cannot be generated. Cause: OneAPI interface address and secret key are not configured correctly, or adaptive settings for the corresponding Embedding model are not enabled in FastGPT.
How to Verify Proper Configuration
- Upload a complete rule change ticket, check if chunked text blocks retain complete rule ID and trigger condition descriptions.
- Enter query terms related to false positive analysis, confirm that the number of returned retrieval results matches the single search limit configuration.
- Test connection to the Embedding model, confirm there are no connection errors and generated vectors can be properly stored in the index.
- Upload a latest blocking log, verify that incremental indexing completes updates within the configured cycle and can be retrieved.
The values provided here are common starting points, and should be evaluated against applicable sample data.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-10-19.