What the data for this category looks like
Data sources for cybersecurity intelligent due diligence reports include asset scan logs, vulnerability scan reports, threat intelligence feeds, compliance audit records, and attack incident review materials. Update rhythms vary by data source type: threat intelligence updates hourly, asset ledgers and regular scan reports update on a weekly or daily cycle, and incident review materials are archived once. Each individual report has a document structure with two parts: structured fields and unstructured text. Structured fields include asset_ip (IPv4/IPv6 format string), cve_id (standard CVE number string), risk_level (enumerated values: low/medium/high/critical), and scan_time (ISO 8601 format timestamp). Unstructured text includes vulnerability remediation suggestions, attack incident timelines, and analysis descriptions. No custom units are used.
What constraints these characteristics impose on vector models and indexing
Differing update rhythms across data sources require indexes to support incremental construction and scheduled synchronization. This avoids resource waste and delays caused by full reindexing. The mixed structured and unstructured document structure requires both exact retrieval indexes and vector indexes. Relying only on vector calculations prevents precise matching of critical structured information. Coexisting long and short content types require different segmentation and encoding strategies. This prevents semantic loss in short fields or vector dimension overload in long texts. The real-time requirements for threat intelligence require controlling vector index update latency within reasonable limits. This avoids retrieving outdated security data.
How to set configurations
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
chunk_size | 800–1200 characters | Cybersecurity due diligence reports include long-form vulnerability analysis and attack incident descriptions. This range preserves semantic integrity while avoiding vector dimension overload |
incremental_index_enabled | enabled | Cybersecurity data includes real-time updated threat intelligence and periodic scan reports. Incremental indexing avoids resource consumption from full reindexing |
vector_model | text-embedding-ada-002 or open-source models of comparable scale | Meets semantic matching needs for unstructured text such as remediation suggestions and incident reviews. This model covers semantic understanding for general security scenarios |
structured_index_fields | asset_ip,cve_id,risk_level,scan_time | Structured fields in reports can be quickly filtered via exact retrieval. This eliminates reliance on vector calculations to improve retrieval efficiency |
recall_top_k | top 10 results | The number of risk events associated with a single due diligence report is limited. Too many retrieved results increase subsequent reranking burden |
similarity_threshold | calibrated via actual testing | Adjustments must be made based on business priority requirements for risk levels. This ensures high-risk entries are retrieved first |
The parameter values provided on this page are all common recommendations used as a starting point for configuration. Actual values are affected by material format, data volume, and business rules. Specific issues require specific analysis. It is recommended to test on your own samples before finalizing settings.
Three common configuration errors
- Searching across hundreds of thousands of index entries returns zero results or far fewer results than expected. The cause is failing to configure the
structured_index_fieldsparameter. Sending all content to the vector model prevents exact matching of critical structured fields such as asset IP and CVE ID. - Index construction times out, returning a
504 Gateway Timeouterror. The cause is not enablingincremental_index_enabled. Performing one-time vector encoding of all historical data exceeds the single-task timeout threshold. - Vector retrieval results include an overly high proportion of low-risk entries. The cause is failing to set the
similarity_thresholdparameter or setting the threshold too low. This leads to incorrect retrieval of low-correlation non-security descriptive text.
How to confirm the configuration is properly set
- View index construction logs. Confirm incremental index tasks trigger automatically per the configured update cycle. No full reindexing errors occur. For version V4.8.20-FIX2, use the system index management panel to check status.
- Run exact retrieval on structured fields. Enter a known
asset_iporcve_idto verify quick return of corresponding entries. Confirm the structured index is active. - Compare vector retrieval and exact retrieval results. Confirm high-risk entries are retrieved first. Adjust
similarity_thresholdto fit business requirements. - Randomly select a single due diligence report. Verify its associated vulnerability entries are properly segmented and vectorized. No semantic loss occurs from text truncation.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-09-14.