What This Type of Data Looks Like
Daily documents for penetration testing services include four core types: test authorization letters, penetration test reports, vulnerability reproduction records, and rectification suggestions. Data sources include target system logs, public vulnerability databases, automated scanner results, and manual verification records from red team engineers. Data is synchronized for update each time a project is completed or a major vulnerability analysis version is released.
Fields for individual documents include vulnerability severity, reproduction steps, authorization validity period, rectification priority, and more. Structural differences across documents are significant: long report-style documents can reach thousands of characters, while short verification records only have hundreds of characters.
What Constraints Do These Characteristics Impose on Knowledge Base Retrieval and Recall
Because data is updated per project and has significant structural differences, retrieval must limit the scope to avoid mixing irrelevant cross-project documents. Otherwise, authorization disputes or false vulnerability judgments may occur. Dense technical terminology in document content requires retrieval to balance semantic matching and exact keyword matching. Plain semantic search may fail to accurately recall detailed content such as vulnerability reproduction steps. The wide range of individual document lengths requires adaptive text chunking and recall strategies to avoid losing critical information from truncated long documents. High-frequency updated data requires the retrieval pipeline to have fast synchronization capabilities, ensuring that ticket Q&A can obtain the latest vulnerability rectification information.
How to Configure Settings
| Configuration Item | Recommended Setting | Rationale |
|---|---|---|
similarity | 0.75–0.85 | Matches the precision requirement for vulnerability details in penetration test reports, prevents irrelevant general security documents from appearing in results |
limit | Top 8 entries | Covers multi-vulnerability scenarios that may be involved in a single ticket, while keeping the total context length within the FastGPT v4.17.0 context limit |
usingReRank | Enabled | Optimizes ranking results for long report-style documents, improves recall priority for detailed content such as vulnerability reproduction steps |
searchMode | Semantic search | Adapts to semantic matching requirements for technical terms in penetration test documents, such as accurate matching of terms like "privilege escalation vulnerability" and "permission bypass" |
datasetSearchUsingExtensionQuery | Enabled | Supplements professional term expansion for ticket questions, for example, expanding "authorization check" to "test authorization letter, authorization validity confirmation" to improve recall precision |
collectionIds | Configured using the project ID associated with the current ticket | Limits retrieval scope to only project data involved in current operations on-call, avoids performance overhead from full-data retrieval |
Three Common Misconfigurations
- Phenomenon: The knowledge base can retrieve documents, but the large model’s answer is concise and does not include specific clause content. Cause:
datasetSearchUsingExtensionQueryis not enabled, and ticket questions are not expanded into queries that match document details, resulting in too short retrieved document fragments that cannot support complete clause output. - Phenomenon: Knowledge base search takes too long, with response timeouts on an 8-core 64G memory configuration. Cause:
collectionIdsis not configured to limit retrieval scope, and retrieval covers all historical project data, leading to vector matching calculation volume exceeding hardware carrying capacity. - Phenomenon: No effect after configuring
usingReRank. Cause: The value oflimitis too small, only a small number of candidate results are retrieved, and the reranking module cannot obtain enough candidate data for ranking optimization, so the reranking effect is not visible.
How to Verify Correct Configuration
- View the knowledge base collection management interface, confirm that
collectionIdsis configured per project, and the retrieval scope is limited to the project collection associated with current operations on-call. - Trigger a test retrieval, check that the
similarityscores of returned results fall within the preset 0.75–0.85 range, and the number of returned results matches thelimitsetting. - Compare retrieval results with and without
usingReRankenabled, confirm that sorted results better match vulnerability detail descriptions in tickets, such as improved priority of vulnerability reproduction steps. - Check the FastGPT v4.17.0 retrieval log, confirm that
datasetSearchUsingExtensionQueryis active, and the retrieval query includes expanded professional terms.
The values provided are common starting points and should be evaluated against the reader’s own samples.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-10-19.