What Data in This Category Includes
Data sources for this category include raw datasets, ETL logs, annotation platforms, and sampling results. Data update frequency follows a daily or per-batch delivery cycle. Daily produced documents include cleaning rules, quality inspection sampling tables, abnormal data lists, and delivery reports. Cleaning rules include rule identifiers, effective scope, and matching fields. Quality inspection sampling tables include total sample count and non-conforming sample percentage fields. Abnormal data lists include unique data identifiers, anomaly types, and processing statuses. Delivery reports include batch numbers and overall quality inspection conclusions.
Constraints Imposed on Knowledge Base Retrieval and Recall
The diversity of data sources requires retrieval to cover both structured rule documents and unstructured logs and reports. It must support recall of knowledge base content in varying formats. The high-frequency update feature requires the retrieval pipeline to support incremental synchronization. This prevents rule call errors caused by data lag. Business scenarios focus on precise matching of rules and anomaly handling solutions. Low-correlation recall results must be filtered to ensure data processing accuracy. Most query demands from business entities revolve around rule configuration and anomaly troubleshooting. Retrieval must prioritize matching business terminology and field identifiers to improve query efficiency.
How to Configure Parameters
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
similarity | 0.75–0.85 | Matching data cleaning rules and abnormal lists requires precise alignment with business terminology, preventing recall of irrelevant rules with low similarity |
limit | Top 8–12 entries | The number of cleaning rules and anomaly handling documents recalled per batch must fit the batch review needs of quality inspectors, avoiding excessive results that interfere with judgment |
searchMode | Precise match first | Most business queries for data cleaning and quality inspection target precise matching of rule IDs and anomaly types. Precise matching reduces irrelevant recall results |
usingReRank | Enabled | Text content in abnormal data lists and ETL logs is lengthy. Reranking filters low-correlation recall results and improves accuracy |
datasetSearchUsingExtensionQuery | Enabled | Data cleaning rules often have associated fields. Extended queries cover synonyms or similar business expressions, adapting to engineers' fuzzy query needs |
collectionIds | Only bind collections corresponding to cleaning rules, quality inspection sampling tables, abnormal data lists, and delivery reports | Narrow the retrieval scope, avoid interference from irrelevant knowledge base content, and improve retrieval efficiency |
Three Common Configuration Errors
- Knowledge base retrieval responses time out, and the interface displays the
504 Gateway Timeoutstatus code. Cause: Retrieval parameters are not adjusted for high-frequency updated datasets used in data cleaning and quality inspection, leading to excessive index load on Milvus with GPUs. - An error popup appears immediately after creating a new knowledge base directory, with a provider-related anomaly prompt shown on the interface. Cause:
collectionIdsis not configured correctly, and bound knowledge base collections either do not exist or have insufficient permissions, preventing normal invocation of the retrieval pipeline. - The number of retrieved results does not match the set
limit, with returned results exceeding or falling short of expected quantities. Cause: Thesimilaritythreshold is not adjusted properly. A threshold that is too low includes too many irrelevant results, while a threshold that is too high filters out valid matching content.
How to Verify Correct Configuration
The following verification steps are based on the interface and parameter logic of FastGPT v4.17.0.
- Retrieve specified cleaning rule documents, verify that returned results include the target rule identifier, and confirm that the
similaritythreshold fits business matching requirements. - Check response times for multiple consecutive retrievals, and adjust the
limitandusingReRankconfigurations based on actual business load. - Test creating a new knowledge base directory and adding new folders, confirm that the
collectionIdsbinding configuration has no errors, and no provider-related error messages appear. - Retrieve abnormal data lists, verify the relevance ranking of returned results, and confirm that the
usingReRankfunction is enabled.
The values provided here are common starting points, and should be evaluated against the reader's own samples.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-10-19.