Model Integration and Configuration for Real-World Evidence

Data Characteristics in This Category

Real-world evidence (RWE) data for clinical trial pre-screening primarily originates from Electronic Health Records (EHR), insurance claims databases, disease registries, and patient-reported outcomes (PROs). This data often combines unstructured text (e.g., clinical notes, pathology reports) with structured data (e.g., diagnostic codes ICD-10, medication records ATC codes, laboratory results LOINC codes). Data update frequencies vary; some are real-time, others are imported in monthly or quarterly batches. Document structures are diverse, containing medical terminology, abbreviations, units of measurement like mg/dL, mmol/L, and extensive free-text descriptions lacking uniform formatting standards.

Constraints Imposed by These Characteristics on "Model Integration and Configuration"

The highly heterogeneous and unstructured nature of real-world evidence data presents challenges for model integration. Large volumes of unstructured text require efficient text segmentation and embedding strategies to ensure information completeness and recall accuracy. The non-real-time nature of data updates necessitates configuring mechanisms for periodic data synchronization and index rebuilding. The density of medical terminology, abbreviations, and specific units of measurement means general word embedding models may struggle to accurately understand context. This requires targeted adjustments to embedding models or the introduction of domain-specific dictionaries. Additionally, sensitive information within the data, such as patient identities, requires anonymization during initial data processing, directly impacting the configuration of data cleaning and preprocessing workflows.

Configuration Guidelines

Configuration Item	Recommended Value	Rationale for Recommendation
`Chunk size` (Segment Length)	`500–800 characters` (characters)	Balances textual context completeness with embedding model processing efficiency, preventing individual segments from becoming too long and diluting key information.
`Chunk Overlap Length` (Segment Overlap Length)	`100–150 characters` (characters)	Ensures critical information across segments is not lost, increasing retrieval recall.
`embedding_model`	`text-embedding-ada-002` or `bge-large-zh`	Considers both general applicability and performance in the Chinese biomedical domain. Further testing may be needed based on actual data types.
`Similarity threshold` (Similarity Threshold)	`0.75–0.85`	Balances recall and precision based on `embedding_model` performance, avoiding excessive irrelevant results.
`Recall count` (Number of Retrieved Items)	`10–20 entries` (items)	Provides enough potentially relevant documents for subsequent re-ranking, covering more possibilities.
`PARSE_FILE_TIMEOUT_SECONDS`	`600 seconds` (seconds)	Addresses the parsing requirements for large or complex documents (e.g., clinical report `PDF`s), preventing processing failures due to timeouts.

Three Common Mistakes

Retrieval results with excessively high (e.g., 1.0) or low similarity, leading to ineffective search filtering. This can result from an inappropriate embedding model choice or ineffective segmentation during data preprocessing.
Model responses that cannot be further processed or output to other modules. This typically occurs when the output of the AI conversation module is not correctly connected to subsequent processing modules in the workflow configuration.
Processing failures or prolonged unresponsiveness after uploading large PDF documents. A common cause is setting the PARSE_FILE_TIMEOUT_SECONDS parameter too low, failing to accommodate the time required for large file parsing.

How to Confirm Correct Configuration

Upload representative real-world evidence documents to verify that document segmentation is reasonable and that segment content maintains medical context completeness.
Perform retrieval tests against specific clinical trial inclusion criteria. Check if the recalled results include all relevant patient records or feature descriptions and evaluate their relevance.
Use FastGPT's debugging interface to observe the embedding model's output vectors. Confirm their distribution consistency with expected results and adjust the threshold to ensure effective search filtering.
Test document uploads of varying sizes and complexities to verify that the PARSE_FILE_TIMEOUT_SECONDS setting successfully processes all files without timeout errors.

The values given are common starting points and should be measured against specific samples.

Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-09-21.