What this type of data looks like
Data sources include structured fields returned by public security network verification APIs, text extracted via OCR from user-uploaded ID scans or photos, and supplementary information entered offline. Update frequency adjusts based on business scenarios. New user verification pulls real-time latest data. Existing customer data synchronizes on a fixed cycle.
Document structure falls into two categories:
- Structured standardized fields including full name, 18-digit resident ID number, household registration address, issuing authority, validity period, and more.
- Unstructured OCR recognition text containing full printed text from the document.
All fields have fixed format requirements. For example, ID numbers consist of 18 digits and an optional uppercase X at the end. Address fields include provincial, municipal, district, and street level information.
Constraints Imposed on Vector Models and Indexing
Mixed structured and unstructured data requires vector models to support both standardized field encoding and natural language text encoding. Repeated features in structured fields must not disrupt vector space distribution.
OCR recognition results have variable completeness and accuracy. This can create deviations in vectorization results for the same document. A preprocessing verification step must be added.
Resident ID numbers are globally unique identifiers. Exact deduplication must complete before index construction to prevent duplicate entries from consuming index resources.
ID documents contain sensitive personal information. Vector generation and storage must strictly follow data desensitization requirements. Plaintext sensitive fields must not be stored.
The business has real-time verification requirements. Index recall latency must align with business response thresholds.
Configuration Settings
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
MAX_CONTEXT_LENGTH | 800–1500 characters | ID document OCR text typically falls within this range. Limiting length avoids vector generation timeouts and feature redundancy caused by overly long text |
EMBEDDING_MODEL | Multimodal semantic model | Adapts to mixed input of structured fields and OCR text, balancing standardized field encoding and natural language semantic understanding capabilities |
DUPLICATE_REMOVE_FIELD | ID number | Resident ID numbers are globally unique identifiers. Using this as the deduplication field efficiently eliminates duplicate index entries |
EMBEDDING_DIMENSION | 768 dimensions | Balances vector representation accuracy and index retrieval efficiency, adapting to encoding needs for general semantics and structured fields |
PARSE_TIMEOUT | 120 seconds | ID document OCR and vectorization processes must balance recognition accuracy and response speed. This duration covers most normal recognition scenarios |
SENSITIVE_MASK_RULE | Keep first 6 and last 4 digits of ID number | Meets data desensitization compliance requirements while retaining limited features for subsequent association verification |
The parameter values provided on this page are general recommendations used as a starting point for configuration. Actual values are affected by material form, data volume, and business rules. Specific issues require case-by-case analysis. It is recommended to test on your own samples before finalizing settings.
Three Common Errors
- Symptom: The total number of stored vector entries is less than the number of source data entries. For example, 100,000 source entries result in just over 90,000 entries after storage. Cause: The
DUPLICATE_REMOVE_FIELDparameter is not configured, or the configured field is not linked to a unique identifier. This automatically filters duplicate entries without triggering a prompt. - Symptom: The collection is created successfully, but the page always displays the "Index Building In Progress" status with no progress updates. Cause: No reasonable threshold is set for the
PARSE_TIMEOUTparameter. OCR recognition or vectorization processes time out without triggering retries, interrupting the index building workflow. - Symptom: Vector recall results perform as expected during local runs, but all text vector scores are identical after packaging into a Docker image for deployment. Cause: The model cache directory is not correctly mounted in the Docker environment, or the model loading path is not properly configured. This activates the default placeholder vector generation logic.
How to Verify Configurations Are Correct
- Check vector model loading logs to confirm the specified multimodal semantic model has loaded successfully, with no missing model or loading failure errors.
- Upload a single ID document sample, confirm the generated vector dimension matches the configured dimension parameter value.
- Import multiple test datasets containing duplicate ID numbers, confirm the total stored entries match the number of deduplicated actual entries.
- Trigger a batch vectorization task, confirm the task execution duration does not exceed the configured timeout threshold, with no timeout interruption records.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-09-14.