What this category of data looks like
Data for this category mainly comes from the regulatory and rule repository, higher-level law database, departmental policy document repository, and administrative normative document filing system. Update frequency is irregular. Daily processed documents include normative document drafts, drafting explanations, legality review opinions, summary of solicited opinions, and filing reports. Most document structures include fields such as article numbers, issuing authorities, effective dates, rights and obligation clauses, and review basis. There is no unified fixed unit identifier. Content is mostly abstract policy rule text.
What constraints these characteristics impose on the embedding models and indexing workflow
Data sources are scattered, and updates have no fixed cycle. Support must be provided for incremental synchronization and incremental indexing of multi-source data to avoid repeated processing of historical documents. Documents contain multiple types of structured and unstructured content. Metadata such as article numbers and issuing authorities must be associated with the index to improve retrieval accuracy. Daily review requires comparison with higher-level laws and authority-based basis. Strict requirements apply to long-text chunking logic to avoid splitting that breaks the contextual coherence of core clauses. Retrieval results must meet review rigor requirements. The proportion of false recalls and missed recalls must be strictly controlled to ensure the legality of policy formulation.
Reference only; verify against the target FastGPT release and deployment documentation.
How to set the configurations
| Configuration Item | Suggested Value | Rationale |
|---|---|---|
| Chunk Length | 800–1200 characters | Normative document articles are mostly continuous logical text, to avoid splitting that breaks the integrity of core content such as rights and obligations and review basis |
| Overlap Length | 100–200 characters | Cover associated content of adjacent chunks, ensure contextual coherence when retrieving higher-level law basis |
| Similarity Threshold | Calibrated based on actual testing | Must match the semantic similarity characteristics of policy texts, avoid false recalls of irrelevant higher-level law articles or missed recalls of valid basis |
| Maximum Search Results per Request | Top 10 | Legality review requires comparison of multiple types of basis, limiting return results ensures review efficiency and information focus |
maxToken | 4000–6000 token | Must carry the contextual content of multiple documents such as higher-level laws and drafting explanations required for review, conforms to model input limits |
defaultToken | 2000 token | Adapts to conventional retrieval and comparison requirements for a single core document, balances contextual length and computing resources |
Three common mistakes
- The status of a text dataset remains "Indexing" for a long time after import. Cause: No incremental indexing trigger rule is configured, or processed historical documents are not skipped during multi-source data synchronization, resulting in a backlog of duplicate indexing tasks.
- Semantic similarity values exceed the normal range after switching embedding models. Cause: The matching logic of the similarity threshold is not adjusted. The similarity score range output by some models does not match the default configuration, causing the threshold setting to fail.
- Retrieval results cannot accurately match higher-level law basis. Cause: Chunk length is set too short, splitting associated article and issuing authority information, resulting in broken semantic connections and reduced retrieval recall accuracy.
How to confirm the configuration is correct
- Import a single typical normative document draft, check if the chunked text retains complete article numbers and core clause content.
- Enter a legality review query, verify that the number of retrieval results matches the maximum search results per request setting.
- Switch to a test embedding model, confirm that the range of similarity scores matches expectations, and adjust the threshold to match the output characteristics of the current model.
- Simulate irregular data updates, trigger an incremental indexing task, and check if new data can complete the indexing process normally.
The values given are common starting points and should be measured against the reader's own samples.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-09-28.