What the data for this category looks like
Data primarily comes from mine scheduling logs, tunneling construction records, ventilation safety reports, shift handover ledgers, and safety regulation documents. Update frequency varies by business scenario: shift handover ledgers are updated daily, scheduling logs are generated in real time, and monthly safety reports are compiled on a set cycle. Most documents are a mix of structured and semi-structured content, with fixed fields including mine number, shift, and operation time. They also include numerical items with units such as gas concentration, ventilation volume, and tunneling footage. The main body of text mostly consists of on-site operation descriptions and hidden danger rectification notes.
What constraints these characteristics impose on vector models and indexes
The mixed semi-structured and structured document structure requires indexes to support multi-field associated retrieval. This avoids losing key metadata such as mine number and shift when only performing vector encoding on the main text. Real-time generated scheduling logs require support for high-frequency incremental indexing, to avoid performance overhead from full index reconstruction. Numerical items with units such as gas concentration and ventilation volume will cause semantic deviation in vector encoding if directly embedded into the main text. These values must be bound to their units for processing before being input into the vector model. The uniformity of field formats across different documents is limited, so indexes need to support flexible metadata filtering to narrow the retrieval scope.
Configuration guidelines
Reference only; verify against the target FastGPT release and deployment documentation.
| Configuration Item | Recommended Range | Rationale |
|---|---|---|
| Chunk Length | 800–1200 characters | Coal mine production documents combine on-site operation descriptions and structured fields. This range balances semantic completeness and vector encoding accuracy |
| Overlap Length | 100–150 characters | Prevents key information from being truncated after chunking, and maintains semantic coherence across chunks |
| Similarity Threshold | 0.72–0.85 | Adapts to the density of professional terminology in coal mine documents, and filters low-relevance retrieval results |
| Single Search Limit | Top 10 results | Controls the volume of retrieval results to avoid exceeding context window limits |
maxToken | 8000 token | Adapts to the splicing of retrieval results from long coal mine documents, and complies with input limits of most vector services |
defaultToken | 512 token | Sets the maximum length of a single vector encoding, and adapts to document content with a high volume of professional terminology |
Three common mistakes
- Phenomenon: Vectorization tasks take too long, and the interface returns a 504 timeout status code. Cause: Only single text chunk requests are submitted, batch vectorization processing is not enabled, and the batch processing logic of the vector service is not adapted.
- Phenomenon: Retrieval results have significant relevance deviations, and some irrelevant administrative documents are retrieved. Cause: The similarity threshold is not adjusted for the density of professional terminology in coal mine documents, and metadata filtering is not enabled to narrow the retrieval scope.
- Phenomenon: Chunked index content cannot be retrieved normally, and the interface returns empty chunked text fields. Cause: The association rules for chunked storage are not configured correctly, and structured fields are not bound to vector chunks for storage.
How to verify correct configuration
- Check the vector service runtime logs to confirm whether batch-submitted text chunk requests exist, and verify that the batch processing logic is functioning.
- Conduct a retrieval test for coal mine documents, check whether the metadata of the retrieval results includes exclusive fields such as mine number and shift, and verify that metadata filtering is working correctly.
- Review the stored chunk index list to confirm that each chunk is associated with the corresponding structured metadata field, and verify that the chunk configuration is correct.
- Check the system parameter configuration interface to confirm that the values of
maxTokenanddefaultTokenmatch the preset settings, and verify that the parameters are applied correctly.
The values provided here are common starting points and should be measured against the reader's own samples.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-09-28.