What the data for this category looks like
Thermal coal category data sources include publicly available industry association statistics, port loading and unloading monitoring logs, railway and sea freight records, data submitted by downstream energy-consuming enterprises, and industry policy documents. Update rhythms vary across data types: spot price data is updated daily, monthly supply and demand briefings are updated monthly, and industry policy documents are released as needed. Document structures cover multiple types: structured spot price tables, semi-structured monthly supply and demand reports, and unstructured policy interpretation texts. Standard fields include origin code, calorific value, total moisture, ash content, and transportation link information. Calorific value uses megajoules per kilogram as its unit, while moisture content uses kilograms per ton.
What constraints these characteristics impose on vector models and indexing
Multi-source heterogeneous data requires the vector processing pipeline to support unified extraction of structured, semi-structured, and unstructured data. This avoids damaging business semantics with a single segmentation logic. Differentiated update rhythms require the indexing system to support flexible switching between incremental and full synchronization. This adapts to real-time daily updates of spot data and batch processing of monthly reports. Differences in field units and meanings require prior standardized mapping. Otherwise, vector similarity calculations will be disrupted by inconsistent units. Wide variation in document length requires the segmentation strategy to have adaptive capabilities. This prevents over-segmenting short documents or causing semantic breaks in long documents. Cross-document association analysis requirements require the index to support multi-field joint recall. This covers associated data across dimensions such as origin, transportation, and demand.
How to set configurations
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
PARSE_DOC_SPLIT_SIZE | 800–1200 characters | Adapts to mixed documents of short spot price sheets and long reports, avoids damaging business semantics through segmentation |
VECTOR_MODEL_NAME | bge-large-zh-v1.5 | Adapts to professional terminology in the thermal coal industry, improves semantic alignment effects |
INDEX_INCREMENTAL_SYNC | Enabled | Adapts to daily updated spot data, reduces time spent on full indexing |
RECALL_TOP_K | Top 10 entries | Covers multi-dimensional associated data required for due diligence, avoids recalling excessive redundant information |
PARSE_FILE_MAX_SIZE | 2000 MB | Supports uploading monthly industry reports containing large numbers of charts |
EMBEDDING_BATCH_SIZE | 32 | Balances memory usage and speed of batch processing, adapts to vector generation for 100,000-level data volume |
The parameter values provided on this page are common recommendations used to determine a starting point for configurations. Actual values are affected by material form, data volume, and business rules. Specific issues require case-by-case analysis. It is recommended to test on your own samples before finalizing settings.
Three common mistakes
- Phenomenon: After uploading CSV-format thermal coal price data, the total number of entries stored in the vector database after vector generation is less than the number of lines in the source file. Cause: Empty lines or abnormally formatted lines exist in the CSV file. The parsing process automatically filters invalid data, resulting in missing vector database entry counts.
- Phenomenon: The vector index creation returns a success status, but the page always displays that the index has not been established. Cause: In a Docker deployment environment, the storage directory of the vector service is not correctly mounted to the host machine, or the directory has no write permission, causing index files to fail to persist.
- Phenomenon: The vector indexing process runs normally locally, but after being packaged into a Docker image and run, the generated vector scores are completely identical across multiple runs. Cause: The Docker image does not mount the model cache directory. Each startup re-downloads and initializes the vector model, resulting in generated vectors not reusing context and producing fixed results.
How to confirm configurations are set correctly
- Upload a single short document, such as a daily spot price sheet, and check the field integrity of the vector generation results to confirm that segmentation has not damaged the semantics of core business fields such as origin and calorific value.
- Import 100,000 pieces of structured thermal coal data, compare the number of lines in the source file with the number of vector database entries, to confirm there is no filtering of abnormal data.
- Start the Docker container, access the health check interface of the vector service, and confirm that the index storage path has write permission.
- Import documents of varying lengths, generate vectors multiple times and compare results, to confirm that the model cache has been correctly mounted and repeated model initialization is avoided.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-09-14.