What the data for this category looks like
Minor metal industry data sources include industry data released by domestic minor metal industry associations, import and export declaration data from the General Administration of Customs, market quotes from futures exchanges, and capacity announcements from mining enterprises. Spot quote data updates daily. Supply and demand balance sheets update monthly. Import and export declaration data updates weekly or monthly. Capacity and commissioning announcements release irregularly. Most data documents are structured tables. Fields include product name, origin, product specification, transaction price, inventory quantity, trading partner country, and more. Units include yuan/ton, yuan/kilogram, ton, ten thousand yuan RMB, and others.
What constraints these characteristics impose on database and operations work
Multi-source heterogeneous data types and inconsistent update cycles require the database to support multi-format structured import and flexible scheduled scheduling configuration. Multi-unit fields need unified conversion rules to avoid data storage chaos. Irregularly released capacity announcements cannot rely on fixed scheduled tasks. The system must support a manual trigger for synchronization entries. During bulk data synchronization, the system must adapt to the scale of single-batch minor metal data volume. This avoids write timeouts or excessive resource usage.
How to set the configurations
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
DB_CONNECTION_POOL_SIZE | 10–15 | Minor metal data comes from multiple sources with varying update frequencies. This value balances synchronization efficiency and server memory usage, and fits 4c16g single-node configurations |
SYNC_DATA_BATCH_SIZE | 500–800 records | Single-batch synchronization volume for minor metals is mostly under 1,000 records. This range avoids bulk write timeouts while reducing the number of synchronization runs |
DATA_UNIT_CONVERSION_ENABLE | Enabled | Minor metal data uses multiple units such as yuan/kilogram and yuan/ton. Enabling this allows unified conversion to standard units for storage |
MANUAL_SYNC_ALLOWED | Enabled | Some minor metal capacity announcements release irregularly. Manual trigger for synchronization tasks must be supported |
DB_WRITE_TIMEOUT | 30 seconds | For bulk writes of multi-source data, this duration covers most single-batch synchronization times and avoids synchronization task interruptions |
DATASOURCE_PLUGIN_OUTPUT_TYPE | SQL execution result | This adapts the need to directly convert datasource plugin output to SQL execution results, and matches custom data storage logic |
The parameter values provided on this page are all common recommendations that serve as a starting point for configuration. Actual values are affected by material form, data volume, and business rules. Specific issues require specific analysis. It is recommended to test on your own samples before finalizing settings.
Three common mistakes
- Issue: The datasource plugin returns a database connection structure instead of structured data, making direct import to the knowledge base impossible. Cause:
DATASOURCE_PLUGIN_OUTPUT_TYPEis not configured as SQL execution result. The plugin returns connection configuration by default, not query results. - Issue: MongoDB restarts failed after migration. Reinstalling FastGPT after deleting all mounted files still does not resolve the issue. Cause: Residual files in the local MongoDB data directory were not cleaned. Only deleting the FastGPT mount directory does not cover the database storage path.
- Issue: The database synchronization task returns the error
Failed to connect to 192.168.xx.xx.:1433 - 38FBA17. Cause: The database whitelist is not configured to allow access from the FastGPT server IP, or port mapping configuration is incorrect.
How to confirm the configuration is correct
- Run a manual synchronization task once. Check if structured tables for the corresponding minor metal data are generated in the database, and if the fields include preset information such as product name, transaction price, and unit.
- View database connection logs. Confirm there are no
Failed to connecterrors, and that connection pool connections do not continuously reach the upper limit. - Simulate multiple concurrent synchronization tasks. Observe server CPU and memory usage rates to confirm they do not exceed the node's carrying threshold.
- Test the unit conversion function. Input test minor metal data with different units, and confirm stored data uses a unified standard unit.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-09-14.