What Data in This Category Looks Like
This data category supports data service project managers, collection teams, and client algorithm teams. Primary data sources include client requirements, public data sources, collection tools, and sample libraries. Four regular document outputs are produced: collection plans, sample specifications, authorization statements, and task schedules. Data updates run on either per-project or weekly cycles. Core fields include sample scope, collection fields, legality verification rules, and task schedule nodes. Fields have no unified fixed units; match the format to the specific collection target. For example, sample count uses "items" as the unit, while authorization validity uses "days" as the unit. The core output of data collection task design focuses on raw data acquisition rules, with clear sample filtering conditions and collection permission boundaries defined.
Constraints on Database and Operations Workflows
Data from this category has scattered sources and volatile update cycles. Database operations must support flexible connections for multiple data source types. Daily documents include both structured task configurations and unstructured authorization statements. This requires compatibility with both structured table storage and unstructured document storage. The sample scope definition and legality verification steps generate high-frequency field validation requests. The database connection pool must adapt to concurrent request fluctuations. Data updates follow per-project cycles, and single-job data volume varies with project scale. Sufficient storage capacity must be reserved. Execution results and logs for collection tasks must be retained long-term. The database must include traceable audit capabilities to prevent data loss or tampering. For FastGPT v4.17.0, additional adaptation to the database connection parameter specifications from the version update is required.
Configuration Settings
| Configuration Item | Recommended Value | Rationale |
|---|---|---|
PG_URL | postgresql://<user>:<pass>@<host>:5432/<dbname>?sslmode=require | Adheres to standard connection format for managed PostgreSQL services, meeting structured data storage needs for collection tasks |
DB_MAX_LINK | 8–15 | Matches concurrent request volume for collection tasks under a single project, preventing exhaustion of database connection resources |
REDIS_URL | redis://:<password>@<host>:6379/0 | Used to cache task status snapshots and temporary verification data for collection tasks |
MONGODB_URI | mongodb://<user>:<pass>@<host>:27017/<dbname>?authSource=admin | Stores full logs of collection tasks, sample metadata, and historical execution records |
Three Common Misconfigurations
- A
text index required for $text queryerror appears during database queries. This happens when a full-text index is not created on the corresponding PostgreSQL field, which blocks support for text retrieval needs of collection tasks. - Collection task data is lost after a FastGPT version upgrade. This occurs when the storage instance linked to the database connection is not backed up before the upgrade, and data migration compatibility with the new version is not verified.
- The sample scope definition field for collection tasks fails to connect to data sources. This happens when database connection permissions are not configured correctly, and the required table read and field query permissions for collection tasks are not granted.
How to Verify Configuration Completion
- Test the specified database parameters to confirm that the data source table structures required for collection tasks can be retrieved normally.
- Submit a small-scale collection task, and check whether task status updates are properly written to the database and cache.
- Review database logs to confirm that collection task execution records are correctly written to the corresponding collection or table.
- Trigger a task schedule update, and verify that task status stored in the cache is synchronized to the database in a timely manner.
The values provided here are common starting points, and should be evaluated against your own specific samples and use cases.
Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-10-19.