Online Incident Troubleshooting and Retrospective for Incident Response and Fault Handling: Scheduled Tasks and Automation

Data for this category originates from monitoring alerts, logging platforms, APM tools, and user feedback. It updates in real time or synchronizes

Data profile for this category

Data for this category originates from monitoring alerts, logging platforms, APM tools, and user feedback. It updates in real time or synchronizes immediately after each event is triggered. Core processing stakeholders include SREs, on-call operations staff, and on-call development leads. Daily deliverables include incident tickets, event timelines, emergency response plans, and fault notifications. Core fields include alert severity, affected service scope, occurrence time, stop-loss operation records, and root cause investigation clues. There are no fixed uniform unit requirements; fields are dynamically adjusted based on event type.

Constraints imposed by these characteristics on scheduled tasks and automation workflows

Real-time updates from multiple data sources require scheduled tasks to adapt to synchronization delays of different data sources, preventing workflow interruptions caused by unready data. Dynamically changing document fields require automated workflows to support custom field mapping to adapt to the document structure of different events. Frequently updated incident data requires scheduled tasks to run at short intervals to ensure timely capture of critical information. Additionally, automation of time-consuming steps such as impact assessment and stop-loss operations must match the real-time nature of event triggers to avoid delays that hinder service recovery.

Configuration guidelines

Configuration ItemRecommended SettingRationale
Scheduled trigger interval60-300 secondsAdapts to the scheduling engine performance of FastGPT v4.17.0, matches the real-time update rhythm of multi-source data, and avoids missing critical events
Data source pull scopelast 1 hour of alert and log dataMatches the update frequency of incident data in this category, avoids pulling redundant data that consumes resources
Incident field mapping rulesCalibrated based on actual testingDocument fields in this category are dynamically adjusted based on events, so custom configuration is required to adapt to different event types
Alert threshold matching rulesmatch alerts with severity P1-P2Prioritizes handling high-impact fault incidents, aligns with the core goal of restoring service first in this category
Failed retry count3 timesBalances workflow stability and resource consumption, avoids resource waste caused by repeated triggers
Notification trigger conditionautomatic stop-loss not completed within 10 minutes after fault occurrenceCovers the most time-consuming stop-loss operation link in this category, triggers manual intervention in a timely manner

Three common configuration errors

  • Symptom: Scheduled tasks stop running after a container restart, with no execution logs. Cause: Persistent storage for scheduled task configuration is not configured, and task configuration relies on local process state storage.
  • Symptom: Automated image recognition workflows return empty results, with logs showing empty image fields. Cause: No image path mapping rule is configured when pulling data sources, and image resources associated with the incident are not linked.
  • Symptom: Automatically generated fault retrospective documents are missing the event timeline field, resulting in incomplete content. Cause: No dynamic field mapping rule is configured, and timestamp fields pulled from APM tools are not synchronized to the retrospective document.

How to verify proper configuration

  • Manually trigger a scheduled task, verify that the pulled data includes alert and log information from the last 1 hour.
  • Simulate triggering a P1-level alert, check whether the workflow automatically executes preset stop-loss operation steps.
  • View task execution logs, confirm that manual notification is triggered only after the failed retry count reaches the configured 3 times.
  • Check the generated retrospective documents, confirm that core fields such as event timeline and affected scope are included.

The values listed here are common starting points, and should be evaluated against the sample data of the respective deployment environments.

Question material comes from public community discussions. Configuration values are common starting points and should be measured against your own samples. Verified on 2026-10-19.