Incident Response Automation

Automation contains response and grouping rules that are conditional triggers that execute response actions automatically, based on the contents of the alert.

An alert is created in Incident Response when a third-party monitoring tool has an event. The system analyzes incoming alerts, once a minute, and runs grouping and response rules accordingly.

Grouping rules

Use grouping rules to automatically group similar alerts and reduce the noise in you queue. Grouping rules are routing that helps you categorize by specific conditions. Each rule combines a set of filters and methods based on fields or condition matches.

Note: All grouping occurs before response rules run.

A system default rule for automatic group creation based on Service and Metric name fields is included when your system is provisioned. It is active by default.

Watch this quick video to know what you can do with grouping rules.

Default grouping rule

View the grouping rules by navigating to the Automation module from your Home page. Select the Grouping rules tab.

Create rules based on any criteria available from the field or condition fields.

Note: Prior to creating grouping rules, you should have teams, schedules, and integrations in place.

Rules are run only once on an alert, however, depending on when the alert or the rule entered the system, different things happen.

If an alert is created, and rule 1 is the only rule, it will run only once on that alert.
If you add rules 2 and 3 after that alert was created, they will not run on it unless the alert is updated. Those rules were never run on the alert to begin with.
If you close and reopen the alert, all three rules will run on it.

If you change an existing rule and an existing alert would have been captured by it, it won’t be. Whatever criteria or actions you have changed won’t be addressed unless the existing alert is closed and reopened. Then, whatever changed rule actions are will take place.

Grouping rules run in the order chosen, from lowest to highest. You can use the Rule Order icon to reorder your rules. Enter a new number and select Update in the Response rules list view.

Note: Administrators can reorder any rule. Responders and managers can reorder only their own or their team's rules.

Renumbering the rule to reorder it

Note:

Order is important for grouping rules because the lower-order rules run first. This can impact whether the conditions of higher-order rules are met, or will process. Lower-order rules can adjust field values or even stop processing.

The grouping rule list view can be arranged in several ways.

Note: This feature does not change the order in which the rules are run.

Order in which the rules run

Order (asc): Order they are in, ascending - Default.
Order (desc): Order they are in, descending
Last Modified (recent): Recent
Last Modified (oldest): Oldest

You can filter the listings using the Filters list menu in the header for:

System Generated Rules: Displays only rules that were created automatically when your account was provisioned.
My Team's Rules: Displays only those rules your team created.
Rules I created: Displays only those rules you created.
Active: Displays only active alerts.
Inactive: Displays only inactive alerts.

Name & Description represents the group rule name and description.

Scope represents who or what is impacted by this rule. Whether the impact is Global or to a team or service.

Criteria represents which grouping method this rule uses. Either Field Match or Condition Match .

Timeframe represents how long this rule should continue to run and group alerts after an initiating alert.

Whether your rules are Active or not is shown using the toggle switch icon.

The order of alert grouping is as follows:

Grouping rules run
Manually created groups are updated
Automated grouping runs

Alerts are grouped under the primary alert. The oldest of the highest severity alerts becomes the primary alert. The rest become related alerts. Grouping activity is recorded in Work notes..

A grouped, related alert contains a Grouped by field. This field indicates how the related alert was grouped. For more information, see Alert workspace.

For some examples of different response rules, see Grouping rules examples.

Note: Managers and responders can activate or deactivate only their own or their team’s rules.

Only administrators can activate or deactivate any rule, including global rules.

When a Service is deleted, its integrations, alerts, incidents, and automation rules are removed. This is not a recoverable action so consider deactivating the service instead.

Response rules

Use response rules to automatically assign and respond to alerts. They run after all grouping has occurred. Response rules are routing that helps you adapt your response to specific conditions. Each rule combines a set of filters and an outcome, usually setting who is assigned, priority, or creating email, Slack channels, or Zoom meetings.

Note: Response rules do not run on incidents.

A system default rule for automatic incident creation for P1 alerts is included when your system is provisioned. It is active by default.

Alt text for alert-rules-default-rule.png

View the response rules by navigating to the Automation module from your Home page.

Create rules based on any criteria available from the condition filter fields. You can also set filter properties on the rules.

Note: Prior to creating response rules, you should have teams, schedules, and integrations in place.

Rules are run only once on an alert, however, depending on when the alert or the rule entered the system, different things happen.

If an alert is created, and rule 1 is the only rule, it will run only once on that alert.
If you add rules 2 and 3 after that alert was created, they will not run on it unless the alert is updated. Those rules were never run on the alert to begin with.
If you close and reopen the alert, all three rules will run on it.

Response rules run in the order chosen, from lowest to highest. Or you can use the Rule Order icon to reorder your rules. Enter a new number and select Update in the Response rules list view.

Note: Administrators can reorder any rule. Responders and managers can reorder only their own or their team's rules.

Note:

Order is important for response rules because the lower-order rules run first. This can impact whether the conditions of higher-order rules are met, or will process. Lower-order rules can adjust field values or even stop processing.

The response rule list view can be arranged in several ways.

Note: This feature does not change the order in which the rules are run.

Alt text for alert-rules-order.png

Order: Order they are in, ascending - Default.
Order: Order they are in, descending
Last Modified: Recent
Last Modified: Oldest

You can filter the listings using the Filters list menu in the header for:

System Generated Rules: Displays only rules that were created automatically when your account was provisioned.
My Team's Rules: Displays only those rules your team created.
Rules I created: Displays only those rules you created.
Active: Displays only active alerts.
Inactive: Displays only inactive alerts.

Name & Description represents the alert rule name and description.

Scope represents who or what is impacted by this rule. Whether the impact is Global or to a team or service.

Actions are denoted by the action icons.

Stop after run contains a pause icon if you wanted processing to stop after that rule runs.

Whether your rules are Active or not is shown using the toggle switch icon.

Response rule activity is recorded in Work notes.

For some examples of different response rules, see Response rule examples.

Note: Managers and responders can activate or deactivate only their own or their team’s rules.

Only administrators can activate or deactivate any rule, including global rules.

Modify field value and/or Create an Incident actions, in a rule, will run before escalation policies run.

When a Service is deleted, its integrations, alerts, incidents, and automation rules are removed. This is not a recoverable action so consider deactivating the service instead.