Alerting and Notifications Plugins

What are the Alerting and Notifications Plugins?

The Alerting plugin can be used to monitor data and create alert notifications when conditions are triggered in one or more indexes. A monitor with trigger conditions can be created to generate various alert notifications to the selected destination. We currently provide support for custom webhooks and Slack as destinations. From OpenSearch 2.0 onwards, the Alerting and Notifications plugins are bundled individually.  For OpenSearch 1.x the Alerting plugin includes the notification functionality.

How to provision clusters with the plugins?

When choosing the eligible OpenSearch version (OpenSearch 1.3.6 and 2.4.0 onwards respectively), the options will be available on the console to provision clusters with the plugins.

The plugins can also be provisioned from API or Terraform V1. More information about API request can be found: https://instaclustr.redoc.ly/#operation/extendedProvisionRequestHandler

Limitations and security considerations

• In the Alerting and Notifications indexes, credentials for webhooks are stored in plain text, so extra care needs to be taken to keep them safe. There are a few things you can do to mitigate the risks:
  • Consider what roles and users need access to the alerting and notification indices when setting up security rules for your cluster. Refer to the OpenSearch project for more specific information.https://opensearch.org/docs/latest/security-plugin/access-control/users-roles/
  • Ideally think about using regular password rotation for these credentials, e.g every 3 Months. If this is unfeasible at least prepare for a leakage event where the credentials need to be rotated in a short time frame.
  • Avoid where possible sharing these credentials with other applications or users. This will reduce impact in the event of leakage.
• If a user has preexisting monitors, removing the resource permissions will not remove their access to those monitors. However, if the user does not have permission to access the index, the monitors will not be executed.
• Similar to the limitation above, users can create monitors for resources that they do not have permissions to, but the monitors will not be executed. 
• Migrating using Alerting from OpenSearch 1.x version to 2.x version requires some manual steps to keep the configurations for Destinations because Destinations have become Channels in the Notifications plugin.

How to use the plugin

Adding Targets/Destinations

The plugins can be found on the left hand sidebar of your OpenSearch dashboards. From OpenSearch 2.0 onwards, the Alerting and Notifications plugins are bundled individually.  For OpenSearch 1.x the Alerting plugin includes the notification functionality.

More information for the plugins can be found:

Alerting: https://opensearch.org/docs/latest/monitoring-plugins/alerting/index/

Notifications: https://opensearch.org/docs/latest/notifications-plugin/index/

Add Firewall Rule in Destination Cluster

To receive information from a source cluster, firewall rules need to be added to allow the necessary ingress traffic. For example, when the source is another cluster, you can add IPs of that source cluster to the Firewall Rules page.