[FEATURE] Decouple Alerting and Anomaly Detection

[amitgalitz]

Is your feature request related to a problem?

Currently Anomaly Detection and Alerting code is coupled throughout the backend in order to create an alerting monitor on top of an existing anomaly detector. The Alerting plugin currently queries the anomaly detection system indices directly, and also has specific code that checks if Anomaly Detection has their filter by backend role setting enabled. We would like to change this so we have a solution that decouples Alerting and Anomaly Detection code as well as creates supports for further enhancements.

What solution would you like?

The first step of change we would like to make is to utilize Common Utils (Alerting already has an interface there https://github.com/sbcd90/common-utils/blob/514fa7acca686c6142ea847cfc1906398886fe41/src/main/kotlin/org/opensearch/commons/alerting/AlertingPluginInterface.kt) in order for Alerting to interact with the transport layer of Anomaly Detection. This has already been done in between alerting and security analytics and will require similar changes. On the alerting side we will add calls to a callback function that Anomaly Detection will register whenever a CRUD operations needs to be done on an detector. The changes will be similar to the ones done here 37b81fa#diff-68866b22ed9703814b4d5db8d3488872bcb972086ecaca10c9b8bfd54db981bc.

[lezzago]

related to opensearch-project/anomaly-detection#947