Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Problem Sending Mail Without Smtp Auth on OpenSearch #863

Open
espala opened this issue Feb 21, 2024 · 1 comment
Open

[BUG] Problem Sending Mail Without Smtp Auth on OpenSearch #863

espala opened this issue Feb 21, 2024 · 1 comment
Labels
bug Something isn't working

Comments

@espala
Copy link

espala commented Feb 21, 2024

Hello everyone,

I am using Opensearch cluster and my cluster is running smoothly. I don’t get any error messages or logs in the standard run cycle.

I wanted to define an alarm specifically, so I defined a new “Monitors” definition from the “Alerting” menu. Then I define “Trigger” and select my “Smtp Channel” definition.

Everything works correctly in the Alerting mechanism, it finds the word I filtered and runs the trigger function. But it cannot send an email using my smtp information I entered in the “Notification” section. Because it is trying to do smtp authentication.

But I selected the “Encryption Method” as “none” in the “Email Senders” section under the “Notifications” menu. When I click the “send test email” button both in the “Channel” I defined and in the Alerting > Monitors > Triggers I defined, I see that it sends emails. I can also see the emails in my inbox.

But when “Alerting > Monitors > Triggers > Action” automatically sends an email, it tries to establish smtp authentication connection.

I want to send email without smtp auth. There is also “none” option on OpenSearch Panel. I think we have such a possibility. But I see in the log that I get a 503 error. 503 here is an error code representing smtp auth.

I am using an internal smtp relay server and I am not using the smtp auth mechanism. I need to be able to send emails without smtp auth as the Opensearch system allows me.

I think this problem is a bug. How can I solve this problem?

My first OpenSearch forum case;
https://forum.opensearch.org/t/about-mail-sending-problem-on-alerting-notification-section/17786

My second case detailing the problem;
https://forum.opensearch.org/t/problem-sending-mail-without-smtp-auth-on-opensearch-critical/17993

OpenSearch-Dashboards Github Projects Bug Report case;
opensearch-project/alerting-dashboards-plugin#890

OpenSearch dashboards-notifications Project Bug Report case;
opensearch-project/dashboards-notifications#162

screenshot of my smtp definition;
Screen Shot 2024-02-21 at 07 58 21
Screen Shot 2024-02-21 at 08 17 14

screenshot of my Alerting/Monitors definition;
Screen Shot 2024-02-21 at 08 13 47
Screen Shot 2024-02-21 at 08 13 55

Log;
Error: Failed running action: OpenSearchStatusException[{"event_status_list": [{"config_id":"Qzjsho0B2TBnxA0NWL9r","config_type":"email","config_name":"Test Email Notification Channel","email_recipient_status":[{"recipient":"[email protected]","delivery_status":{"status_code":"503","status_text":"sendEmail Error, status:Couldn't connect to host, port: smtp.mydomain.com, 25; timeout -1"}}],"delivery_status":{"status_code":"503","status_text":"sendEmail Error, status:Couldn't connect to host, port: smtp.mydomain.com, 25; timeout -1"}}]}]


Opensearch and my linux versions

[root@server6 ~]# yum list installed |grep opensearch
opensearch.x86_64                             2.11.1-1                                  @opensearch-2.x
opensearch-dashboards.x86_64                  2.11.1-1                                  @opensearch-dashboards-2.x

[root@server6 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux release 8.8 (Ootpa)

Related sections in my logs

...
[2024-02-21T07:59:52,113][INFO ][o.o.a.MonitorRunnerService] [server6] Executing scheduled monitor - id: , type: QUERY_LEVEL_MONITOR, periodStart: 2024-02-21T04:54:52.112Z, periodEnd: 2024-02-21T04:59:52.112Z, dryrun: true, executionId: _2024-02-21T04:59:52.113348729_cf70ca80-2682-4222-936f-b830f3ba42c1
[2024-02-21T07:59:52,162][INFO ][o.o.a.t.TransportExecuteMonitorAction] [server6] Executing monitor from API - id: , type: QUERY_LEVEL_MONITOR, periodStart: 2024-02-21T04:54:52.161Z, periodEnd: 2024-02-21T04:59:52.161Z, dryrun: true
[2024-02-21T07:59:52,163][INFO ][o.o.a.MonitorRunnerService] [server6] Executing scheduled monitor - id: , type: QUERY_LEVEL_MONITOR, periodStart: 2024-02-21T04:54:52.161Z, periodEnd: 2024-02-21T04:59:52.161Z, dryrun: true, executionId: _2024-02-21T04:59:52.163613799_ed76123f-6835-4285-8d35-3967a9c53af0
[2024-02-21T07:59:52,302][INFO ][o.o.a.t.TransportExecuteMonitorAction] [server6] Executing monitor from API - id: , type: QUERY_LEVEL_MONITOR, periodStart: 2024-02-21T04:54:52.301Z, periodEnd: 2024-02-21T04:59:52.301Z, dryrun: true
[2024-02-21T07:59:52,302][INFO ][o.o.a.MonitorRunnerService] [server6] Executing scheduled monitor - id: , type: QUERY_LEVEL_MONITOR, periodStart: 2024-02-21T04:54:52.301Z, periodEnd: 2024-02-21T04:59:52.301Z, dryrun: true, executionId: _2024-02-21T04:59:52.302626749_4da076b3-d59b-4128-9666-5af0273ce234
[2024-02-21T07:59:52,339][INFO ][o.o.a.t.TransportExecuteMonitorAction] [server6] Executing monitor from API - id: , type: QUERY_LEVEL_MONITOR, periodStart: 2024-02-21T04:54:52.339Z, periodEnd: 2024-02-21T04:59:52.339Z, dryrun: true
[2024-02-21T07:59:52,340][INFO ][o.o.a.MonitorRunnerService] [server6] Executing scheduled monitor - id: , type: QUERY_LEVEL_MONITOR, periodStart: 2024-02-21T04:54:52.339Z, periodEnd: 2024-02-21T04:59:52.339Z, dryrun: true, executionId: _2024-02-21T04:59:52.339981829_405f8627-c4d2-45c8-95be-9f636852f566
[2024-02-21T07:59:56,935][INFO ][o.o.a.t.TransportIndexMonitorAction] [server6] Updating monitor, hq0Xl40BzHB4GwYe75E5, from: {"monitor":{"type":"monitor","schema_version":8,"name":"opensearch Disaster  Error Monitor","monitor_type":"query_level_monitor","user":{"name":"admin","backend_roles":["admin"],"roles":["own_index","all_access"],"custom_attribute_names":[],"user_requested_tenant":null},"enabled":true,"enabled_time":1707914606685,"schedule":{"period":{"interval":5,"unit":"MINUTES"}},"inputs":[{"search":{"indices":["opensearch-disaster-*"],"query":{"size":0,"query":{"bool":{"filter":[{"range":{"@timestamp":{"from":"{{period_end}}||-3h","to":"{{period_end}}","include_lower":true,"include_upper":true,"format":"epoch_millis","boost":1.0}}},{"match_phrase":{"level":{"query":"error","slop":0,"zero_terms_query":"NONE","boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}}}}],"triggers":[{"query_level_trigger":{"id":"CzSzp40BPraQSw1iuOEJ","name":"Email Trigger","severity":"1","condition":{"script":{"source":"ctx.results[0].hits.total.value > 1","lang":"painless"}},"actions":[{"id":"notification915423","name":"act","destination_id":"Qzjsho0B2TBnxA0NWL9r","message_template":{"source":"Monitor {{ctx.monitor.name}} just entered alert status. Please investigate the issue.\n  - Trigger: {{ctx.trigger.name}}\n  - Severity: {{ctx.trigger.severity}}\n  - Period start: {{ctx.periodStart}}\n  - Period end: {{ctx.periodEnd}}","lang":"mustache"},"throttle_enabled":false,"subject_template":{"source":"Alerting Notification action","lang":"mustache"}}]}}],"last_update_time":1708424762064,"ui_metadata":{"schedule":{"cronExpression":"0 */1 * * *","period":{"unit":"MINUTES","interval":5},"timezone":null,"daily":0,"monthly":{"type":"day","day":1},"weekly":{"tue":false,"wed":false,"thur":false,"sat":false,"fri":false,"mon":false,"sun":false},"frequency":"interval"},"search":{"searchType":"graph","bucketValue":3,"timeField":"@timestamp","bucketUnitOfTime":"h","filters":[{"fieldName":[{"label":"level","type":"text"}],"fieldValue":"error","operator":"is"}],"aggregations":[],"cleanedGroupBy":[]},"triggers":{"Email Trigger":{"value":1,"enum":"ABOVE"}},"monitor_type":"query_level_monitor"},"data_sources":{"query_index":".opensearch-alerting-queries","findings_index":".opensearch-alerting-finding-history-write","findings_index_pattern":"<.opensearch-alerting-finding-history-{now/d}-1>","alerts_index":".opendistro-alerting-alerts","alerts_history_index":".opendistro-alerting-alert-history-write","alerts_history_index_pattern":"<.opendistro-alerting-alert-history-{now/d}-1>","query_index_mappings_by_type":{},"findings_enabled":false},"owner":"alerting"}}
 to: {"monitor":{"type":"monitor","schema_version":8,"name":"opensearch Disaster  Error Monitor","monitor_type":"query_level_monitor","user":{"name":"admin","backend_roles":["admin"],"roles":["own_index","all_access"],"custom_attribute_names":[],"user_requested_tenant":null},"enabled":true,"enabled_time":1707914606685,"schedule":{"period":{"interval":5,"unit":"MINUTES"}},"inputs":[{"search":{"indices":["opensearch-disaster-*"],"query":{"size":0,"query":{"bool":{"filter":[{"range":{"@timestamp":{"from":"{{period_end}}||-7h","to":"{{period_end}}","include_lower":true,"include_upper":true,"format":"epoch_millis","boost":1.0}}},{"match_phrase":{"level":{"query":"error","slop":0,"zero_terms_query":"NONE","boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}}}}}],"triggers":[{"query_level_trigger":{"id":"CzSzp40BPraQSw1iuOEJ","name":"Email Trigger","severity":"1","condition":{"script":{"source":"ctx.results[0].hits.total.value > 1","lang":"painless"}},"actions":[{"id":"notification915423","name":"act","destination_id":"Qzjsho0B2TBnxA0NWL9r","message_template":{"source":"Monitor {{ctx.monitor.name}} just entered alert status. Please investigate the issue.\n  - Trigger: {{ctx.trigger.name}}\n  - Severity: {{ctx.trigger.severity}}\n  - Period start: {{ctx.periodStart}}\n  - Period end: {{ctx.periodEnd}}","lang":"mustache"},"throttle_enabled":false,"subject_template":{"source":"Alerting Notification action","lang":"mustache"}}]}}],"last_update_time":1708491595830,"ui_metadata":{"schedule":{"cronExpression":"0 */1 * * *","period":{"unit":"MINUTES","interval":5},"timezone":null,"daily":0,"monthly":{"type":"day","day":1},"weekly":{"tue":false,"wed":false,"thur":false,"sat":false,"fri":false,"mon":false,"sun":false},"frequency":"interval"},"search":{"searchType":"graph","bucketValue":7,"timeField":"@timestamp","bucketUnitOfTime":"h","filters":[{"fieldName":[{"label":"level","type":"text"}],"fieldValue":"error","operator":"is"}],"aggregations":[],"cleanedGroupBy":[]},"triggers":{"Email Trigger":{"value":1,"enum":"ABOVE"}},"monitor_type":"query_level_monitor"},"data_sources":{"query_index":".opensearch-alerting-queries","findings_index":".opensearch-alerting-finding-history-write","findings_index_pattern":"<.opensearch-alerting-finding-history-{now/d}-1>","alerts_index":".opendistro-alerting-alerts","alerts_history_index":".opendistro-alerting-alert-history-write","alerts_history_index_pattern":"<.opendistro-alerting-alert-history-{now/d}-1>","query_index_mappings_by_type":{},"findings_enabled":false},"owner":"alerting"}}
[2024-02-21T07:59:58,104][INFO ][o.o.a.r.RestSearchMonitorAction] [server6] The monitor parsing failed. Will return response as is.
...

2. Subject
At the same time, when it can’t find the word I filtered for the alarm in “Alerting” > “Monitor”, it says that the “Trigger” and “Action” functions are working in the Alerts section and shows that it sent the email. But this is not true either.

My filter works correctly. I am looking for the word “error” in the log data for the last 3 hours. When it really can’t find the word “error” in the last 3 hours, the “History” section goes to green “no alerts”. This works correctly.

But when I change the status of the alarms under the “Alerts” heading at the bottom of the dashboard to “Completed”, it shows it as “Email Trigger”.

In this case, it does not seem to reflect an true result. Because even though there was an alarm trigger, the action gave an error and could not send the email.

Screen Shot 2024-02-21 at 07 59 34

@espala espala added bug Something isn't working untriaged labels Feb 21, 2024
@espala espala changed the title [BUG] [BUG] Problem Sending Mail Without Smtp Auth on OpenSearch Feb 21, 2024
@riysaxen-amzn
Copy link
Contributor

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants