Open Heartbeat Issues

[KaraMelih]

There are still a few things that need to be addressed on the heartbeat side;

• The warning frequency and possible duplicate warnings;
  We are now sending warnings after mu+3sigma time has elapsed and there were no new HB from a given experiment. However, by definition this sends a warning ~once every 100 heartbeats. On top of that, on XENONnT tests I noticed that each time there is a warning, I get a second warning with ms-sec differences which is not supposed to happen. (mentioned here Heartbeat Warning emails #99)

• FAR computation at the time of the alert, not at the time of the coincidence
  The FAR should be computed taking into account the number of detectors that were online at the time of the "coincidence" however the script looks at the number of detectors that have a HB in the hb cache (last 24hr) which means, they could have sent a heartbeat 20hr ago and have a registered beat but are offline since then. Thus, at the time of the coincidence they should not be counted. (mentioned here Online Detectors Check missing logic #100)

• heartbeat feedback request. This works as expected except for the attachment. For experiments to understand their latencies and cross-check their missed heartbeats, we wanted to send some more information. This includes a figure showing the registered beats, status and latencies. The figure is created but it fails to sent via email. We can also consider sending a csv file with the beats for that experiment within 24 hr for user to play around with.