All agents offline (agent overdue)

[tekkiit]

Server Info (please complete the following information):

• OS: [e.g. Ubuntu 20.04, Debian 10]
• Browser: [e.g. chrome, safari]
• RMM Version (as shown in top left of web UI): v0.15.8 (recent upgrade from v0.15.7 - issue was present on v0.15.7

Installation Method:

• Standard
• Docker

Agent Info (please complete the following information):

• Agent version (as shown in the 'Summary' tab of the agent from web UI):
• Agent OS: [e.g. Win 10 v2004, Server 2012 R2]

Describe the bug
Every agent showing as offline (agent overdue).
Upgraded from v0.15.7 to v0.15.8 - no change

agents are a mix of linux, windows 10 and macos.
I can confirm a majority of the computers are online

thanks

[Serenfo]

Renew certificat =)

[tekkiit]

Hi,
Certs were renewed before this issue. I re ran the ./update.sh --force script, but no change

[wh1te909]

yea check your cert. what's the output of sudo systemctl status --full --no-pager nats

[tekkiit]

● nats.service - NATS Server
Loaded: loaded (/etc/systemd/system/nats.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2023-03-29 22:45:21 UTC; 23h ago
Main PID: 649360 (nats-server)
Tasks: 8 (limit: 2336)
Memory: 32.3M
CPU: 3min 7.734s
CGroup: /system.slice/nats.service
└─649360 /usr/local/bin/nats-server -c /rmm/api/tacticalrmm/nats-rmm.conf

Mar 30 22:35:53 XXXX nats-server[649360]: [649360] 2023/03/30 22:35:53.985596 [ERR] XXXX:52118 - cid:25390 - TLS handshake error: remote error: tls: bad certificate
Mar 30 22:35:56 XXXX nats-server[649360]: [649360] 2023/03/30 22:35:56.827936 [ERR] XXXX:52122 - cid:25391 - TLS handshake error: remote error: tls: bad certificate
Mar 30 22:36:04 XXXX nats-server[649360]: [649360] 2023/03/30 22:36:04.082486 [ERR] XXXX:51896 - cid:25392 - TLS handshake error: remote error: tls: bad certificate
Mar 30 22:36:06 XXXX nats-server[649360]: [649360] 2023/03/30 22:36:06.691070 [ERR] XXXX:51906 - cid:25393 - TLS handshake error: remote error: tls: bad certificate
Mar 30 22:36:09 XXXX nats-server[649360]: [649360] 2023/03/30 22:36:09.547466 [ERR] XXXX:51914 - cid:25394 - TLS handshake error: remote error: tls: bad certificate
Mar 30 22:36:12 XXXX nats-server[649360]: [649360] 2023/03/30 22:36:12.225299 [ERR] XXXX:51930 - cid:25395 - TLS handshake error: remote error: tls: bad certificate
Mar 30 22:36:19 XXXX nats-server[649360]: [649360] 2023/03/30 22:36:19.337120 [ERR] XXXX:54092 - cid:25396 - TLS handshake error: remote error: tls: bad certificate
Mar 30 22:36:21 XXXX nats-server[649360]: [649360] 2023/03/30 22:36:21.690284 [ERR] XXXX:54106 - cid:25397 - TLS handshake error: remote error: tls: bad certificate
Mar 30 22:36:24 XXXX nats-server[649360]: [649360] 2023/03/30 22:36:24.016853 [ERR] XXXX:51232 - cid:25398 - TLS handshake error: remote error: tls: bad certificate
Mar 30 22:36:26 XXXX nats-server[649360]: [649360] 2023/03/30 22:36:26.756379 [ERR] XXXX:51240 - cid:25399 - TLS handshake error: remote error: tls: bad certificate

[wh1te909]

there u go, bad cert. sounds like they didn't renew properly

[tekkiit]

Thanks for the pointer.
I loaded a server snapshot and went through the cert renewal process again. Only difference from previous attempt was that I renewed wildcard domain instead of each domain individually.
Agents are back online.
Thanks.

[wh1te909]

np yea here are the docs for how to update cert (it has wildcard method there): https://docs.tacticalrmm.com/update_server/#keeping-your-lets-encrypt-ssl-certificate-up-to-date

[silversword411]

Also...we're just wild guessing here because you've provided no usable troubleshooting data.

https://docs.tacticalrmm.com/troubleshooting/

This is almost a perfect textbook example of the wrong way to ask a question
https://discord.com/channels/736478043522072608/921501747648811038/936258143380443226