mismatch between IPs from CrowdSec servers and those seen in nftables, reverse octet order (endianness)

[bughunter2]

What happened?

2024-05-13

Problem:
--------

These symptoms seem to hint at one or more bugs.

There's a mismatch between the octet order in both the IPv4 and IPv6 addresses
added to nftables and the IP addresses returned by the CrowdSec servers and those
shown in the crowdsec-firewall-bouncer.log file (when log_level is set to debug).

At the very least, this can be a source of confusion.
At worst, it means CrowdSec is banning the wrong IP addresses.

And because there's a mismatch between what the CrowdSec servers return and
the IP addresses that get added to nftables, it might actually be
that CrowdSec is banning the wrong IPs. I surely hope this is not the case?

Also, you can't easily 'grep' your nftables for an IP address from
a CrowdSec log file, and instead you have to reverse the octets before
looking for that address in your firewall.

Questions:
----------

  - It seems to be an endianness bug, but which of the following, if any, is true?

  - Does CrowdSec ban the wrong IP addresses (reverse octet order)?

  - Do the CrowdSec servers return the wrong IP addresses (reverse octet order) through their HTTP API?

  - Does the crowdsec-firewall-bouncer.log file show IP addresses in the wrong octet order?

  - Are IP addresses added in the wrong octet order to nftables?

Troubleshooting details:
------------------------

1. Stop the crowdsec services.
2. Set crowdsec-firewall-bouncer's log_level to 'debug'.
3. Start the crowdsec service.
4. Start the crowdsec-firewall-bouncer service.
5. Observe the following:

The CrowdSec servers return something like this (output shown only partially here):

/var/log/crowdsec-firewall-bouncer.log contains a line like this (contains our example: 101.42.161.99):

time="13-05-2024 22:31:12" level=debug msg="Response: HTTP/1.1 200 OK\r\nTransfer-Encoding: chunked\r\n
Content-Type: application/json; charset=utf-8\r\nDate: Mon, 13 May 2024 20:31:12 GMT\r\n\r\n231e5a\r\n{\"deleted\":null,
"new\":[{\"duration\":\"155h56m43.973575713s\",\"id\":14973,\"origin\":\"CAPI\",\"scenario\":\"crowdsecurity/http-bad-user-
agent\",\"scope\":\"Ip\",\"type\":\"ban\",\"value\":\"34.219.212.12\"},{\"duration\":\"155h56m43.973571691s\",\"id\":
14974,\"origin\":\"CAPI\",\"scenario\":\"crowdsecurity/http-probing\",\"scope\":\"Ip\",\"type\":\"ban\",\"value\":
\"4.227.114.13\"},{\"duration\":\"155h56m43.97356988s\",\"id\":14975,\"origin\":\"CAPI\",\"scenario\"...
...
...
r-agent\",\"scope\":\"Ip\",\"type\":\"ban\",\"value\":\"35.227.24.137\"},{\"duration\":\"155h56m43.97353777s\",\"id\":
14995,\"origin\":\"CAPI\",\"scenario\":\"crowdsecurity/ssh-slow-bf\",\"scope\":\"Ip\",\"type\":\"ban\",\"value\":
\"101.42.161.99\"},{\"duration\":\"155h56m43.973536851s\",\"id\":14996,\"origin\":\"CAPI\",\"scenario\":\"crowdsecurity/
http-bad-user-agent\",\"scope\":\"Ip\",\"type\":\"ban\",\"value\":\"54.185.132.207\"},{\"duration\":
\"155h56m43.973535581s\",\"id\":14997,\"origin\":\"CAPI\",\"scenario\":\"crowdsecurity/http-bad-user-agent\",\"...
...

/var/log/crowdsec-firewall-bouncer.log contains a line like this:

time="13-05-2024 22:31:12" level=debug msg="Adding '101.42.161.99' for '155h56m43.97353777s'"

... but the 'nft list ruleset' output does NOT contain IPv4 address 101.42.161.99.
However, it does contain the IPv4 address 99.161.42.101 ...:

# nft list ruleset | grep -in "101.42.161.99" # <== No matches.
# nft list ruleset | grep -in "99.161.42.101" # <== Does match:

3173:			     99.161.42.101 timeout 6d11h56m43s972ms expires 6d11h55m29s16ms, 99.166.70.54 timeout 6d18h56m43s968ms expires 6d18h55m31s604ms,

The same happens with IPv6 addresses:
-------------------------------------

/var/log/crowdsec-firewall-bouncer.log contains a line like this:

time="13-05-2024 22:31:19" level=debug msg="adding 2a03:4000:0:39c:e40c:92ff:fe2d:8e38 to buffer "

While 'nft list ruleset' contains:

# nft list ruleset | grep -in "ff92"

7596:			     388e:2dfe:ff92:ce4:9c03:0:40:32a timeout 6d18h56m43s968ms expires 6d18h55m35s444ms,

So which is correct?
  - 2a03:4000:0:39c:e40c:92ff:fe2d:8e38 # Returned by CrowdSec servers and shown in crowdsec-firewall-bouncer.log
  - 388e:2dfe:ff92:ce4:9c03:0:40:32a    # Shown in nft list ruleset.

Again, we can see that the octet order is reversed:

It seems to be "double reversed", in the sense that within each octet, the order is reversed as well.
For example, the beginning was 2a03, but it moves to the end and becomes 32a.

2a03 => 32a
4000 => 40
0    => 0
39c  => 9c03
e40c => ce4
92ff => ff92
fe2d => 2dfe
8e38 => 388e

What did you expect to happen?

That there's no mismatch between the IPv4 and IPv6 addresses returned by the CrowdSec servers, what the crowdsec-firewall-bouncer.log file shows, and what nftables shows (nft list ruleset). They should all be in agreement.

How can we reproduce it (as minimally and precisely as possible)?

See above.

Anything else we need to know?

No response

Crowdsec version

$ cscli version
2024/05/14 00:33:26 version: v1.4.6-6~deb12u1-debian
2024/05/14 00:33:26 Codename: alphaga
2024/05/14 00:33:26 BuildDate: 2023-07-15_09:29:33
2024/05/14 00:33:26 GoVersion: 1.19.8
2024/05/14 00:33:26 Platform: linux
2024/05/14 00:33:26 Constraint_parser: >= 1.0, <= 2.0
2024/05/14 00:33:26 Constraint_scenario: >= 1.0, < 3.0
2024/05/14 00:33:26 Constraint_api: v1
2024/05/14 00:33:26 Constraint_acquis: >= 1.0, < 2.0

OS version

$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

$ uname -a
6.1.0-20-amd64

Enabled collections and parsers

$ cscli hub list -o raw
crowdsecurity/apache2,enabled,0.1,apache2 support : parser and generic http scenarios ,collections
crowdsecurity/base-http-scenarios,enabled,0.6,http common : scanners detection,collections
crowdsecurity/http-cve,enabled,1.9,,collections
crowdsecurity/linux,enabled,0.2,core linux support : syslog+ssh,collections
crowdsecurity/nginx,enabled,0.2,nginx support : parser and generic http scenarios,collections
crowdsecurity/sshd,enabled,0.2,sshd support : parser and brute-force detection,collections
crowdsecurity/apache2-logs,enabled,1.3,Parse Apache2 access and error logs,parsers
crowdsecurity/dateparse-enrich,enabled,0.2,,parsers
crowdsecurity/http-logs,enabled,1.1,"Parse more Specifically HTTP logs, such as HTTP Code, HTTP path, HTTP args and if its a static ressource",parsers
crowdsecurity/nginx-logs,enabled,1.3,Parse nginx access and error logs,parsers
crowdsecurity/sshd-logs,enabled,2.0,Parse openSSH logs,parsers
crowdsecurity/syslog-logs,enabled,0.8,,parsers
crowdsecurity/whitelists,enabled,0.2,Whitelist events from private ipv4 addresses,parsers
crowdsecurity/CVE-2022-26134,enabled,0.1,Detect CVE-2022-26134 exploits,scenarios
crowdsecurity/CVE-2022-35914,enabled,0.1,Detect CVE-2022-35914 exploits,scenarios
crowdsecurity/CVE-2022-37042,enabled,0.1,Detect CVE-2022-37042 exploits,scenarios
crowdsecurity/CVE-2022-40684,enabled,0.2,Detect cve-2022-40684 exploitation attempts,scenarios
crowdsecurity/CVE-2022-41082,enabled,0.3,Detect CVE-2022-41082 exploits,scenarios
crowdsecurity/CVE-2022-41697,enabled,0.1,Detect CVE-2022-41697 enumeration,scenarios
crowdsecurity/CVE-2022-42889,enabled,0.2,Detect CVE-2022-42889 exploits (Text4Shell),scenarios
crowdsecurity/CVE-2022-44877,enabled,0.2,Detect CVE-2022-44877 exploits,scenarios
crowdsecurity/CVE-2022-46169,enabled,0.1,Detect CVE-2022-46169 brute forcing,scenarios
crowdsecurity/apache_log4j2_cve-2021-44228,enabled,0.4,Detect cve-2021-44228 exploitation attemps,scenarios
crowdsecurity/f5-big-ip-cve-2020-5902,enabled,0.1,Detect cve-2020-5902 exploitation attemps,scenarios
crowdsecurity/fortinet-cve-2018-13379,enabled,0.2,Detect cve-2018-13379 exploitation attemps,scenarios
crowdsecurity/grafana-cve-2021-43798,enabled,0.1,Detect cve-2021-43798 exploitation attemps,scenarios
crowdsecurity/http-backdoors-attempts,enabled,0.3,Detect attempt to common backdoors,scenarios
crowdsecurity/http-bad-user-agent,enabled,0.7,Detect bad user-agents,scenarios
crowdsecurity/http-crawl-non_statics,enabled,0.3,Detect aggressive crawl from single ip,scenarios
crowdsecurity/http-cve-2021-41773,enabled,0.1,cve-2021-41773,scenarios
crowdsecurity/http-cve-2021-42013,enabled,0.1,cve-2021-42013,scenarios
crowdsecurity/http-generic-bf,enabled,0.4,Detect generic http brute force,scenarios
crowdsecurity/http-open-proxy,enabled,0.3,Detect scan for open proxy,scenarios
crowdsecurity/http-path-traversal-probing,enabled,0.2,Detect path traversal attempt,scenarios
crowdsecurity/http-probing,enabled,0.2,Detect site scanning/probing from a single ip,scenarios
crowdsecurity/http-sensitive-files,enabled,0.2,"Detect attempt to access to sensitive files (.log, .db ..) or folders (.git)",scenarios
crowdsecurity/http-sqli-probing,enabled,0.2,A scenario that detects SQL injection probing with minimal false positives,scenarios
crowdsecurity/http-xss-probing,enabled,0.2,A scenario that detects XSS probing with minimal false positives,scenarios
crowdsecurity/jira_cve-2021-26086,enabled,0.1,Detect Atlassian Jira CVE-2021-26086 exploitation attemps,scenarios
crowdsecurity/nginx-req-limit-exceeded,enabled,0.1,Detects IPs which violate nginx's user set request limit.,scenarios
crowdsecurity/pulse-secure-sslvpn-cve-2019-11510,enabled,0.2,Detect cve-2019-11510 exploitation attemps,scenarios
crowdsecurity/spring4shell_cve-2022-22965,enabled,0.2,Detect cve-2022-22965 probing,scenarios
crowdsecurity/ssh-bf,enabled,0.1,Detect ssh bruteforce,scenarios
crowdsecurity/ssh-slow-bf,enabled,0.2,Detect slow ssh bruteforce,scenarios
crowdsecurity/thinkphp-cve-2018-20062,enabled,0.3,Detect ThinkPHP CVE-2018-20062 exploitation attemps,scenarios
crowdsecurity/vmware-cve-2022-22954,enabled,0.2,Detect Vmware CVE-2022-22954 exploitation attempts,scenarios
crowdsecurity/vmware-vcenter-vmsa-2021-0027,enabled,0.1,Detect VMSA-2021-0027 exploitation attemps,scenarios
ltsich/http-w00tw00t,enabled,0.1,detect w00tw00t,scenarios

Acquisition config

```console # On Linux: $ cat /etc/crowdsec/acquis.yaml /etc/crowdsec/acquis.d/* # paste output here

On Windows:

C:> Get-Content C:\ProgramData\CrowdSec\config\acquis.yaml

paste output here

Config show

$ cscli config show
Global:
   - Configuration Folder   : /etc/crowdsec
   - Data Folder            : /var/lib/crowdsec/data
   - Hub Folder             : /var/lib/crowdsec/hub
   - Simulation File        : /etc/crowdsec/simulation.yaml
   - Log Folder             : /var/log/
   - Log level              : info
   - Log Media              : file
Crowdsec:
  - Acquisition File        : /etc/crowdsec/acquis.yaml
  - Parsers routines        : 1
  - Acquisition Folder      : /etc/crowdsec/acquis.d
cscli:
  - Output                  : human
  - Hub Branch              : 
  - Hub Folder              : /var/lib/crowdsec/hub
Local API Server:
  - Listen URL              : 127.0.0.1:8080
  - Profile File            : /etc/crowdsec/profiles.yaml
  - Trusted IPs: 
      - 127.0.0.1
      - ::1
  - Database:
      - Type                : sqlite
      - Path                : /var/lib/crowdsec/data/crowdsec.db
      - Flush age           : 7d
      - Flush size          : 5000

Prometheus metrics

$ cscli metrics
# paste output here

Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.

[github-actions]

@bughunter2: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

1. Check Crowdsec Documentation to see if your issue can be self resolved.
2. You can also join our Discord.
3. Check Releases to make sure your agent is on the latest version.

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[blotus]

Hello,

Looking at the version number of crowdsec, it seems that you are using the packages available in your distro repository.
I cannot reproduce the issue with either the latest crowdsec + firewall bouncer version installed from our own repositories OR with a manual build on a debian 12 of crowdsec 1.4.6 and firewall bouncer 0.0.25 (which are the version shipped in debian), which would indicate the issue is coming specifically from the debian build.

The easiest fix for you would be to switch to our repositories (https://docs.crowdsec.net/docs/next/getting_started/install_crowdsec), that way you will be able to install much more recent and "known to work" versions.

Now, for the issue itself, it seems like it's coming from the firewall bouncer: when I manually add a decision in crowdsec and query LAPI directly, I see the proper IP in the stream, so the only place for the IP to get reversed is inside the bouncer itself.
I'll try to contact the debian package maintainer to see if he has any idea where this could come from.

(I'm also moving the issue to the firewall bouncer repository, and I'll leave it open for now just for tracking purposes)

[github-actions]

@bughunter2: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

1. Check Documentation to see if your issue can be self resolved.
2. You can also join our Discord

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[github-actions]

@bughunter2: There are no 'kind' label on this issue. You need a 'kind' label to start the triage process.

• /kind feature
• /kind enhancement
• /kind bug
• /kind packaging

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[CyrilBrulebois]

Thanks for filing/forwarding the report, I'm able to reproduce this on my bookworm/amd64 system and I'm equally surprised and ashamed to have missed it until today…

That's likely an issue in the google/nftables module, and there's an issue + associated commit that looks promising:

• AddSet IPv4 wrong byte order on Ubuntu 22.04 google/nftables#225
• google/nftables@d746ecb

I'll check what happens when I cherry-pick that in the Debian package, and rebuild the bouncer against it. If that helps, I'll upload the package to unstable and request a rebuild of the bouncer (or perform an upload if there are other changes to include along the way). Then I'll file a request to get that updated in stable as well (plus a rebuild).

Sorry about that…

[CyrilBrulebois]

I've just opened https://bugs.debian.org/1071247 for tracking purposes.

TL;DR: Applying the aforementioned patch and rebuilding the bouncer against it fixes the issue on LE systems, and doesn't regress on BE systems. I'll coordinate remediation with the security team (if that warrants a DSA) or with the release team (if it doesn't).

[CyrilBrulebois]

I haven't heard back from the security team yet, but I've just uploaded both packages (golang-github-google-nftables and crowdsec-firewall-bouncer) to unstable, which urgency=high. Hopefully they'll reach testing soon.

As for bookworm, I've published rebuilds of both packages with version numbers that match what would be used for bookworm-security or bookworm-proposed-updates in a custom repository, which can be configured with:

deb [trusted=yes] https://crowdsec.apt.debamax.com/debian bookworm main

(The signing key is available at that URL if you wish to configure it.)

[CyrilBrulebois]

The Debian security team decided this didn't warrant a security announcement, and that we should get this fixed via the upcoming point release instead: following a green light from the release team, I've just uploaded both packages. They should be available shortly in the bookworm-proposed-updates repository, and be merged into the bookworm one during the 12.6 point release, scheduled at the end of the month.

[CyrilBrulebois]

Packages have been accepted, built, and published in bookworm-proposed-updates:

kibi@tokyo:~$ apt-cache policy crowdsec-firewall-bouncer 
crowdsec-firewall-bouncer:
  Installed: (none)
  Candidate: 0.0.25-4~deb12u1
  Version table:
→    0.0.25-4~deb12u1 500
→       500 http://deb.debian.org/debian bookworm-proposed-updates/main amd64 Packages
     0.0.25-3 500
        500 http://deb.debian.org/debian bookworm/main amd64 Packages

(The fixed version is highlighted with arrows.)

[LaurenceJJones]

Thank you @CyrilBrulebois for all the updates on the matter, as this seems to be backported and fixed can we class the issue as resolved. Feel free @bughunter2 to unresolved the issue if this doesn't satisfy as a resolution.