Appsec breaks uploading of large files to Nextcloud

[YeapGuy]

Hi. I'm using NPMplus as a reverse proxy for my Nextcloud installation - it includes this bouncer with appsec features.
When I enable appsec and try to upload a large file using WebDAV, I get errors like this in my log:

2024/06/06 10:15:35 [warn] 19486#19486: *35705 a client request body is buffered to a temporary file /usr/local/nginx/client_body_temp/0000000018 while reading request body, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00001 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:15:37 [warn] 19486#19486: *35705 a client request body is buffered to a temporary file /usr/local/nginx/client_body_temp/0000000019 while reading request body, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00002 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:16:01 [error] 19486#19486: *35705 lua tcp socket write timed out, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00002 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:16:01 [error] 19486#19486: *35705 [lua] crowdsec.lua:578: AppSecCheck(): Fallback because of err: timeout, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00002 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:16:01 [error] 19486#19486: *35705 [lua] crowdsec.lua:651: Allow(): AppSec check: timeout, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00002 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:16:01 [alert] 19486#19486: *35705 [lua] crowdsec.lua:718: Allow(): [Crowdsec] denied '192.168.0.1' with 'ban' (by appsec), client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00002 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:16:01 [warn] 19486#19486: *35705 a client request body is buffered to a temporary file /usr/local/nginx/client_body_temp/0000000020 while reading request body, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00003 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:16:41 [error] 19486#19486: *35705 lua tcp socket write timed out, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00003 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:16:41 [error] 19486#19486: *35705 [lua] crowdsec.lua:578: AppSecCheck(): Fallback because of err: timeout, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00003 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:16:41 [error] 19486#19486: *35705 [lua] crowdsec.lua:651: Allow(): AppSec check: timeout, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00003 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:16:41 [alert] 19486#19486: *35705 [lua] crowdsec.lua:718: Allow(): [Crowdsec] denied '192.168.0.1' with 'ban' (by appsec), client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00003 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:16:42 [warn] 19486#19486: *35705 a client request body is buffered to a temporary file /usr/local/nginx/client_body_temp/0000000021 while reading request body, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00004 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:17:30 [error] 19486#19486: *35705 lua tcp socket read timed out, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00004 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:17:30 [error] 19486#19486: *35705 [lua] crowdsec.lua:578: AppSecCheck(): Fallback because of err: timeout, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00004 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:17:30 [error] 19486#19486: *35705 [lua] crowdsec.lua:651: Allow(): AppSec check: timeout, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00004 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:17:30 [alert] 19486#19486: *35705 [lua] crowdsec.lua:718: Allow(): [Crowdsec] denied '192.168.0.1' with 'ban' (by appsec), client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00004 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:17:32 [warn] 19486#19486: *35705 a client request body is buffered to a temporary file /usr/local/nginx/client_body_temp/0000000022 while reading request body, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00005 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:18:25 [error] 19486#19486: *35705 lua tcp socket write timed out, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00005 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:18:25 [error] 19486#19486: *35705 [lua] crowdsec.lua:578: AppSecCheck(): Fallback because of err: timeout, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00005 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:18:25 [error] 19486#19486: *35705 [lua] crowdsec.lua:651: Allow(): AppSec check: timeout, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00005 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:18:25 [alert] 19486#19486: *35705 [lua] crowdsec.lua:718: Allow(): [Crowdsec] denied '192.168.0.1' with 'ban' (by appsec), client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00005 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:18:27 [warn] 19486#19486: *35705 a client request body is buffered to a temporary file /usr/local/nginx/client_body_temp/0000000023 while reading request body, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00006 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:19:51 [error] 19486#19486: *35705 lua tcp socket write timed out, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00006 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:19:51 [error] 19486#19486: *35705 [lua] crowdsec.lua:578: AppSecCheck(): Fallback because of err: timeout, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00006 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:19:51 [error] 19486#19486: *35705 [lua] crowdsec.lua:651: Allow(): AppSec check: timeout, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00006 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:19:51 [alert] 19486#19486: *35705 [lua] crowdsec.lua:718: Allow(): [Crowdsec] denied '192.168.0.1' with 'ban' (by appsec), client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00006 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:19:51 [warn] 19486#19486: *35705 a client request body is buffered to a temporary file /usr/local/nginx/client_body_temp/0000000024 while reading request body, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00007 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:21:43 [error] 19486#19486: *35705 lua tcp socket read timed out, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00007 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:21:43 [error] 19486#19486: *35705 [lua] crowdsec.lua:578: AppSecCheck(): Fallback because of err: timeout, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00007 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:21:43 [error] 19486#19486: *35705 [lua] crowdsec.lua:651: Allow(): AppSec check: timeout, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00007 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:21:43 [alert] 19486#19486: *35705 [lua] crowdsec.lua:718: Allow(): [Crowdsec] denied '192.168.0.1' with 'ban' (by appsec), client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00007 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:21:44 [warn] 19486#19486: *35705 a client request body is buffered to a temporary file /usr/local/nginx/client_body_temp/0000000027 while reading request body, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00008 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:22:14 [error] 19486#19486: *35705 lua tcp socket write timed out, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00008 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:22:14 [error] 19486#19486: *35705 [lua] crowdsec.lua:578: AppSecCheck(): Fallback because of err: timeout, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00008 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:22:14 [error] 19486#19486: *35705 [lua] crowdsec.lua:651: Allow(): AppSec check: timeout, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00008 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:22:14 [alert] 19486#19486: *35705 [lua] crowdsec.lua:718: Allow(): [Crowdsec] denied '192.168.0.1' with 'ban' (by appsec), client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00008 HTTP/1.1", host: "my-nextcloud.redacted.tld"
2024/06/06 10:22:16 [warn] 19486#19486: *35705 a client request body is buffered to a temporary file /usr/local/nginx/client_body_temp/0000000028 while reading request body, client: 192.168.0.1, server: my-nextcloud.redacted.tld, request: "PUT /remote.php/dav/uploads/user/3803052717/00009 HTTP/1.1", host: "my-nextcloud.redacted.tld"

And the memory usage goes like this:

I can't use appsec due to this.

I was directed here by NPMplus' developer from here ZoeyVid/NPMplus#873

[blotus]

Hello,

So that we can investigate, how large are the files you are attempting to upload ?

The messages a client request body is buffered to a temporary file are normal, nginx will buffer the file to the disk if the size is greater than client_body_buffer_size, so you can ignore those.

Analysing requests with huge files can be problematic, there's not much we can do if we need to handle 1GB of data for example.
We'll probably look into adding a configuration option to disable body inspection if the request is bigger than a given size (although this can be a kind of foot gun, as this could introduce trivial bypasses) and probably introduce per location configuration to have more granular control.

[YeapGuy]

The file I was trying to upload over 4GB, but I can reproduce this with a ~1GB file too.
Yeah, I understand that it takes a lot of memory and time to analyze such huge requests, no wonder it times out. Per location configuration would be the way to go here I think - disabling appsec on just the upload routes.

[VeryFunnyMonkey]

The file I was trying to upload over 4GB, but I can reproduce this with a ~1GB file too. Yeah, I understand that it takes a lot of memory and time to analyze such huge requests, no wonder it times out. Per location configuration would be the way to go here I think - disabling appsec on just the upload routes.

Hey were you able to come up with a workaround for this? I am also facing this exact same issue.

[YeapGuy]

Nope. I had to turn off appsec for Nextcloud.

[VeryFunnyMonkey]

Nope. I had to turn off appsec for Nextcloud.

Is there a way to turn it off specifically for one proxy host in npmplus? Or did you have to turn off appsec for your entire npmplus install?

[YeapGuy]

You can either turn off appsec for your entire install or turn off modsec (which includes appsec) for one proxy host.