Inconsistent /var/www/html ownership in lima hosted ddev

[deviantintegral]

I ran into difficulties on a project running Task, this time in a project not using mutagen. It turned out the problem is this line, and it's failing on the ownership check:

https://github.com/go-task/task/blob/fb9f6c20abb95b5cd120c49507492cf509c828f7/taskfile/taskfile.go#L155

I spun up a basic Drupal 10 project I had, and it exhibited the same behaviour. Yet, when I looked with ddev ssh, the ownership looked fine. After debugging task, I did confirm that the owner was being returned as root, and not my user account.

So why would ddev ssh show different ownership than ddev exec?

Check out the differences in this output, specifically the Uid / Gid:

$ docker exec -it ddev-drupal-10-composer-web stat /var/www/html
  File: /var/www/html
  Size: 384       	Blocks: 0          IO Block: 4096   directory
Device: 0,44	Inode: 4           Links: 12
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-04-16 23:47:01.883048971 +0000
Modify: 2024-04-16 23:47:01.384423359 +0000
Change: 2024-04-16 23:47:01.384423359 +0000
 Birth: -

$ docker exec -it ddev-drupal-10-composer-web bash -c 'stat /var/www/html'
  File: /var/www/html
  Size: 384       	Blocks: 0          IO Block: 4096   directory
Device: 0,44	Inode: 4           Links: 12
Access: (0755/drwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2024-04-16 23:47:01.883048971 +0000
Modify: 2024-04-16 23:47:01.384423359 +0000
Change: 2024-04-16 23:47:01.384423359 +0000
 Birth: -

$ docker exec -it ddev-drupal-10-composer-web bash
andrew@drupal-10-composer-web:/var/www/html$ stat /var/www/html
  File: /var/www/html
  Size: 384       	Blocks: 0          IO Block: 4096   directory
Device: 0,44	Inode: 4           Links: 12
Access: (0755/drwxr-xr-x)  Uid: (  501/  andrew)   Gid: (   20/ dialout)
Access: 2024-04-16 23:47:01.883048971 +0000
Modify: 2024-04-16 23:47:01.384423359 +0000
Change: 2024-04-16 23:47:01.384423359 +0000
 Birth: -

So with this, we know there's no issue in the exec or ssh calls themselves. It's like spawning an interactive app (bash) versus one that isn't (the first two) is causing different ownership to be returned.

This feels like a file handle on /var/www/html has been opened by Docker, something then runs a chown, and exec'ing a non-interactive process is returning cached ownership info.

Yet normally I'd expect the whole file stat structure to be one set of cached object. I can touch the project directory on my host, and the Access / Modify / Change stamps all update, but the root owner is preserved. So weird!

I checked with OrbStack, and this isn't a problem there. Though, strangely ls doesn't show the usual . and .. directories, so I feel like OrbStack is likely also applying its own ownership rules.

I also checked by just using lima (and not colima) with the same behaviour.

Any ideas? I searched both ddev and lima / colima for any chowns, and while this totally feels like it's a chown running on ddev start, I couldn't find one anywhere.

[rfay]

It's super important to know if you were using Virtiofs ? And what Lima version? And recent Lima instance? they just switched to Ubuntu 24.04; you don't get that new version without a rebuild.

It's quite normal on Linux for mounted directories to take the ownership of the underlying mount point, so could that be it?

[rfay]

I can see your results on Lima with VirtioFS. Completely baffling.

[rfay]

I see the same results on a brand new lima instance built today; virtiofs, Ubuntu 24.04.

% ddev exec 'ls -ltd /var/www/html'
drwxr-xr-x 6 testbot dialout 4096 Jun  5 00:28 /var/www/html

Also

$ echo 'ls -ltd /var/www/html' | ddev ssh
drwxr-xr-x 17 root root 544 Jun  5 00:21 /var/www/html

But:

$ docker run -it -u 501:501 --rm -v /Users/rfay/workspace/j:/var/tmp/j busybox ls -ld /var/tmp/j/web
drwxr-xr-x    2 501      501             64 Jun  5 00:38 /var/tmp/j/web

[rfay]

Actually, Virtiofs has had this class of problem for ages, and it's the reason that Docker Desktop for Linux still is unusable. I think it's still this way on docker Desktop with Virtiofs, and it's one of the reasons for Mutagen everywhere.

I think if you use another mount type you won't see this, but Virtiofs has been pretty good with mutagen.

Docker Desktop:

• VirtioFS is not handling permissions as expected. All mount permissions are owned by root regardless of chown. docker/for-mac#6243

[rfay]

Overall, count me baffled.

[deviantintegral]

I ran into this on a project where we're hosting around 300 sites in a single ddev instance - and they all have to have separate code checkouts. Unfortunately, this brings mutagen to its knees (5.1M files!), so I thought I was making things better by using a regular virtiofs mount with a non-synced docker volume for the code.

[deviantintegral]

More cursed ownership behaviour - it looks like ownership is also calculated based on the running user 😮 :

andrew@drupal-10-composer-web:/var/www/html$ ls -la
total 2196
drwxr-xr-x 12 andrew dialout     384 Jun  4 23:24 .
drwxrwxrwx  3 root   root       4096 May 14 14:24 ..
drwxr-xr-x 23 andrew dialout     736 Jun  5 12:00 .ddev
-rw-r--r--  1 andrew dialout     357 Aug  9  2023 .editorconfig
-rw-------  1 andrew dialout    4222 Apr 16 23:44 .gitattributes
drwxr-xr-x 18 andrew dialout     576 Feb 16 00:51 .idea
-rw-------  1 andrew dialout    4824 Apr 16 23:44 composer.json
-rw-------  1 andrew dialout  431587 Apr 16 23:47 composer.lock
-rw-------  1 andrew dialout     497 Nov 17  2023 phpcs.xml.dist
-rw-r--r--  1 andrew dialout 1782576 Feb 15 22:08 umami.sql.gz
drwxr-xr-x 53 andrew dialout    1696 Nov 17  2023 vendor
drwxr-xr-x 20 andrew dialout     640 Feb 14 19:23 web
andrew@drupal-10-composer-web:/var/www/html$ sudo ls -la
total 2196
drwxr-xr-x 12 andrew dialout     384 Jun  4 23:24 .
drwxrwxrwx  3 root   root       4096 May 14 14:24 ..
drwxr-xr-x 23 root   root        736 Jun  5 12:00 .ddev
-rw-r--r--  1 root   root        357 Aug  9  2023 .editorconfig
-rw-------  1 root   root       4222 Apr 16 23:44 .gitattributes
drwxr-xr-x 18 root   root        576 Feb 16 00:51 .idea
-rw-------  1 root   root       4824 Apr 16 23:44 composer.json
-rw-------  1 root   root     431587 Apr 16 23:47 composer.lock
-rw-------  1 root   root        497 Nov 17  2023 phpcs.xml.dist
-rw-r--r--  1 root   root    1782576 Feb 15 22:08 umami.sql.gz
drwxr-xr-x 53 root   root       1696 Nov 17  2023 vendor
drwxr-xr-x 20 root   root        640 Feb 14 19:23 web

I'm going to switch this project to use mutagen again, and exclude the code directories in its config.

[deviantintegral]

It should change the ownership based on using sudo or not though, right? Or is virtiofs doing some magic and showing all files as owned by the user of the running process?

[rfay]

I would expect mounts to happen with the user the container is running as. That works lots of places, including Linux/docker-ce.

rfay@ub-2404:~/workspace/d10$ ddev exec 'ls -ld /var/www/html'
drwxr-xr-x 5 rfay rfay 4096 Apr 27 02:09 /var/www/html

[rfay]

Normally attempts to use sudo to chown or chmod a mounted directory will fail. That's true in most environments, not just Docker bind-mounts.

[deviantintegral]

Normally ownership should be based on running user, and that's what we see most of the time.

Yes, meaning "the user account that ran ddev start"? What's weird is that the ownership seen by running programs changes based on the user the program is running under. ls -la should return consistent results regardless of which user is running it. This is all done with no calls to chown - see how the files change from andrew / root / www-data:

andrew@drupal-10-composer-web:/var/www/html$ ls -la
total 2196
drwxr-xr-x 12 andrew dialout     384 Jun  7 13:56 .
drwxrwxrwx  3 root   root       4096 May 14 14:24 ..
drwxr-xr-x 23 andrew dialout     736 Jun  7 13:55 .ddev
-rw-r--r--  1 andrew dialout     357 Aug  9  2023 .editorconfig
-rw-------  1 andrew dialout    4222 Apr 16 23:44 .gitattributes
drwxr-xr-x 18 andrew dialout     576 Feb 16 00:51 .idea
-rw-------  1 andrew dialout    4824 Apr 16 23:44 composer.json
-rw-------  1 andrew dialout  431587 Apr 16 23:47 composer.lock
-rw-------  1 andrew dialout     497 Nov 17  2023 phpcs.xml.dist
-rw-r--r--  1 andrew dialout 1782576 Feb 15 22:08 umami.sql.gz
drwxr-xr-x 53 andrew dialout    1696 Nov 17  2023 vendor
drwxr-xr-x 20 andrew dialout     640 Feb 14 19:23 web
andrew@drupal-10-composer-web:/var/www/html$ sudo -i
root@drupal-10-composer-web:~# ls -la
total 36
drwx------ 1 root root 4096 Jun  4 20:39 .
drwxr-xr-x 1 root root 4096 Jun  7 13:56 ..
-rw-r--r-- 1 root root  571 Apr 10  2021 .bashrc
drwxr-xr-x 3 root root 4096 Jun  4 20:39 .config
drwxr-xr-x 3 root root 4096 Jun  4 20:39 .local
drwxr-xr-x 4 root root 4096 May 14 13:14 .npm
-rw-r--r-- 1 root root  161 Jul  9  2019 .profile
drwx------ 2 root root 4096 May 14 13:19 .ssh
-rw-r--r-- 1 root root  165 May 14 14:06 .wget-hsts
root@drupal-10-composer-web:~# su -m www-data
bash: /root/.bashrc: Permission denied
www-data@drupal-10-composer-web:~$ cd /var/www/html
www-data@drupal-10-composer-web:/var/www/html$ ls -la
total 2196
drwxr-xr-x 12 www-data www-data     384 Jun  7 13:56 .
drwxrwxrwx  3 root     root        4096 May 14 14:24 ..
drwxr-xr-x 23 www-data www-data     736 Jun  7 13:55 .ddev
-rw-r--r--  1 www-data www-data     357 Aug  9  2023 .editorconfig
-rw-------  1 www-data www-data    4222 Apr 16 23:44 .gitattributes
drwxr-xr-x 18 www-data www-data     576 Feb 16 00:51 .idea
-rw-------  1 www-data www-data    4824 Apr 16 23:44 composer.json
-rw-------  1 www-data www-data  431587 Apr 16 23:47 composer.lock
-rw-------  1 www-data www-data     497 Nov 17  2023 phpcs.xml.dist
-rw-r--r--  1 www-data www-data 1782576 Feb 15 22:08 umami.sql.gz
drwxr-xr-x 53 www-data www-data    1696 Nov 17  2023 vendor
drwxr-xr-x 20 www-data www-data     640 Feb 14 19:23 web

I'd sure be interested in what you're doing with 300 sites in one DDEV instance! And why you're doing it on macOS!

We're auditing a large number of Drupal 7 sites, and while they have mostly common code they also have unique modules. Those modules have been uploaded directly to the host. We don't want to make any changes to the sites, so we aren't converting them to a more traditional Drupal multisite single repo.

By putting them all in ddev, we can easily explore the sites, audit them, run SQL queries and drush commands across all of them, etc. I originally started by putting them into individual ddev projects. While I thought RAM may be the limiting issue, it turned out to be ports because we needed > 254 sites running at once!

We're doing this on macOS because that's what most of our team uses, though Linux is a common second choice.

[rfay]

Yes, meaning "the user account that ran ddev start

Well, specifically I mean the User used to run the container. If you look at your .ddev/.ddev-docker-compose-base.yaml you'll see
user: '$DDEV_UID:$DDEV_GID'
and in the .ddev/docker-compose-full.yaml it's filled out to
user: "501:20" for example.

So we're talking about the (limited) user that DDEV is using to run the container. This is for security and convenience reasons. We don't want containers running as the default root because they can create setuid-root files on the host and other complications. But most of all, we want people to have a comfortable and familiar experience inside the container. In the situation here, all processes inside the container run as uid 501 (which is also added as a username, in this case rfay.

In your situation it looks like you're sometimes running the container directly, and seeing www-data ownership, but that's a different matter.