docs/container: Flesh out a bit more and tweak

There's a whole lot of other stuff to fill out, but let's start.

docs/container.md

@@ -28,14 +28,77 @@ is more interesting:
 
 ## Using custom builds
 
-The ostree container functionality supports layered container images.
-See [fcos-derivation-example](https://github.com/coreos/fcos-derivation-example)
-for an example.
-
-This functionality is explicitly experimental; it is unlikely to break booting
-or anything like that, but the container format or buildsystem may change.
-For example, see [this issue](https://github.com/ostreedev/ostree-rs-ext/issues/159)
-where we may require a command to be run as part of the build in the future.
+The ostree container functionality supports layered container images; you can
+use any container buildsystem you like to add additional layers.
+See [coreos-layering-examples](https://github.com/coreos/coreos-layering-examples)
+many examples.  Note: The functionality here is not specific to (Fedora) CoreOS, but
+it happens to be the farthest along in productizing this at the time of this writing.
+
+This functionality is currently classified as experimental, but it is rapidly
+heading to stabiliziation.
+
+## Filesystem layout model
+
+The ostree model defines effectively 3 partitions:
+
+- `/usr`: Read-only (at runtime, by default) binaries and data files
+- `/etc`: Mutable machine-local configuration files
+- `/var`: All other state
+
+This means that it will not currently work to install e.g. RPM packages
+that add files in `/opt` by default.
+
+### Installing packages
+
+You can use e.g. `rpm-ostree install` to install packages.  This functions
+the same as with e.g. `dnf` or `microdnf`.  It's also possible to use `rpm`
+directly, e.g. `rpm -Uvh https://mirror.example.com/iptables-1.2.3.rpm`.
+
+### Installing config files
+
+You can use any tooling you want to generate config files in `/etc`.  When
+a booted system pulls an updated container images, the changes will also
+be applied.
+
+### Installing non-RPM content
+
+A major change compared to previous rpm-ostree is that it is now clearly
+supported to install non-RPM binaries into `/usr` - these are equally
+"first-class" as binaries from the base image.
+
+### Adapting software
+
+The way ostree works may require some changes in software.
+
+#### Dealing with `/opt`
+
+Some RPMs install files in `/opt`, which in the ostree model is `/var/opt`.
+In  the case where the files in `/opt` are just binaries, one approach
+is to move them at build time:
+
+```
+FROM quay.io/fedora/fedora-coreos:testing-devel
+RUN mkdir /var/opt && \
+    rpm -Uvh https://downloads.linux.hpe.com/repo/stk/rhel/7/x86_64/current/hp-scripting-tools-11.60-20.rhel7.x86_64.rpm && \
+    mv /var/opt/hp/ /usr/lib/hp && \
+    echo 'L /opt/hp - - - - ../../usr/lib/hp' > /usr/lib/tmpfiles.d/hp.conf && \
+    ostree container commit
+```
+
+#### Users and groups
+
+At the current time, `rpm-ostree` will auto-synthesize [systemd-sysusers](https://www.freedesktop.org/software/systemd/man/systemd-sysusers.html)
+snippets when `useradd` or `groupadd` are invoked during the process of e.g. `rpm-ostree install`.
+
+This means that user and group IDs are allocated per machine.
+
+### Using "ostree container commit"
+
+In a container build, it's a current best practice to invoke this at the end
+of each `RUN` instruction (or equivalent).  This will verify compatibility
+of `/var`, and also clean up extraneous files in e.g. `/tmp`.
+
+In the future, this command may perform more operations.
 
 ## Creating base images