Deploys are using the wrong image (no updates)

[freeidah]

I've used kamal previously and had no issue but I'm getting a bit frustrated with my most recent project. Changes that I commit and deploy are not reflected on the server, it seems stuck on the first version I deployed. Any help would be appreciated 🙏

Basically my deploy failed because of a misconfiguration of database.yml. I made the appropriate changes but subsequent deploys failed with the same error. I ssh'd into a container using:

kamal deploy -d staging
kamal app exec -i bash -d staging
cat config/database.yml

I could see that my latest change had not been applied, even though they were committed (locally and on remote).

To work around it I ssh'd onto the server and ran

docker rmi -f $(docker images -q)

Then the next deploy the changes take hold:

kamal deploy -d staging
kamal app exec -i bash -d staging
cat config/database.yml

I see the expected updates.

However, any subsequent changes are again not picked up again and I have to do the whole clean out of docker on the target server (web) and run the deploy once more.

Has anyone seen this before, or experienced anything like this? I have a feeling I'm missing something pretty fundamental.

Other things I have tried:

• Disabling builder caching in deploy.yml
• New ec2 instance
• Docker image builds fine locally

Version & setup info

• Kamal version 1.8.0
• Server is AWS Amazon Linux 2
• Docker version 25.0.3, build 4debf41
• Local machine mac M2 OSX Sonoma

It's an Older rails app v6.1.7.8 using webpacker.

The Dockerfile:

# syntax = docker/dockerfile:1

# Make sure RUBY_VERSION matches the Ruby version in .ruby-version and Gemfile
ARG RUBY_VERSION=3.3.4
FROM registry.docker.com/library/ruby:$RUBY_VERSION-slim AS base

# Rails app lives here
WORKDIR /rails

# Install base packages
RUN apt-get update -qq && \
    apt-get install --no-install-recommends -y curl libjemalloc2 libvips postgresql-client && \
    rm -rf /var/lib/apt/lists /var/cache/apt/archives

# Set production environment
ENV RAILS_ENV="production" \
    BUNDLE_DEPLOYMENT="1" \
    BUNDLE_PATH="/usr/local/bundle" \
    BUNDLE_WITHOUT="development"

# Throw-away build stage to reduce size of final image
FROM base AS build

# Install packages needed to build gems and node modules
RUN apt-get update -qq && \
    apt-get install --no-install-recommends -y build-essential git libpq-dev libvips node-gyp pkg-config python-is-python3 && \
    rm -rf /var/lib/apt/lists /var/cache/apt/archives

# Install JavaScript dependencies
ARG NODE_VERSION=20.2.0
ARG YARN_VERSION=1.22.19
ENV PATH=/usr/local/node/bin:$PATH
RUN curl -sL https://github.com/nodenv/node-build/archive/master.tar.gz | tar xz -C /tmp/ && \
    /tmp/node-build-master/bin/node-build "${NODE_VERSION}" /usr/local/node && \
    npm install -g yarn@$YARN_VERSION && \
    rm -rf /tmp/node-build-master

# Install application gems
COPY Gemfile Gemfile.lock ./
RUN bundle install && \
    rm -rf ~/.bundle/ "${BUNDLE_PATH}"/ruby/*/cache "${BUNDLE_PATH}"/ruby/*/bundler/gems/*/.git && \
    bundle exec bootsnap precompile --gemfile

# Install node modules
COPY package.json yarn.lock ./
RUN yarn install --frozen-lockfile

# Copy application code
COPY . .

# Precompile bootsnap code for faster boot times
RUN bundle exec bootsnap precompile app/ lib/

# Precompiling assets for production without requiring secret RAILS_MASTER_KEY
# RUN SECRET_KEY_BASE_DUMMY=1 ./bin/rails assets:precompile

# SECRET_KEY_BASE_DUMMY=1 is only available on Rails 7.1+,
# workaround for Rails 7.0 and below.
# NODE_OPTIONS=--openssl-legacy-provider needed while on old webpacker
RUN SECRET_KEY_BASE=1 DATABASE_URL=postgresql://dummy@localhost/dummy RAILS_ENV=$RAILS_ENV \
NODE_OPTIONS="--openssl-legacy-provider" bundle exec rake assets:precompile

RUN rm -rf node_modules

# Final stage for app image
FROM base

# Copy built artifacts: gems, application
COPY --from=build "${BUNDLE_PATH}" "${BUNDLE_PATH}"
COPY --from=build /usr/local/node /usr/local/node
COPY --from=build /rails /rails

# Ensure Node.js is available in the PATH
ENV PATH=/usr/local/node/bin:$PATH

# Run and own only the runtime files as a non-root user for security
RUN groupadd --system --gid 1000 rails && \
    useradd rails --uid 1000 --gid 1000 --create-home --shell /bin/bash && \
    chown -R rails:rails db log storage tmp
USER 1000:1000

# Entrypoint prepares the database.
ENTRYPOINT ["/rails/bin/docker-entrypoint"]

# Start the server by default, this can be overwritten at runtime
EXPOSE 3000
CMD ["./bin/rails", "server"]

deploy.yml

service: kamal-test

# Name of the container image.
image: [MY IMAGE NAME]

# Credentials for your image host.
registry:
  server: ghcr.io
  username: freeidah
  password:
    - KAMAL_REGISTRY_PASSWORD

ssh:
  user: ec2-user

asset_path: /rails/public/assets

builder:
  remote:
    arch: amd64
    host: ssh://ec2-user@[BUILDER IP]
  cache:
      type: registry
      options: mode=max

deploy.staging.yml

servers:
  web:
    hosts:
      - [STAGING IP]
env:
  clear:
    RAILS_ENV: staging
    APP_REVISION: <%= `git rev-parse HEAD`.strip %>
  secret:
    - RAILS_MASTER_KEY

[freeidah]

Worked out what was going on. New containers were going up but failing the healthcheck. The app was ok but took longer than the default time. Increasing that meant I got a successful deploy and the container was tagged appropriately.

Using the kamal app exec -i bash -d staging while in a fail state seemed to always default to the first container in the pile which was out of date (I've not looked into the logic of that).