Unable to find image

[etmartinkazoo]

I have been using Kamal with Rails apps and love it.

Recently one of my apps failed to build "container unhealthy"

I ran kamal healthcheck and got the following:

ERROR (SSHKit::Command::Failed): Exception while executing on host <redacted IP Address>: docker exit status: 125
docker stdout: Nothing written
docker stderr: Unable to find image 'user/image:hash' locally
docker: Error response from daemon: manifest for user/image:hash not found: manifest unknown: manifest unknown.
See 'docker run --help'.

I have been searching for a solution for days and would love some guidance if anyone has had a similar issue.

Thank you!

[nickhammond]

@etmartinkazoo Is the user/image:hash portion what it's actually saying or redacted? This is for a role and not an accessory?

[etmartinkazoo]

The user/image:hash is redacted.

Here is an example: myusername/coolcodeproject

It is really strange

[nickhammond]

@etmartinkazoo If you run a docker image ls on the server I'm guessing it's not in that list? Any other hashes?

[etmartinkazoo]

No, it seems to be a local docker issue. My other Kamal projects are working so I am truly perplexed.

[etmartinkazoo]

Here is some more:

ERROR (SSHKit::Command::Failed): Exception while executing on host <IP Address>: docker exit status: 125
docker stdout: Nothing written
docker stderr: Unable to find image '<user>/<image>:812d99ab118162281d753221871ec84d0a34e839_uncommitted_3605a0e352a18206' locally
docker: Error response from daemon: manifest for <user>/<image>:812d99ab118162281d753221871ec84d0a34e839_uncommitted_3605a0e352a18206 not found: manifest unknown: manifest unknown.
See 'docker run --help'.

[nickhammond]

Oh, so the healthcheck is probably going to fail when you run it one-off like that. Notice the 812d99ab118162281d753221871ec84d0a34e839_uncommitted_3605a0e352a18206 on the image name. Have you deployed with that specific version? You can see the image versions on your server by running kamal app images.

The main issue looks to be rails credentials decryption which is the error that's raising in your entrypoint script ActiveSupport::MessageEncryptor::InvalidMessage:. Are you using Rails credentials? How can you tell your master key is getting loaded properly?

[etmartinkazoo]

It is strange because I have the correct mater.key in the config directory. I have that image on the server but cannot boot from it...

[nickhammond]

You have config/master.key and config/credentials.yml.enc in your running container? I'd try removing the db:prepare line and just do a kamal app exec -i "/bin/bash" and poke around in there to confirm the credentials are in there and the correct ones.

[etmartinkazoo]

@nickhammond - Thank you so much. Your help was extremely helpful in getting to the bottom of this. It ended up being an error in the application and your last suggestions helped in that debugging. Thank you so very much!

[nickhammond]

@etmartinkazoo No problem, glad you got it figured out! Can you mark this one answered?