Cloudformation Stack stuck in UPDATE_ROLLBACK_FAILED due to ConnectionStack resource failure

[alex-griffiths]

How did you install the Amplify CLI?

npm

If applicable, what version of Node.js are you using?

22.0.0

Amplify CLI Version

12.10.10

What operating system are you using?

Windows

Did you make any manual changes to the cloud resources managed by Amplify? Please describe the changes made.

No manual changes made

Describe the bug

I have just tried to push a change to my development environment. The change was simply adding an Auth group to an existing table.

type Asset @model @auth(rules: [
  {allow: private, provider: iam},
  {allow: groups, groupsField: "admin_groups"},
 + {allow: groups, groupsField: "user_groups", operations: [read]}
  ]){
    id: ID!
...
   channel: Channel @hasOne
   admin_groups: [String!]
+  user_groups: [String!]
}

type Channel @model @auth(rules: [
  {allow: private, provider: iam}
]) {
  id: ID!
...
  assetID: ID @index(name: "byChannelAsset", queryField: "getChannelByAsset")
  asset: Asset @belongsTo(fields: ["assetID"])
}

Pushing this up resulted in my Cloudformation stack getting stuck in the UPDATE_ROLLBACK_FAILED state, with various nested stacks in the same state, or some variation of UPDATE_ROLLBACK....

The offending stack appears to be my ConnectionStack which failed with the status "The following resource(s) failed to update: [EFORassetResolver, ChannelassetResolver, DORItemassetResolver, DeviceEngineCycleassetResolver].". Digging into this, the ChannelassetResolver failed with a not found error:

Resource  handler returned message: "No resolver found. (Service: AppSync, Status  Code: 404, Request ID: f0030f5b-3c5c-4fac-9d74-d644bba0a4cc)"  (RequestToken: 149f909d-3ce5-9070-811b-953c45559b91, HandlerErrorCode:  NotFound)

I've previously run into issues with the Cloudformation stacks, and the advice those times has usually been along the lines of, manually rollback the root resources, while skipping the problematic stack, and then fix that stack somehow because the rest of the Cloudformation stack expects it to be a certain way when it is not. I'm not prepared to go through that currently, and would instead like some help understanding first how I got into this mess before trying to make those changes.

Expected behavior

That the Cloudformation stack doesn't getting deadlocked when pushing a change

Reproduction steps

1. Add auth group to existing table
2. Push changes with amplify push -y

Project Identifier

90ee5db58b8b8ec82b9721ec8bcea978

Log output

# Put your logs below this line

Additional information

No response

Before submitting, please confirm:

• I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.
• I have removed any sensitive information from my code snippets and submission.

[alex-griffiths]

Ok, a little more information as I continue to dig into this. Looking at AppSync directly, the resolvers for the byChannelAsset index appear to be missing (which I guess is what Cloudformation is complaining about). Now I've got no idea why that would be the case, I haven't touched anything in AppSync, so can only assume that amplify must do some sort of cleanup/rewrite of resolvers during a push operation.

Currently, my plan of attack is to rollback the cloudformation stack while skipping the problematic resources, manually recreate the Channelasset resolver in AppSync, and then try an amplify push. I'll keep doing a bit more research about this, so any advice would be appreciated. One thing I'd like direction on is if I should do an amplify pull after rolling back the cloudformation stack? In my mind this would mean that I my local copy of the project should then match the remote copy, and that doing a push should ultimately result in something with no changes?

[ykethan]

transferring the issue to API repository for better assistance

[AnilMaktala]

Hey @alex-griffiths, Thanks for raising this. Have you made any other changes to schema? Also, Could you please verify if the ChannelassetResolver exist in your local directory amplify/backend/api/<apiname>/build/api/resolver?

[alex-griffiths]

These were the only changes I made to the schema, and ‘ChannelassetResolver’ does exist in my local directory

…

On Tue, Aug 13, 2024 at 4:42 am, Anil Maktala ***@***.***(mailto:On Tue, Aug 13, 2024 at 4:42 am, Anil Maktala <<a href=)> wrote: Hey ***@***.***(https://github.com/alex-griffiths), Thanks for raising this. Have you made any other changes to schema? Also, Could you please verify if the ChannelassetResolver exist in your local directory amplify/backend/api/<apiname>/build/api/resolver? — Reply to this email directly, [view it on GitHub](#2756 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AEWUL7YCXK5JKMVKHBQ4CUDZRD627AVCNFSM6AAAAABMIZUDWCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOBUGY4DENZTHA). You are receiving this because you were mentioned.Message ID: ***@***.***>