Amplify Pull Request creates: Failed to pull the backend.

[phreitom]

How did you install the Amplify CLI?

npm

If applicable, what version of Node.js are you using?

18.18.0

Amplify CLI Version

12.7.1

What operating system are you using?

Windows

Did you make any manual changes to the cloud resources managed by Amplify? Please describe the changes made.

I have been going along nice and wonderfully using Amplify. This weekend, I decided to do something awesome and push a Mutation for a function that was generated outside of Amplify. It was a brilliant idea in my head and how much damage could it do if it did not work?

It did not work. It did a lot of damage.

I finally got my changes backed out from CloudFormation and Cloudformation was in a UPDATE SUCCESSFULL state.

1. I created a Appsync function to mimc the function that Cloudformation was saying was missing.
2. I deleted an created a new API key (because the current API key had expired during this endeavor AND it was causing more problems...fortunately, I was able to delete the API key and create a much bigger problem).

Describe the bug

I have been going along nice and wonderfully using Amplify. This weekend, I decided to do something awesome and push a Mutation for a function that was generated outside of Amplify. It was a brilliant idea in my head and how much damage could it do if it did not work?

It did not work. It did a lot of damage.

I finally got my changes backed out from CloudFormation and Cloudformation was in a UPDATE SUCCESSFULL state.

Now...when I commit changes to github to kick of a new build, the build fails with:
🛑 Failed to pull the backend.
!!! Build Failed
!!! Non-Zero Exit Code detected.

...You can imagine my surprise. The coder was so out of his mind about the seriousness of this error that he put THREE exclamation marks BEFORE the message.

When I try to pull the backend through Amplify Pull on my laptop, I get:

🛑 Failed to pull the backend.

I think the coder had calmed down that they did not put any exclamation marks.

I know its Friday so everyone is fried and ready for the weekend. Unfortunately, I have lost a week on this issue and am trying to get caught up this weekend.

Any and all help is greatly appreciated. :))))

Expected behavior

I expected my Amplify pull to pull down the environment. And I expected the build to work in Amplify.

Reproduction steps

Commit changes to Github through Visual Code.
Watch build fail on Amplify Console.
Download build logs from Amplify console.
Try to pull enviromnent into a new environment on laptop

Project Identifier

Project Identifier: af45a07d874c083e1a72f541024c8c0c

Log output

2023-11-03T15:51:55.161Z|info : amplify push api  
2023-11-03T15:51:55.304Z|info : @aws-amplify/amplify-cli-core.banner-message/index.ts.fetch banner messages from https://aws-amplify.github.io/amplify-cli/banner-message.json({}
2023-11-03T15:52:09.037Z|info : amplify-provider-awscloudformation.system-config-manager.getProfileConfig(["*******"])
2023-11-03T15:52:09.038Z|info : amplify-provider-awscloudformation.system-config-manager.getProfiledAwsConfig.profileConfig([{"region":"*******"}])
2023-11-03T15:52:09.038Z|info : amplify-provider-awscloudformation.system-config-manager.getProfileCredentials(["*******"])
2023-11-03T15:57:28.038Z|info : amplify pull core  {"appId":"[***]sqn607","envName":"[***]od","yes":false}
2023-11-03T15:57:28.080Z|info : @aws-amplify/amplify-cli-core.banner-message/index.ts.fetch banner messages from https://aws-amplify.github.io/amplify-cli/banner-message.json({}
2023-11-03T16:00:11.105Z|info : amplify-provider-awscloudformation.amplify-service-permission-check.checkAmplifyServiceIAMPermission.amplifyClient.listApps([])
2023-11-03T16:00:11.492Z|info : amplify-provider-awscloudformation.attach-backend.getAmplifyApp.amplifyClient.getApp([{"appId":"[***]sqn607"}])
2023-11-03T16:00:11.896Z|info : amplify-provider-awscloudformation.attach-backend.getBackendEnv.amplifyClient.getBackendEnvironment([{"appId":"[***]sqn607","environmentName":"[***]od"}])
2023-11-03T16:00:12.275Z|info : amplify-provider-awscloudformation.attach-backend.downloadBackend.s3.getObject([{"Key":"[***]rent-[***]ud-[***]d.zip","Bucket":"[***]ify-[***]-[***]od-[***]339-[***]ment"}])
2023-11-03T16:00:20.524Z|error : Failed to pull the backend.
PullBackendFault: Failed to pull the backend.
2023-11-03T16:40:45.214Z|info : amplify version core  
2023-11-03T16:44:54.954Z|info : amplify diagnose core  {"send-report":true,"yes":false}
2023-11-03T16:44:55.005Z|info : @aws-amplify/amplify-cli-core.banner-message/index.ts.fetch banner messages from https://aws-amplify.github.io/amplify-cli/banner-message.json({}

Additional information

I

Before submitting, please confirm:

• I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.
• I have removed any sensitive information from my code snippets and submission.

[ykethan]

Hey @phreitom, thank you for reaching out. Could you add the --debug on the pull command and provide us the output.

[phreitom]

PS C:\rhyteItNew> amplify pull --appId ##### --envName prod --debug
Opening link: https://#####.admin.amplifyapp.com/admin/######/prod/verify/?loginVersion=1
⠹ Confirm login in the browser or manually paste in your CLI login key:
Successfully received Amplify Studio tokens.
Amplify AppID found: ######. Amplify App name is: ####
Backend environment prod found in Amplify Console app: ####
🛑 Failed to pull the backend.

Learn more at: https://docs.amplify.aws/cli/project/troubleshooting/

PullBackendFault: Failed to pull the backend.
at attachBackend (C:\snapshot\amplify-cli\build\node_modules@aws-amplify\cli-internal\lib\attach-backend.js:55:15)
at async Object.run (C:\snapshot\amplify-cli\build\node_modules@aws-amplify\cli-internal\lib\commands\pull.js:64:9)
at async Object.executeAmplifyCommand (C:\snapshot\amplify-cli\build\node_modules@aws-amplify\cli-internal\lib\index.js:194:9)
at async executePluginModuleCommand (C:\snapshot\amplify-cli\build\node_modules@aws-amplify\cli-internal\lib\execution-manager.js:139:5)
at async executeCommand (C:\snapshot\amplify-cli\build\node_modules@aws-amplify\cli-internal\lib\execution-manager.js:37:9)
at async Object.run (C:\snapshot\amplify-cli\build\node_modules@aws-amplify\cli-internal\lib\index.js:121:5)

File at path: 'C:####\amplify#current-cloud-backend\amplify-meta.json' does not exist
Error: File at path: 'C:####\amplify#current-cloud-backend\amplify-meta.json' does not exist
at JSONUtilities.readJson (C:\snapshot\amplify-cli\build\node_modules@aws-amplify\amplify-cli-core\lib\jsonUtilities.js:44:19)
at AmplifyToolkit.readJsonFile (C:\snapshot\amplify-cli\build\node_modules@aws-amplify\cli-internal\lib\extensions\amplify-helpers\read-json-file.js:6:45)
at ensureAmplifyMeta (C:\snapshot\amplify-cli\build\node_modules@aws-amplify\amplify-provider-awscloudformation\lib\attach-backend.js:127:46)
at Object.run (C:\snapshot\amplify-cli\build\node_modules@aws-amplify\amplify-provider-awscloudformation\lib\attach-backend.js:118:36)
at async Object.attachBackend (C:\snapshot\amplify-cli\build\node_modules@aws-amplify\amplify-provider-awscloudformation\lib\index.js:196:3)
at async queryProvider (C:\snapshot\amplify-cli\build\node_modules@aws-amplify\cli-internal\lib\attach-backend-steps\a10-queryProvider.js:36:5)
at async attachBackend (C:\snapshot\amplify-cli\build\node_modules@aws-amplify\cli-internal\lib\attach-backend.js:43:9)
at async Object.run (C:\snapshot\amplify-cli\build\node_modules@aws-amplify\cli-internal\lib\commands\pull.js:64:9)
at async Object.executeAmplifyCommand (C:\snapshot\amplify-cli\build\node_modules@aws-amplify\cli-internal\lib\index.js:194:9)
at async executePluginModuleCommand (C:\snapshot\amplify-cli\build\node_modules@aws-amplify\cli-internal\lib\execution-manager.js:139:5)
at async executeCommand (C:\snapshot\amplify-cli\build\node_modules@aws-amplify\cli-internal\lib\execution-manager.js:37:9)
at async Object.run (C:\snapshot\amplify-cli\build\node_modules@aws-amplify\cli-internal\lib\index.js:121:5)

Session Identifier: d33b5d55-c253-401e-aaba-279a3e9509f8

[phreitom]

If I try to do an amplify api push I get:

PS C:#####> amplify api push -y -f --debug
🛑 Packaging overrides failed.
InvalidOverrideError: Packaging overrides failed.
at C:\snapshot\amplify-cli\build\node_modules@aws-amplify\amplify-category-api\lib\index.js:309:23
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async transformCategoryStack (C:\snapshot\amplify-cli\build\node_modules@aws-amplify\amplify-category-api\lib\index.js:308:29)
at async transformResourceWithOverrides (C:\snapshot\amplify-cli\build\node_modules@aws-amplify\amplify-provider-awscloudformation\lib\override-manager\transform-resource.js:66:9)
at async Object.buildOverrides (C:\snapshot\amplify-cli\build\node_modules@aws-amplify\amplify-provider-awscloudformation\lib\utility-functions.js:88:7)
at async AmplifyToolkit.pushResources (C:\snapshot\amplify-cli\build\node_modules@aws-amplify\cli-internal\lib\extensions\amplify-helpers\push-resources.js:77:5)
at async Object.executeAmplifyCommand (C:\snapshot\amplify-cli\build\node_modules@aws-amplify\amplify-category-api\lib\index.js:231:9)
at async executePluginModuleCommand (C:\snapshot\amplify-cli\build\node_modules@aws-amplify\cli-internal\lib\execution-manager.js:139:5)
at async executeCommand (C:\snapshot\amplify-cli\build\node_modules@aws-amplify\cli-internal\lib\execution-manager.js:37:9)
at async Object.run (C:\snapshot\amplify-cli\build\node_modules@aws-amplify\cli-internal\lib\index.js:121:5)

[ykethan]

Hey @phreitom, thank you for the information.
The error: File at path: 'C:####\amplify#current-cloud-backend\amplify-meta.json' does not exist does stand here.
This appears to be similar to #12766

The zip in the S3 deployment bucket, seems to be missing the amplify-meta.json file in the #current-cloud-backend.
Could you check the AWS S3 console for the deployment bucket, the name should be similar to amplify-<app-name>-<env-name>-34648-deployment.
Download and open the #current-cloud-backend.zip, verify if the amplify-meta.json does exist?

[phreitom]

Yup it is there. I downloaded it from s3/buckets/amplify-####-prod-######-deployment
#current-cloud-backend.zip

[phreitom]

Could it be that #current-cloud_backend.zip has an extra folder in at the top CurrentCloudBackEnd. So amplify-meta.json is at a level deeper than expected?

Ignore me if I have no idea what I am talking about :))) Bored and trying to help :))

[ykethan]

@phreitom just a quick check, noticed amplify#current-cloud-backend doesn't not have \ in the path. Was this mistakenly modified when removing the AppName?
additionally, in Amplify hosting could you add a environment variable in the App settings AMPLIFY_ENABLE_DEBUG_OUTPUT as true and re-build the app. Could you provide us the output from the backend build?

[phreitom]

just a quick check, noticed amplify#current-cloud-backend doesn't not have \ in the path. Was this mistakenly modified when removing the AppName?

Yeah I noticed that too.

I accidentally removed it while copying I guess. Because when I reran the pull, the proper folder structure was there.

File at path: 'C:#####\amplify#current-cloud-backend\amplify-meta.json' does not exist
Error: File at path: 'C:######\amplify#current-cloud-backend\amplify-meta.json' does not exist

Working on adding environment variable. Will get you output soon :)

[phreitom]

Here is the output with AMPLIFY_ENABLE_DEBUG_OUTPUT as true:

AmplifyBuildLogs.txt

[phreitom]

I am still here :)) Looked like github issues down for a bit. :))

[phreitom]

Hi Gang :) Any update?

[ykethan]

Hey @phreitom, Thank you for the information.
Are you running the pull in a new empty directory?

Could it be that #current-cloud_backend.zip has an extra folder in at the top CurrentCloudBackEnd. So amplify-meta.json is at a level deeper than expected?

looking at the screenshot it appears the zip has a nested folder
we may have to rezip the current-cloud-backend and then pull.
Would you be open for a quick call, i am available on discord at handle ykethan

[phreitom]

Just sent you a message on Discord :)

[ykethan]

Hey @phreitom, thank you for hopping on the call, it was amazing chatting with you.
To summarize the issue:

we noticed that the #current-cloud-backend.zip had a nested current-cloud-backend folder. On re-zipping and uploading the zip; we were able to successfully run a amplify pull.
when running a push with a small change to the API we encountered an expired API key, on utilizing the mitigation steps in aws-amplify/amplify-category-api#636 (comment) we were able to regenerate the API key.
Additionally, we encountered StreamsArn missing error, on enabling the streams with New and Old Images as section we were able to mitigate the error and successfully push.

Closing the issue, please feel free in reaching out to us again.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[phreitom]

Thanks Ykethan for all your help!!! It was very appreciated!