[test-recorder] can not run test case if migrate test-recorder from v3 to v4

[kazrael2119]

We have a rp "StandByPool" need to GA with typespec, but in previous release, it is generated from swagger which means in the past, we used test-recorder and test credential with v3+v1, but now , they should upgrade to v4+v2.

Then I met an issue when running "export TEST_MODE=record && rushx test":

[vitest]  RUN  v2.0.5 D:/Git/azure-sdk-for-js/sdk/standbypool/arm-standbypool
[vitest]
[vitest]  ❯ test/public/standbypool_operations_test.spec.ts  (1 test | 1 failed) 313ms
[vitest]    × StandbyPool test > operations list test
[vitest]      → Start request failed.
[vitest]      → Cannot read properties of undefined (reading 'stop')
[vitest]
[vitest] ⎯⎯⎯⎯⎯⎯⎯ Failed Tests 1 ⎯⎯⎯⎯⎯⎯⎯
[vitest]
[vitest]  FAIL  test/public/standbypool_operations_test.spec.ts > StandbyPool test > operations list test
[vitest] RecorderError: Start request failed.
[vitest]  ❯ Recorder.start ../../test-utils/recorder/src/recorder.ts:336:19
[vitest]  ❯ Module.createRecorder test/public/utils/recordedClient.ts:25:3
[vitest]      23| export async function createRecorder(context: VitestTestContext): Prom...
[vitest]      24|   const recorder = new Recorder(context);
[vitest]      25|   await recorder.start(recorderEnvSetup);
[vitest]        |   ^
[vitest]      26|   return recorder;
[vitest]      27| }
[vitest]  ❯ test/public/standbypool_operations_test.spec.ts:26:16
[vitest]
[vitest] ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[1/2]⎯
[vitest]
[vitest]  FAIL  test/public/standbypool_operations_test.spec.ts > StandbyPool test > operations list test
[vitest] TypeError: Cannot read properties of undefined (reading 'stop')
[vitest]  ❯ test/public/standbypool_operations_test.spec.ts:38:20
[vitest]      36|
[vitest]      37|   afterEach(async function () {
[vitest]      38|     await recorder.stop();
[vitest]        |                    ^
[vitest]      39|   });
[vitest]      40|
[vitest]
[vitest] ⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯⎯[2/2]⎯
[vitest]
[vitest]  Test Files  1 failed (1)
[vitest]       Tests  1 failed (1)
[vitest]    Start at  10:40:28
[vitest]    Duration  3.57s (transform 850ms, setup 0ms, collect 2.68s, tests 313ms, environment 0ms, prepare 236ms)

don't know if this is a bug if migrate the version from v3 to v4. for v3 we use context from mocha, but for v4, we use vitestcontext from vitest

here is the test diff in my pr :

the deleted file is the test cases using test-recorder v3
and these two new files are the test cases using test-recorder-v4

Steps to repro:
1: checkout to #31087
2: clean up the .asset lib if standbypool recordings existed
3: revert the assets.json change from sdk/standbypool/arm-standbypool as the old tag was recorded with v3+v1
4: run rush update && rush build -t @azure/arm-standbypool and cd sdk/standbypool/arm-standbypool
5: run export TEST_MODE=record && rushx test

Current solution:
1 delete assets.json
2 run npx dev-tool test-proxy init
3 re-run test cases with record mode
4 run npx dev-tool test-proxy push

[jeremymeng]

@HarshaNalluru is it expected that we need to re-record after migrating?

[HarshaNalluru]

Looking at the recordings in the assets repo before and after from the PR #31087. This doesn't seem like a problem with the recorder.

• old tag - js/standbypool/arm-standbypool_a3c6f11959
• new tag - js/standbypool/arm-standbypool_ebb9a2d294

https://github.com/Azure/azure-sdk-assets/blob/ebb9a2d2944a93e40261368dabfdc11029ee05bb/js/sdk/standbypool/arm-standbypool/recordings/node/standbypool_test/recording_operations_list_test.json#L36

New recording has a 404 error from the service.

  "ResponseBody": {
    "error": {
      "code": "InvalidResourceType",
      "message": "The resource type 'Operations' could not be found in the namespace 'Microsoft.StandbyPool' for api version '2024-03-01'. The supported api-versions are '2023-06-01-preview,2023-12-01-preview,2024-03-01-preview'."
    }
  }

Essentially, the live test itself has changed the requests, with the new api-version. This should reproduce as a live test too given service may not have deployed the version fully (?).

[kazrael2119]

Looking at the recordings in the assets repo before and after from the PR #31087. This doesn't seem like a problem with the recorder.

• old tag - js/standbypool/arm-standbypool_a3c6f11959
• new tag - js/standbypool/arm-standbypool_ebb9a2d294

https://github.com/Azure/azure-sdk-assets/blob/ebb9a2d2944a93e40261368dabfdc11029ee05bb/js/sdk/standbypool/arm-standbypool/recordings/node/standbypool_test/recording_operations_list_test.json#L36

New recording has a 404 error from the service.

  "ResponseBody": {
    "error": {
      "code": "InvalidResourceType",
      "message": "The resource type 'Operations' could not be found in the namespace 'Microsoft.StandbyPool' for api version '2024-03-01'. The supported api-versions are '2023-06-01-preview,2023-12-01-preview,2024-03-01-preview'."
    }
  }

Essentially, the live test itself has changed the requests, with the new api-version. This should reproduce as a live test too given service may not have deployed the version fully (?).

@HarshaNalluru , yes, the new api version has not been deployed. Sorry, I posted the wrong log, now I have updated the error logs in the description as the correct one, could you help check again? thanks