indexer-agent: Error Tx reverted "Failed to redeem RAV", but actually didn't fail

[aasseman]

I'm not sure if it's specific to TAP RAV redeeming, or just a quirk of how the Txs on arbitrum-one are handled, but I just encountered this error (tl;dr reverted because of gas estimate failure?):

{
  "level": 50,
  "time": 1722050881279,
  "pid": 1,
  "hostname": "graph-network-indexer-agent-0",
  "name": "IndexerAgent",
  "component": "AllocationReceiptCollector",
  "indexer": "0x6f8a032B4b1Ee622EF2F0fC091bdbB98CFAE81A3",
  "protocolNetwork": "eip155:42161",
  "operator": "0x2963E4767f5fa21F637bB11FaFA619fEaD21f4d1",
  "function": "submitRAVs()",
  "ravsToSubmit": 1,
  "err": {
    "type": "IndexerError",
    "message": "Failed to redeem query fee voucher",
    "stack": "IndexerError: Failed to redeem query fee voucher\n    at indexerError (/opt/indexer/packages/indexer-common/dist/errors.js:175:12)\n    at AllocationReceiptCollector.submitRAVs (/opt/indexer/packages/indexer-common/dist/allocations/query-fees.js:633:47)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async /opt/indexer/packages/indexer-common/dist/allocations/query-fees.js:260:43",
    "code": "IE055",
    "explanation": "https://github.com/graphprotocol/indexer/blob/main/docs/errors.md#ie055",
    "cause": {
      "type": "Error",
      "message": "cannot estimate gas; transaction may fail or may require manual gas limit [ See: https://links.ethers.org/v5-errors-UNPREDICTABLE_GAS_LIMIT ] (reason=\"execution reverted\", method=\"estimateGas\", transaction={\"from\":\"0x2963E4767f5fa21F637bB11FaFA619fEaD21f4d1\",\"to\":\"0x8f477709eF277d4A880801D01A140a9CF88bA0d3\",\"data\":\"0x456b431600000000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000000140000000000000000000000000398451e73d19783127c95bad9c6ca5b17e165e2900000000000000000000000000000000000000000000000017e5e94294c7c26d000000000000000000000000000000000000000000000000f9cc8c0c7d18d6c300000000000000000000000000000000000000000000000000000000000000800000000000000000000000000000000000000000000000000000000000000041846bb93ce4272c5c082f61d51a0f4854355a004f76fe16341f7429c29da8324d7c69d8d25685e62f401bff0f47e914df19d699c533e2197c4d4d72ebfb960d831c000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000041e6befa882f054dd82c3f930389707cccf32b45f878b5d51a06b2cca549e3b0072087d2864c23d95852ae5ae727b014e51dbd341f00d8982fb2d577acc1ed92101c00000000000000000000000000000000000000000000000000000000000000\",\"accessList\":null}, error={\"reason\":\"processing response error\",\"code\":\"SERVER_ERROR\",\"body\":\"{\\\"jsonrpc\\\":\\\"2.0\\\",\\\"id\\\":325549,\\\"error\\\":{\\\"code\\\":3,\\\"message\\\":\\\"execution reverted\\\",\\\"data\\\":\\\"0x4f5f0b4f000000000000000000000000dde4cffd3d9052a9cb618fc05a1cd02be1f2f467000000000000000000000000398451e73d19783127c95bad9c6ca5b17e165e29\\\"}}\\n\",\"error\":{\"code\":3,\"data\":\"0x4f5f0b4f000000000000000000000000dde4cffd3d9052a9cb618fc05a1cd02be1f2f467000000000000000000000000398451e73d19783127c95bad9c6ca5b17e165e29\"},\"requestBody\":\"{\\\"method\\\":\\\"eth_estimateGas\\\",\\\"params\\\":[{\\\"from\\\":\\\"0x2963e4767f5fa21f637bb11fafa619fead21f4d1\\\",\\\"to\\\":\\\"0x8f477709ef277d4a880801d01a140a9cf88ba0d3\\\",\\\"data\\\":\\\"0x456b431600000000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000000140000000000000000000000000398451e73d19783127c95bad9c6ca5b17e165e2900000000000000000000000000000000000000000000000017e5e94294c7c26d000000000000000000000000000000000000000000000000f9cc8c0c7d18d6c300000000000000000000000000000000000000000000000000000000000000800000000000000000000000000000000000000000000000000000000000000041846bb93ce4272c5c082f61d51a0f4854355a004f76fe16341f7429c29da8324d7c69d8d25685e62f401bff0f47e914df19d699c533e2197c4d4d72ebfb960d831c000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000041e6befa882f054dd82c3f930389707cccf32b45f878b5d51a06b2cca549e3b0072087d2864c23d95852ae5ae727b014e51dbd341f00d8982fb2d577acc1ed92101c00000000000000000000000000000000000000000000000000000000000000\\\"}],\\\"id\\\":325549,\\\"jsonrpc\\\":\\\"2.0\\\"}\",\"requestMethod\":\"POST\",\"url\":\"https://arbitrum-mainnet.core.chainstack.com/1eb3810c179be2fe3a7f1713f4d6255f\"}, code=UNPREDICTABLE_GAS_LIMIT, version=providers/5.7.0)",
      "stack": "Error: cannot estimate gas; transaction may fail or may require manual gas limit [ See: https://links.ethers.org/v5-errors-UNPREDICTABLE_GAS_LIMIT ] (reason=\"execution reverted\", method=\"estimateGas\", transaction={\"from\":\"0x2963E4767f5fa21F637bB11FaFA619fEaD21f4d1\",\"to\":\"0x8f477709eF277d4A880801D01A140a9CF88bA0d3\",\"data\":\"0x456b431600000000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000000140000000000000000000000000398451e73d19783127c95bad9c6ca5b17e165e2900000000000000000000000000000000000000000000000017e5e94294c7c26d000000000000000000000000000000000000000000000000f9cc8c0c7d18d6c300000000000000000000000000000000000000000000000000000000000000800000000000000000000000000000000000000000000000000000000000000041846bb93ce4272c5c082f61d51a0f4854355a004f76fe16341f7429c29da8324d7c69d8d25685e62f401bff0f47e914df19d699c533e2197c4d4d72ebfb960d831c000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000041e6befa882f054dd82c3f930389707cccf32b45f878b5d51a06b2cca549e3b0072087d2864c23d95852ae5ae727b014e51dbd341f00d8982fb2d577acc1ed92101c00000000000000000000000000000000000000000000000000000000000000\",\"accessList\":null}, error={\"reason\":\"processing response error\",\"code\":\"SERVER_ERROR\",\"body\":\"{\\\"jsonrpc\\\":\\\"2.0\\\",\\\"id\\\":325549,\\\"error\\\":{\\\"code\\\":3,\\\"message\\\":\\\"execution reverted\\\",\\\"data\\\":\\\"0x4f5f0b4f000000000000000000000000dde4cffd3d9052a9cb618fc05a1cd02be1f2f467000000000000000000000000398451e73d19783127c95bad9c6ca5b17e165e29\\\"}}\\n\",\"error\":{\"code\":3,\"data\":\"0x4f5f0b4f000000000000000000000000dde4cffd3d9052a9cb618fc05a1cd02be1f2f467000000000000000000000000398451e73d19783127c95bad9c6ca5b17e165e29\"},\"requestBody\":\"{\\\"method\\\":\\\"eth_estimateGas\\\",\\\"params\\\":[{\\\"from\\\":\\\"0x2963e4767f5fa21f637bb11fafa619fead21f4d1\\\",\\\"to\\\":\\\"0x8f477709ef277d4a880801d01a140a9cf88ba0d3\\\",\\\"data\\\":\\\"0x456b431600000000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000000140000000000000000000000000398451e73d19783127c95bad9c6ca5b17e165e2900000000000000000000000000000000000000000000000017e5e94294c7c26d000000000000000000000000000000000000000000000000f9cc8c0c7d18d6c300000000000000000000000000000000000000000000000000000000000000800000000000000000000000000000000000000000000000000000000000000041846bb93ce4272c5c082f61d51a0f4854355a004f76fe16341f7429c29da8324d7c69d8d25685e62f401bff0f47e914df19d699c533e2197c4d4d72ebfb960d831c000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000041e6befa882f054dd82c3f930389707cccf32b45f878b5d51a06b2cca549e3b0072087d2864c23d95852ae5ae727b014e51dbd341f00d8982fb2d577acc1ed92101c00000000000000000000000000000000000000000000000000000000000000\\\"}],\\\"id\\\":325549,\\\"jsonrpc\\\":\\\"2.0\\\"}\",\"requestMethod\":\"POST\",\"url\":\"https://arbitrum-mainnet.core.chainstack.com/1eb3810c179be2fe3a7f1713f4d6255f\"}, code=UNPREDICTABLE_GAS_LIMIT, version=providers/5.7.0)\n    at Logger.makeError (/opt/indexer/node_modules/@ethersproject/logger/lib/index.js:238:21)\n    at Logger.throwError (/opt/indexer/node_modules/@ethersproject/logger/lib/index.js:247:20)\n    at checkError (/opt/indexer/node_modules/@ethersproject/providers/lib/json-rpc-provider.js:122:20)\n    at StaticJsonRpcProvider.<anonymous> (/opt/indexer/node_modules/@ethersproject/providers/lib/json-rpc-provider.js:751:47)\n    at step (/opt/indexer/node_modules/@ethersproject/providers/lib/json-rpc-provider.js:48:23)\n    at Object.throw (/opt/indexer/node_modules/@ethersproject/providers/lib/json-rpc-provider.js:29:53)\n    at rejected (/opt/indexer/node_modules/@ethersproject/providers/lib/json-rpc-provider.js:21:65)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)",
      "reason": "execution reverted",
      "code": "UNPREDICTABLE_GAS_LIMIT",
      "method": "estimateGas",
      "transaction": {
        "from": "0x2963E4767f5fa21F637bB11FaFA619fEaD21f4d1",
        "to": "0x8f477709eF277d4A880801D01A140a9CF88bA0d3",
        "data": "0x456b431600000000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000000140000000000000000000000000398451e73d19783127c95bad9c6ca5b17e165e2900000000000000000000000000000000000000000000000017e5e94294c7c26d000000000000000000000000000000000000000000000000f9cc8c0c7d18d6c300000000000000000000000000000000000000000000000000000000000000800000000000000000000000000000000000000000000000000000000000000041846bb93ce4272c5c082f61d51a0f4854355a004f76fe16341f7429c29da8324d7c69d8d25685e62f401bff0f47e914df19d699c533e2197c4d4d72ebfb960d831c000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000041e6befa882f054dd82c3f930389707cccf32b45f878b5d51a06b2cca549e3b0072087d2864c23d95852ae5ae727b014e51dbd341f00d8982fb2d577acc1ed92101c00000000000000000000000000000000000000000000000000000000000000",
        "accessList": null
      },
      "error": {
        "type": "Error",
        "message": "processing response error (body=\"{\\\"jsonrpc\\\":\\\"2.0\\\",\\\"id\\\":325549,\\\"error\\\":{\\\"code\\\":3,\\\"message\\\":\\\"execution reverted\\\",\\\"data\\\":\\\"0x4f5f0b4f000000000000000000000000dde4cffd3d9052a9cb618fc05a1cd02be1f2f467000000000000000000000000398451e73d19783127c95bad9c6ca5b17e165e29\\\"}}\\n\", error={\"code\":3,\"data\":\"0x4f5f0b4f000000000000000000000000dde4cffd3d9052a9cb618fc05a1cd02be1f2f467000000000000000000000000398451e73d19783127c95bad9c6ca5b17e165e29\"}, requestBody=\"{\\\"method\\\":\\\"eth_estimateGas\\\",\\\"params\\\":[{\\\"from\\\":\\\"0x2963e4767f5fa21f637bb11fafa619fead21f4d1\\\",\\\"to\\\":\\\"0x8f477709ef277d4a880801d01a140a9cf88ba0d3\\\",\\\"data\\\":\\\"0x456b431600000000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000000140000000000000000000000000398451e73d19783127c95bad9c6ca5b17e165e2900000000000000000000000000000000000000000000000017e5e94294c7c26d000000000000000000000000000000000000000000000000f9cc8c0c7d18d6c300000000000000000000000000000000000000000000000000000000000000800000000000000000000000000000000000000000000000000000000000000041846bb93ce4272c5c082f61d51a0f4854355a004f76fe16341f7429c29da8324d7c69d8d25685e62f401bff0f47e914df19d699c533e2197c4d4d72ebfb960d831c000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000041e6befa882f054dd82c3f930389707cccf32b45f878b5d51a06b2cca549e3b0072087d2864c23d95852ae5ae727b014e51dbd341f00d8982fb2d577acc1ed92101c00000000000000000000000000000000000000000000000000000000000000\\\"}],\\\"id\\\":325549,\\\"jsonrpc\\\":\\\"2.0\\\"}\", requestMethod=\"POST\", url=\"https://arbitrum-mainnet.core.chainstack.com/1eb3810c179be2fe3a7f1713f4d6255f\", code=SERVER_ERROR, version=web/5.7.1)",
        "stack": "Error: processing response error (body=\"{\\\"jsonrpc\\\":\\\"2.0\\\",\\\"id\\\":325549,\\\"error\\\":{\\\"code\\\":3,\\\"message\\\":\\\"execution reverted\\\",\\\"data\\\":\\\"0x4f5f0b4f000000000000000000000000dde4cffd3d9052a9cb618fc05a1cd02be1f2f467000000000000000000000000398451e73d19783127c95bad9c6ca5b17e165e29\\\"}}\\n\", error={\"code\":3,\"data\":\"0x4f5f0b4f000000000000000000000000dde4cffd3d9052a9cb618fc05a1cd02be1f2f467000000000000000000000000398451e73d19783127c95bad9c6ca5b17e165e29\"}, requestBody=\"{\\\"method\\\":\\\"eth_estimateGas\\\",\\\"params\\\":[{\\\"from\\\":\\\"0x2963e4767f5fa21f637bb11fafa619fead21f4d1\\\",\\\"to\\\":\\\"0x8f477709ef277d4a880801d01a140a9cf88ba0d3\\\",\\\"data\\\":\\\"0x456b431600000000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000000140000000000000000000000000398451e73d19783127c95bad9c6ca5b17e165e2900000000000000000000000000000000000000000000000017e5e94294c7c26d000000000000000000000000000000000000000000000000f9cc8c0c7d18d6c300000000000000000000000000000000000000000000000000000000000000800000000000000000000000000000000000000000000000000000000000000041846bb93ce4272c5c082f61d51a0f4854355a004f76fe16341f7429c29da8324d7c69d8d25685e62f401bff0f47e914df19d699c533e2197c4d4d72ebfb960d831c000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000041e6befa882f054dd82c3f930389707cccf32b45f878b5d51a06b2cca549e3b0072087d2864c23d95852ae5ae727b014e51dbd341f00d8982fb2d577acc1ed92101c00000000000000000000000000000000000000000000000000000000000000\\\"}],\\\"id\\\":325549,\\\"jsonrpc\\\":\\\"2.0\\\"}\", requestMethod=\"POST\", url=\"https://arbitrum-mainnet.core.chainstack.com/1eb3810c179be2fe3a7f1713f4d6255f\", code=SERVER_ERROR, version=web/5.7.1)\n    at Logger.makeError (/opt/indexer/node_modules/@ethersproject/logger/lib/index.js:238:21)\n    at Logger.throwError (/opt/indexer/node_modules/@ethersproject/logger/lib/index.js:247:20)\n    at /opt/indexer/node_modules/@ethersproject/web/lib/index.js:313:32\n    at step (/opt/indexer/node_modules/@ethersproject/web/lib/index.js:33:23)\n    at Object.next (/opt/indexer/node_modules/@ethersproject/web/lib/index.js:14:53)\n    at fulfilled (/opt/indexer/node_modules/@ethersproject/web/lib/index.js:5:58)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)",
        "reason": "processing response error",
        "code": "SERVER_ERROR",
        "body": "{\"jsonrpc\":\"2.0\",\"id\":325549,\"error\":{\"code\":3,\"message\":\"execution reverted\",\"data\":\"0x4f5f0b4f000000000000000000000000dde4cffd3d9052a9cb618fc05a1cd02be1f2f467000000000000000000000000398451e73d19783127c95bad9c6ca5b17e165e29\"}}\n",
        "error": {
          "type": "Error",
          "message": "execution reverted",
          "stack": "Error: execution reverted\n    at getResult (/opt/indexer/node_modules/@ethersproject/providers/lib/json-rpc-provider.js:191:21)\n    at processJsonFunc (/opt/indexer/node_modules/@ethersproject/web/lib/index.js:356:22)\n    at /opt/indexer/node_modules/@ethersproject/web/lib/index.js:288:46\n    at step (/opt/indexer/node_modules/@ethersproject/web/lib/index.js:33:23)\n    at Object.next (/opt/indexer/node_modules/@ethersproject/web/lib/index.js:14:53)\n    at fulfilled (/opt/indexer/node_modules/@ethersproject/web/lib/index.js:5:58)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)",
          "code": 3,
          "data": "0x4f5f0b4f000000000000000000000000dde4cffd3d9052a9cb618fc05a1cd02be1f2f467000000000000000000000000398451e73d19783127c95bad9c6ca5b17e165e29"
        },
        "requestBody": "{\"method\":\"eth_estimateGas\",\"params\":[{\"from\":\"0x2963e4767f5fa21f637bb11fafa619fead21f4d1\",\"to\":\"0x8f477709ef277d4a880801d01a140a9cf88ba0d3\",\"data\":\"0x456b431600000000000000000000000000000000000000000000000000000000000000400000000000000000000000000000000000000000000000000000000000000140000000000000000000000000398451e73d19783127c95bad9c6ca5b17e165e2900000000000000000000000000000000000000000000000017e5e94294c7c26d000000000000000000000000000000000000000000000000f9cc8c0c7d18d6c300000000000000000000000000000000000000000000000000000000000000800000000000000000000000000000000000000000000000000000000000000041846bb93ce4272c5c082f61d51a0f4854355a004f76fe16341f7429c29da8324d7c69d8d25685e62f401bff0f47e914df19d699c533e2197c4d4d72ebfb960d831c000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000041e6befa882f054dd82c3f930389707cccf32b45f878b5d51a06b2cca549e3b0072087d2864c23d95852ae5ae727b014e51dbd341f00d8982fb2d577acc1ed92101c00000000000000000000000000000000000000000000000000000000000000\"}],\"id\":325549,\"jsonrpc\":\"2.0\"}",
        "requestMethod": "POST",
        "url": "https://arbitrum-mainnet.core.chainstack.com/1eb3810c179be2fe3a7f1713f4d6255f"
      }
    }
  },
  "msg": "Failed to redeem RAV"
}

But it actually didn't fail at all:
https://dashboard.tenderly.co/tx/arbitrum/0x49223a70f01eee0d3b41b8c7fe9b57d348469222ad65208656ee30fd4b63c139

[aasseman]

Also the indexer-agent never-marked the RAV as final, even though it's been ~3 days now:

indexer=# select * from scalar_tap_ravs;
-[ RECORD 1 ]---+-------------------------------------------------------------------------------------------------------------------------------------
allocation_id   | 398451e73d19783127c95bad9c6ca5b17e165e29
sender_address  | dde4cffd3d9052a9cb618fc05a1cd02be1f2f467
signature       | \x846bb93ce4272c5c082f61d51a0f4854355a004f76fe16341f7429c29da8324d7c69d8d25685e62f401bff0f47e914df19d699c533e2197c4d4d72ebfb960d831c
timestamp_ns    | 1722038904712315501
value_aggregate | 17999915795959305923
final           | f
last            | t
redeemed_at     | 
created_at      | 2024-07-26 18:57:39.445949+00
updated_at      | 2024-07-27 03:14:17.694499+00