[BUG][INTEGTEST][2.0.1] Failed on without security integTest for Index Management

[zelinh]

Error context below.

2022-06-13 21:33:42 INFO     Running integration tests for index-management
2022-06-13 21:33:42 INFO     ===============================================
2022-06-13 21:33:42 INFO     Executing "/var/jenkins/workspace/integ-test/scripts/default/integtest.sh -b localhost -p 9200 -s true -v 2.0.1" in /tmp/tmpfygkshul/index-management
2022-06-13 21:57:58 INFO     Recording component test results for index-management at /var/jenkins/workspace/integ-test/test-results/1/integ-test/index-management/with-security/test-results
2022-06-13 21:57:58 INFO     Integration test run failed for component index-management
2022-06-13 21:57:58 INFO     OpenJDK 64-Bit Server VM warning: Ignoring option --illegal-access=warn; support was removed in 17.0

2022-06-13 21:57:58 INFO     Sending SIGKILL to PID 537
2022-06-13 21:57:58 INFO     Process killed with exit code None
2022-06-13 21:57:58 INFO     Cleanup /tmp/tmpfygkshul/local-test-cluster/* content after the test
2022-06-13 21:57:59 INFO     Recording local cluster logs for index-management with test configuration as with-security at /var/jenkins/workspace/integ-test/test-results/1/integ-test/index-management/with-security/local-cluster-logs
2022-06-13 21:57:59 INFO     Additional config found: {'path.repo': ['/tmp']}
2022-06-13 21:57:59 INFO     opensearch distribution: tar
2022-06-13 21:57:59 INFO     Downloading bundle artifact
2022-06-13 21:57:59 INFO     Downloading https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.0.1/3895/linux/x64/tar/dist/opensearch/opensearch-2.0.1-linux-x64.tar.gz into /tmp/tmpfygkshul/local-test-cluster/opensearch-2.0.1-linux-x64.tar.gz ...
2022-06-13 21:58:01 INFO     Downloaded bundle to /tmp/tmpfygkshul/local-test-cluster/opensearch-2.0.1-linux-x64.tar.gz
2022-06-13 21:58:01 INFO     Installing /tmp/tmpfygkshul/local-test-cluster/opensearch-2.0.1-linux-x64.tar.gz in /tmp/tmpfygkshul/local-test-cluster/opensearch-2.0.1
2022-06-13 21:58:05 INFO     Started OpenSearch with parent PID 6495
2022-06-13 21:58:05 INFO     Waiting for service to become available
2022-06-13 21:58:05 INFO     Pinging service attempt 0
2022-06-13 21:58:05 INFO     Pinging http://localhost:9200/_cluster/health
2022-06-13 21:58:05 INFO     Service not available, yet
2022-06-13 21:58:15 INFO     Pinging service attempt 1
2022-06-13 21:58:15 INFO     Pinging http://localhost:9200/_cluster/health
2022-06-13 21:58:15 INFO     Service not available, yet
2022-06-13 21:58:25 INFO     Pinging service attempt 2
2022-06-13 21:58:25 INFO     Pinging http://localhost:9200/_cluster/health
2022-06-13 21:58:25 INFO     200: {"cluster_name":"opensearch","status":"green","timed_out":false,"number_of_nodes":1,"number_of_data_nodes":1,"discovered_master":true,"discovered_cluster_manager":true,"active_primary_shards":0,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":100.0}
2022-06-13 21:58:25 INFO     Service is available
2022-06-13 21:58:25 INFO     ===============================================
2022-06-13 21:58:25 INFO     Running integration tests for index-management
2022-06-13 21:58:25 INFO     ===============================================
2022-06-13 21:58:25 INFO     Executing "/var/jenkins/workspace/integ-test/scripts/default/integtest.sh -b localhost -p 9200 -s false -v 2.0.1" in /tmp/tmpfygkshul/index-management
2022-06-13 22:20:26 INFO     Recording component test results for index-management at /var/jenkins/workspace/integ-test/test-results/1/integ-test/index-management/without-security/test-results
2022-06-13 22:20:26 INFO     Integration test run failed for component index-management
2022-06-13 22:20:26 INFO     OpenJDK 64-Bit Server VM warning: Ignoring option --illegal-access=warn; support was removed in 17.0

REPRODUCE WITH: ./gradlew ':integTest' --tests "org.opensearch.indexmanagement.transform.TransformRunnerIT.test transform with data filter" -Dtests.seed=7EAECE3F4938CDAD -Dtests.security.manager=false -Dtests.locale=en-SG -Dtests.timezone=NZ-CHAT -Druntime.java=17


Suite: Test class org.opensearch.indexmanagement.transform.TransformRunnerIT
  2> REPRODUCE WITH: ./gradlew ':integTest' --tests "org.opensearch.indexmanagement.transform.TransformRunnerIT.test transform with data filter" -Dtests.seed=7EAECE3F4938CDAD -Dtests.security.manager=false -Dtests.locale=en-SG -Dtests.timezone=NZ-CHAT -Druntime.java=17
  2> java.lang.AssertionError: Didn't capture search time
        at __randomizedtesting.SeedInfo.seed([7EAECE3F4938CDAD:5892CED6F8C3F6A1]:0)
        at org.junit.Assert.fail(Assert.java:89)
        at org.junit.Assert.assertTrue(Assert.java:42)
        at org.opensearch.indexmanagement.transform.TransformRunnerIT.test transform with data filter(TransformRunnerIT.kt:108)
  2> NOTE: leaving temporary files on disk at: /tmp/tmpfygkshul/index-management/build/testrun/integTest/temp/org.opensearch.indexmanagement.transform.TransformRunnerIT_7EAECE3F4938CDAD-001
  2> NOTE: test params are: codec=Lucene91, sim=Asserting(RandomSimilarity(queryNorm=false): {}), locale=en-SG, timezone=NZ-CHAT
  2> NOTE: Linux 4.14.243-185.433.amzn2.x86_64 amd64/Eclipse Adoptium 17.0.2 (64-bit)/cpus=16,threads=1,free=499553120,total=536870912
  2> NOTE: All tests run in this JVM: [IndexManagementIndicesIT, ActionRetryIT, ActionTimeoutIT, AllocationActionIT, CloseActionIT, DeleteActionIT, ForceMergeActionIT, IndexPriorityActionIT, IndexStateManagementHistoryIT, OpenActionIT, ReadOnlyActionIT, ReadWriteActionIT, ReplicaCountActionIT, RolloverActionIT, RollupActionIT, ShrinkActionIT, SnapshotActionIT, TransitionActionIT, ManagedIndexCoordinatorIT, MigrationServicesIT, ISMTemplateRestAPIIT, IndexStateManagementRestApiIT, RestAddPolicyActionIT, RestChangePolicyActionIT, RestExplainActionIT, RestRemovePolicyActionIT, RestRetryFailedManagedIndexActionIT, ManagedIndexRunnerIT, RefreshSearchAnalyzerActionIT, RestRefreshSearchAnalyzerActionIT, FieldCapsFilterIT, RollupInterceptorIT, RestDeleteRollupActionIT, RestExplainRollupActionIT, RestGetRollupActionIT, RestIndexRollupActionIT, RestStartRollupActionIT, RestStopRollupActionIT, RollupRunnerIT, TransformRunnerIT]

256 tests completed, 1 failed, 6 skipped

=== Standard output of node `node{::integTest-0}` ===

=== Standard error of node `node{::integTest-0}` ===

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':integTest'.
> There were failing tests. See the report at: file:///tmp/tmpfygkshul/index-management/build/reports/tests/integTest/index.html

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org/

BUILD FAILED in 22m

2022-06-13 22:20:26 INFO     Sending SIGKILL to PID 6495
2022-06-13 22:20:26 INFO     Process killed with exit code None
2022-06-13 22:20:26 INFO     Cleanup /tmp/tmpfygkshul/local-test-cluster/* content after the test
2022-06-13 22:20:26 INFO     Recording local cluster logs for index-management with test configuration as without-security at /var/jenkins/workspace/integ-test/test-results/1/integ-test/index-management/without-security/local-cluster-logs
2022-06-13 22:20:26 INFO     Skipping security
2022-06-13 22:20:26 INFO     Skipping cross-cluster-replication
2022-06-13 22:20:26 INFO     | index-management     | with-security        | PASS  |    0 |
2022-06-13 22:20:26 ERROR    | index-management     | without-security     | FAIL  |    1 |

[downsrob]

This looks like a flaky failure and I was not able to reproduce locally. We have seen that test failing before here #90 (comment)
Do you happen to have any additional logs in case I can look into it further?

[downsrob]

I copied over the recent flaky test failures from the recent jenkins builds to our flaky test issue here. I will work to identify the root cause but it is a flaky failure and we have not introduced any code changes into 2.0.1. Closing this issue now as investigation can continue in the flaky test issue, feel free to reopen if you have anything else.