Setup Initial Jenkins Git Structure #744

lewijacn · 2024-06-19T19:32:03Z

Description

This change introduces a shared library structure for our Jenkins pipelines. As can be seen, this allows creating a pipeline template that our actual pipeline files can utilize and configure as needed.

Along with this, this change includes some common functions that will be needed for testing and have been added to the console library, with a future change expected to move our python testing files to be able to use this console library as well.

Issues Resolved

Partially resolves: https://opensearch.atlassian.net/browse/MIGRATIONS-1753

Testing

Local/Cloud testing

Check List

New functionality includes testing
- All tests pass, including unit test, integration test and doctest
New functionality has been documented
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Tanner Lewis <[email protected]>

codecov · 2024-06-19T19:33:13Z

Codecov Report

Attention: Patch coverage is 54.28571% with 32 lines in your changes missing coverage. Please review.

Project coverage is 68.21%. Comparing base (787fd26) to head (fd9ba8d).
Report is 4 commits behind head on main.

Files	Patch %	Lines
...le/lib/console_link/console_link/logic/clusters.py	40.62%	19 Missing ⚠️
...le/lib/console_link/console_link/models/cluster.py	21.42%	11 Missing ⚠️
...rationConsole/lib/console_link/console_link/cli.py	91.66%	2 Missing ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##               main     #744       +/-   ##
=============================================
- Coverage     89.37%   68.21%   -21.17%     
- Complexity        0     1578     +1578     
=============================================
  Files            46      270      +224     
  Lines          2702    11139     +8437     
  Branches          0      736      +736     
=============================================
+ Hits           2415     7598     +5183     
- Misses          287     3138     +2851     
- Partials          0      403      +403

Flag	Coverage Δ
gradle-test	`61.50% <ø> (?)`
python-test	`88.47% <54.28%> (-0.90%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Tanner Lewis <[email protected]>

mikaylathompson · 2024-06-20T16:37:18Z

...kerSolution/src/main/docker/migrationConsole/lib/console_link/console_link/logic/clusters.py

+        access_result['connection_message'] = "Successfully connected!"
+        access_result['cluster_version'] = response_json['version']['number']
+        access_result['connection_established'] = True


might be nice to make this access_result something like a named tuple or a very simple class so we can make a type guarantee on the return and the caller knows exactly what to assume they can get

Good idea have made a simple class around this

mikaylathompson · 2024-06-20T16:41:13Z

...kerSolution/src/main/docker/migrationConsole/lib/console_link/console_link/logic/clusters.py

+
+
+# As a default we exclude system indices and searchguard indices
+def clear_indices(cluster: Cluster):


Is this used anywhere? It's definitely very helpful for our testing, but I think we should put it behind a verification step where the user explicitly acknowledges that they're going to delete all the data in their cluster.

(helper for that https://click.palletsprojects.com/en/8.1.x/prompts/#confirmation-prompts)

potentially even do something like only create a CLI command for it (or at least only enable it) if there's an env variable like DEBUG_MODE or I_WANT_TO_DO_DANGEROUS_THINGS_TO_MY_CLUSTER set

Went ahead and added this as a CLI option, with a confirmation step added to it, thanks for the link

mikaylathompson · 2024-06-20T16:47:06Z

...kerSolution/src/main/docker/migrationConsole/lib/console_link/console_link/models/cluster.py

+            raise NotImplementedError(f"Auth type {self.auth_type} is not currently support for executing "
+                                      f"benchmark workloads")
+        logger.info(f"Running opensearch-benchmark with '{workload}' workload")
+        subprocess.run(f"opensearch-benchmark execute-test --distribution-version=1.0.0 "


Do the benchmark outputs print to stdout or are they captured here? Kind of nice to show the user that something's happening, so I think I'm moderately in favor of not-capturing, but don't feel super strongly.

Right now they are printing to stdout which I kinda like as well. I'm open to change that though if we decide we don't like it

Signed-off-by: Tanner Lewis <[email protected]>

# Conflicts: # TrafficCapture/dockerSolution/src/main/docker/migrationConsole/lib/console_link/console_link/models/cluster.py

Signed-off-by: Tanner Lewis <[email protected]>

mikaylathompson · 2024-06-21T16:25:59Z

...Capture/dockerSolution/src/main/docker/migrationConsole/lib/console_link/console_link/cli.py

+        click.echo("Performing clear indices operation...")
+        click.echo(logic_clusters.clear_indices(cluster_focus))
+    else:
+        if click.confirm('Clearing indices can result in data loss, are you sure you want to continue?'):


can we phrase this slightly more strongly: "Clearing indices WILL result in the loss of all data on the {source/target} cluster. Are you sure you want to continue?"

Updated to this wording

mikaylathompson · 2024-06-21T16:26:37Z

...Capture/dockerSolution/src/main/docker/migrationConsole/lib/console_link/console_link/cli.py

+    """[Caution] Clear indices on a source or target cluster"""
+    cluster_focus = ctx.env.source_cluster if cluster.lower() == 'source' else ctx.env.target_cluster
+    if acknowledge_risk:
+        click.echo("Performing clear indices operation...")


minor, but I'd also reiterate in these messages which cluster is being cleared.

I like it, added

mikaylathompson · 2024-06-21T16:34:03Z

...icCapture/dockerSolution/src/main/docker/migrationConsole/lib/console_link/tests/test_cli.py

+    assert result.exit_code == 0
+
+
+def test_cli_cluster_clear_indices(runner, env, mocker):


can you do a version of this test without --acknowledge-risk to make sure that it doesn't call clear?

mikaylathompson · 2024-06-21T16:37:50Z

jenkins/migrationIntegPipelines/ec2SourceE2EPipeline.groovy

+        //deployStep: {
+        //    echo 'Custom Test Step'
+        //}


intentional?

I was trying to use it as a reference that steps could be replaced if needed

Signed-off-by: Tanner Lewis <[email protected]>

lewijacn added 21 commits June 17, 2024 12:43

Add in default jenkinsFile

7e6f30e

Signed-off-by: Tanner Lewis <[email protected]>

Test 2

14ad3a9

Signed-off-by: Tanner Lewis <[email protected]>

Test 3

2fea84a

Signed-off-by: Tanner Lewis <[email protected]>

Update agent

dcf9096

Signed-off-by: Tanner Lewis <[email protected]>

Load pipeline

a63765e

Signed-off-by: Tanner Lewis <[email protected]>

Load pipeline 2

ab555d6

Signed-off-by: Tanner Lewis <[email protected]>

Add shared library

97dd360

Signed-off-by: Tanner Lewis <[email protected]>

Update agent default pipeline

dd676fd

Signed-off-by: Tanner Lewis <[email protected]>

Shared library changes

b34da81

Signed-off-by: Tanner Lewis <[email protected]>

Update shared lib

3a9d6fc

Signed-off-by: Tanner Lewis <[email protected]>

Test parameters

f5226c1

Signed-off-by: Tanner Lewis <[email protected]>

Update vars

6c5c2c4

Signed-off-by: Tanner Lewis <[email protected]>

Update vars

1a98a1c

Signed-off-by: Tanner Lewis <[email protected]>

Test shared lib

708df68

Signed-off-by: Tanner Lewis <[email protected]>

Test shared lib

3f6b900

Signed-off-by: Tanner Lewis <[email protected]>

Test shared lib

1abb6a4

Signed-off-by: Tanner Lewis <[email protected]>

Test shared lib

24c0a37

Signed-off-by: Tanner Lewis <[email protected]>

Test shared lib

63eae8a

Signed-off-by: Tanner Lewis <[email protected]>

Test shared lib

81563a1

Signed-off-by: Tanner Lewis <[email protected]>

Test shared lib

ff62416

Signed-off-by: Tanner Lewis <[email protected]>

Test shared lib

3c7ad9d

Signed-off-by: Tanner Lewis <[email protected]>

lewijacn added 7 commits June 19, 2024 15:33

Update pipeline files for main

ff1d9ac

Signed-off-by: Tanner Lewis <[email protected]>

Merge remote-tracking branch 'origin/main' into checkin-jenkinsfile

13865ac

Update comments

958fe73

Signed-off-by: Tanner Lewis <[email protected]>

Update for testing

6d5a519

Signed-off-by: Tanner Lewis <[email protected]>

Update backfill test

94071fc

Signed-off-by: Tanner Lewis <[email protected]>

Common functions added to console library

ca798d5

Signed-off-by: Tanner Lewis <[email protected]>

Minor fixes

b08cdc1

Signed-off-by: Tanner Lewis <[email protected]>

lewijacn marked this pull request as ready for review June 20, 2024 15:22

lewijacn requested review from AndreKurait, chelma, gregschohn, mikaylathompson, okhasawn, peternied and sumobrian as code owners June 20, 2024 15:22

mikaylathompson reviewed Jun 20, 2024

View reviewed changes

lewijacn added 7 commits June 21, 2024 10:46

PR feedback part 1

32dd6f7

Signed-off-by: Tanner Lewis <[email protected]>

PR feedback part 2

9b5993a

Signed-off-by: Tanner Lewis <[email protected]>

Merge remote-tracking branch 'origin/main' into checkin-jenkinsfile

8a3013c

# Conflicts: # TrafficCapture/dockerSolution/src/main/docker/migrationConsole/lib/console_link/console_link/models/cluster.py

Remove user string

02d9551

Signed-off-by: Tanner Lewis <[email protected]>

Lint fixes

93563ed

Signed-off-by: Tanner Lewis <[email protected]>

Add basic cli test cases

fc62873

Signed-off-by: Tanner Lewis <[email protected]>

Lint fixes

14c4cee

Signed-off-by: Tanner Lewis <[email protected]>

mikaylathompson reviewed Jun 21, 2024

View reviewed changes

lewijacn added 3 commits June 21, 2024 13:06

Merge remote-tracking branch 'origin/main' into checkin-jenkinsfile

cc11974

Signed-off-by: Tanner Lewis <[email protected]>

PR feedback adjustments

2ca87d0

Signed-off-by: Tanner Lewis <[email protected]>

Merge remote-tracking branch 'origin/main' into checkin-jenkinsfile

fd9ba8d

mikaylathompson approved these changes Jun 21, 2024

View reviewed changes

lewijacn merged commit 8ecb500 into opensearch-project:main Jun 21, 2024
10 of 12 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Setup Initial Jenkins Git Structure #744

Setup Initial Jenkins Git Structure #744

lewijacn commented Jun 19, 2024 •

edited

Loading

codecov bot commented Jun 19, 2024 •

edited

Loading

mikaylathompson Jun 20, 2024

lewijacn Jun 21, 2024

mikaylathompson Jun 20, 2024

mikaylathompson Jun 20, 2024

lewijacn Jun 21, 2024

mikaylathompson Jun 20, 2024

lewijacn Jun 21, 2024

mikaylathompson Jun 21, 2024

lewijacn Jun 21, 2024

mikaylathompson Jun 21, 2024

lewijacn Jun 21, 2024

mikaylathompson Jun 21, 2024

lewijacn Jun 21, 2024

mikaylathompson Jun 21, 2024

lewijacn Jun 21, 2024



		# As a default we exclude system indices and searchguard indices
		def clear_indices(cluster: Cluster):

		assert result.exit_code == 0


		def test_cli_cluster_clear_indices(runner, env, mocker):

Setup Initial Jenkins Git Structure #744

Setup Initial Jenkins Git Structure #744

Conversation

lewijacn commented Jun 19, 2024 • edited Loading

Description

Issues Resolved

Testing

Check List

codecov bot commented Jun 19, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lewijacn commented Jun 19, 2024 •

edited

Loading

codecov bot commented Jun 19, 2024 •

edited

Loading