Add initial migration console snapshot support #751

AndreKurait · 2024-06-20T05:01:32Z

Description

Adds console snapshot create command full stack.
Adds stubbing for console snapshot status

Category: New feature
Why these changes are required? Enhance coverage of console library for end to end migrations
What is the old behavior before changes and new behavior after changes? No snapshot group in console library

Issues Resolved

Part 1 for MIGRATIONS-1792

Is this a backport? If so, please add backport PR # and/or commits #

Testing

Manual testing in both cloud and local environments.

root@ip-12-0-2-241:~# console snapshot create
17:36:46.236 INFO  Running RfsWorker
17:36:47.428 INFO  Checking if work remains in the Snapshot Phase...
17:36:49.526 INFO  Snapshot not yet completed, entering Snapshot Phase...
17:36:49.526 INFO  Snapshot CMS Entry not found, attempting to create it...
17:36:50.024 INFO  Snapshot CMS Entry created
17:36:50.025 INFO  Attempting to initiate the snapshot...
17:36:51.393 INFO  Snapshot repo registration successful
17:36:51.462 INFO  Snapshot rfs-snapshot creation initiated
17:36:51.463 INFO  Snapshot in progress...
17:36:51.722 INFO  Snapshot not finished yet; sleeping for 5 seconds...
17:36:56.730 INFO  Snapshot not finished yet; sleeping for 5 seconds...
17:37:01.745 INFO  Snapshot not finished yet; sleeping for 5 seconds...
17:37:06.753 INFO  Snapshot not finished yet; sleeping for 5 seconds...
...

Check List

New functionality includes testing
- All tests pass, including unit test, integration test and doctest
New functionality has been documented
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

codecov · 2024-06-20T05:22:59Z

Codecov Report

Attention: Patch coverage is 51.47059% with 66 lines in your changes missing coverage. Please review.

Project coverage is 64.79%. Comparing base (0501b56) to head (11dc524).
Report is 3 commits behind head on main.

Files	Patch %	Lines
RFS/src/main/java/com/rfs/common/RestClient.java	0.00%	26 Missing ⚠️
...le/lib/console_link/console_link/logic/snapshot.py	25.00%	18 Missing ⚠️
...e/lib/console_link/console_link/models/snapshot.py	63.82%	17 Missing ⚠️
...rationConsole/lib/console_link/console_link/cli.py	77.27%	5 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main     #751      +/-   ##
============================================
- Coverage     64.90%   64.79%   -0.11%     
- Complexity     1586     1588       +2     
============================================
  Files           239      241       +2     
  Lines          9919    10039     +120     
  Branches        771      772       +1     
============================================
+ Hits           6438     6505      +67     
- Misses         3072     3126      +54     
+ Partials        409      408       -1

Flag	Coverage Δ
gradle-test	`63.97% <13.33%> (-0.07%)`	⬇️
python-test	`72.89% <62.26%> (-1.57%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

mikaylathompson

Okay, after linting and tests are fixed up and this is merged, remaining pieces are:

implement status
add to CDK/ECS (I think I forgot to put that one in the task)

and I think that's it?

mikaylathompson · 2024-06-20T12:00:27Z

...dockerSolution/src/main/docker/migrationConsole/lib/console_link/console_link/environment.py

@@ -61,3 +64,11 @@ def __init__(self, config_file: str):
            logger.info(f"Backfill migration initialized: {self.backfill}")
        else:
            logger.info("No backfill provided")
+
+        if 'snapshot' in self.config:
+            self.snapshot: Snapshot = S3Snapshot(self.config["snapshot"],


Down the line, we likely want to clean this up and use a factory to give us a s3 snapshot vs local vs whatever, but I think it's good for now.

mikaylathompson · 2024-06-20T12:04:30Z

...erSolution/src/main/docker/migrationConsole/lib/console_link/console_link/models/snapshot.py

+        try:
+            subprocess.run(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, check=True)
+            logger.info(f"Snapshot {self.config['snapshot_name']} created successfully")
+            return CommandResult(success=True, value=f"Snapshot {self.config['snapshot_name']} created successfully")


ah, I kind of forgot that we don't have to do any of the s3 stuff ourselves! 👍

mikaylathompson · 2024-06-20T12:05:23Z

...icCapture/dockerSolution/src/main/docker/migrationConsole/lib/console_link/tests/test_cli.py

@@ -66,6 +67,35 @@ def test_cli_with_backfill_describe(runner, env, mocker):
    assert result.exit_code == 0


+def test_cli_snapshot_create(runner, env, mocker):
+    mock_create = mocker.patch('console_link.models.snapshot.Snapshot.create')


does this work? I think you might have to do a patch object, because create isn't a static method

Fixed by moving to mock the logic function

Signed-off-by: Andre Kurait <[email protected]>

AndreKurait · 2024-06-20T17:42:03Z

Okay, after linting and tests are fixed up and this is merged, remaining pieces are:

implement status

add to CDK/ECS (I think I forgot to put that one in the task)

and I think that's it?

I'll add status in a separate PR as it's tied with making create async

mikaylathompson · 2024-06-20T17:47:02Z

...icCapture/dockerSolution/src/main/docker/migrationConsole/lib/console_link/tests/test_cli.py

@@ -66,6 +67,40 @@ def test_cli_with_backfill_describe(runner, env, mocker):
    assert result.exit_code == 0


+def test_cli_snapshot_create(runner, env, mocker):
+    mock = mocker.patch('console_link.logic.snapshot.create')


the disadvantage of mocking this point instead is that we don't have test coverage for anything in the model, but I'm okay with improving this later.

AndreKurait force-pushed the console-lib-snapshot-create branch 2 times, most recently from dc24252 to d8d3641 Compare June 20, 2024 05:10

mikaylathompson approved these changes Jun 20, 2024

View reviewed changes

AndreKurait force-pushed the console-lib-snapshot-create branch 2 times, most recently from 49946c9 to 959dee9 Compare June 20, 2024 16:55

AndreKurait added 3 commits June 20, 2024 11:55

Add initial migration console snapshot support

e0ec666

Signed-off-by: Andre Kurait <[email protected]>

Send snapshot output while command is running

c19a155

Signed-off-by: Andre Kurait <[email protected]>

Fix test_cli.py for snapshot create

6a47641

Signed-off-by: Andre Kurait <[email protected]>

AndreKurait force-pushed the console-lib-snapshot-create branch from 959dee9 to 6a47641 Compare June 20, 2024 16:55

AndreKurait added 2 commits June 20, 2024 12:11

Add cdk for snapshot create

6a31fab

Signed-off-by: Andre Kurait <[email protected]>

Fix cluster allow_insecure

11dc524

Signed-off-by: Andre Kurait <[email protected]>

AndreKurait marked this pull request as ready for review June 20, 2024 17:37

AndreKurait requested review from chelma, gregschohn, lewijacn, okhasawn, peternied and sumobrian as code owners June 20, 2024 17:37

mikaylathompson reviewed Jun 20, 2024

View reviewed changes

AndreKurait merged commit 5413b91 into opensearch-project:main Jun 20, 2024
10 of 11 checks passed

AndreKurait deleted the console-lib-snapshot-create branch June 20, 2024 17:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add initial migration console snapshot support #751

Add initial migration console snapshot support #751

AndreKurait commented Jun 20, 2024 •

edited

Loading

codecov bot commented Jun 20, 2024 •

edited

Loading

mikaylathompson left a comment

mikaylathompson Jun 20, 2024

mikaylathompson Jun 20, 2024

mikaylathompson Jun 20, 2024

AndreKurait Jun 20, 2024

AndreKurait commented Jun 20, 2024

mikaylathompson Jun 20, 2024

Add initial migration console snapshot support #751

Add initial migration console snapshot support #751

Conversation

AndreKurait commented Jun 20, 2024 • edited Loading

Description

Issues Resolved

Testing

Check List

codecov bot commented Jun 20, 2024 • edited Loading

Codecov Report

mikaylathompson left a comment

Choose a reason for hiding this comment

mikaylathompson Jun 20, 2024

Choose a reason for hiding this comment

mikaylathompson Jun 20, 2024

Choose a reason for hiding this comment

mikaylathompson Jun 20, 2024

Choose a reason for hiding this comment

AndreKurait Jun 20, 2024

Choose a reason for hiding this comment

AndreKurait commented Jun 20, 2024

mikaylathompson Jun 20, 2024

Choose a reason for hiding this comment

AndreKurait commented Jun 20, 2024 •

edited

Loading

codecov bot commented Jun 20, 2024 •

edited

Loading