Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for user input and documentation improvements to smoke testing workflow #416

Open
wants to merge 37 commits into
base: main
Choose a base branch
from

Conversation

mitchelbaker-cisa
Copy link
Collaborator

@mitchelbaker-cisa mitchelbaker-cisa commented Sep 18, 2024

🗣 Description

This PR expands on the smoke testing workflow to be more dynamic, such that it can be triggered on push and pull_request events and manually with custom user input.

Made a couple other quality of life improvements to the workflow as well:

  • removed environment: Development key/value from the workflow which removes the x user deployed to Development 1 hour ago -- with GitHub Actions messages that cluttered pull request timelines.
  • the "SCuBA GWS Secure Configuration Baseline Reports" update in BaselineReports.html broke the selenium test since it was checking for the old format.

💭 Motivation and context

The smoke testing workflow is currently hardcoded to run against the following operating systems, python/OPA versions: ['windows-latest', 'macos-latest'], ['3.9', '3.12'], and v0.59.0. This is because the workflow_dispatch event for manual runs is only available after a workflow is committed to the main branch.

Closes #340
Closes #341
Closes #415

🧪 Testing

Go to the run_smoke_test.yml workflow in the GitHub Actions tab. Click the "Run workflow" button then set 340-smoketest-add-user-input as the ref branch.

Screenshot (216)

The default values are the following:

Screenshot (214)

Feel free to play around with the inputs then click the "Run workflow" button when ready. The workflow will create a matrix strategy for each combination. For example, passing ['windows-latest', 'macos-latest'], ['3.10', '3.11', 3.12'], and OPA version 0.60.0 will create the following:

Screenshot (218)

Some factors to consider:

  • Each input is required so an empty string will fail validation. [], [''], ['', ] may also cause the workflow to error out, although this is expected behavior.
  • ubuntu-latest has not been tested as a value for operating system. Support can be added for this, although its dependent on if this is something we want to test for ScubaGoggles as a whole.
  • Python versions <3.10.x are not supported and will cause the smoke test workflow to fail.
  • Due to the lack of an array input type from GitHub, the required format is an array of strings for the operating system and python version inputs. This is something to capture as a future todo once arrays are available.

The workflow runs successfully when testing push events with default values.
The workflow runs successfully when testing pull_request opened events with default values.

✅ Pre-approval checklist

  • This PR has an informative and human-readable title.
  • Changes are limited to a single goal - eschew scope creep!
  • If applicable, All future TODOs are captured in issues, which are referenced in the PR description.
  • The relevant issues PR resolves are linked preferably via closing keywords.
  • All relevant type-of-change labels have been added.
  • I have read and agree to the CONTRIBUTING.md document.
  • These code changes follow cisagov code standards.
  • All relevant repo and/or project documentation has been updated to reflect the changes in this PR.
  • Tests have been added and/or modified to cover the changes in this PR.
  • All new and existing tests pass.

✅ Pre-merge Checklist

  • This PR has been smoke tested to ensure main is in a functional state when this PR is merged.
  • Squash all commits into one PR level commit using the Squash and merge button.

✅ Post-merge Checklist

  • Delete the branch to clean up.
  • Close issues resolved by this PR if the closing keywords did not activate.

…w-size argument to chrome_options to fix failing selenium test
@mitchelbaker-cisa mitchelbaker-cisa changed the title Add support for user input when running smoke test workflow Add support for user input when running smoke test workflow and documentation improvements Sep 18, 2024
@mitchelbaker-cisa mitchelbaker-cisa changed the title Add support for user input when running smoke test workflow and documentation improvements Add support for user input and documentation improvements to smoke testing workflow Sep 18, 2024
@mitchelbaker-cisa mitchelbaker-cisa marked this pull request as ready for review September 18, 2024 23:11
@mitchelbaker-cisa mitchelbaker-cisa added the bug This issue or pull request addresses broken functionality label Sep 18, 2024
Copy link
Collaborator

@adhilto adhilto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes look good. I successfully manually triggered the smoke test, all tests passed. I have one minor suggestion for the README, but approving either way.

Testing/Functional/SmokeTests/README.md Outdated Show resolved Hide resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug This issue or pull request addresses broken functionality enhancement
Projects
None yet
2 participants