0.6.0

QIT CLI 0.6.0 Release Notes

We are excited to announce the release of QIT CLI version 0.6.0! This update introduces the Activation Test 2.0, our first smart generic test, along with several enhancements designed to improve your testing workflow.

This is the first test type built on top of our Custom Tests platform!

What's New

Activation Test 2.0

• Improved Coverage: Now visits pages added by your extensions, catching errors that previous tests might have missed.
• Smart Generic Tests: Automatically detects and tests essential functionalities without custom scripts.
• Fast Execution: Completes tests in 50 seconds to 3 minutes, depending on your setup.
• Local Testing: Run tests directly on your machine or CI environment for immediate feedback.
• Error Detection: Captures PHP and JavaScript notices, warnings, and errors, including those on custom plugin pages, as well as Database errors.
• Configurable Environment:
  • PHP Versions: Test with PHP 7.2 up to 8.4 RC 3.
  • WooCommerce & WordPress Versions: Choose any version, including development builds.
• Plugin Dependencies: Activate additional plugins by their slugs to simulate real-world scenarios.
• Notifications: Use --notify to receive immediate alerts when issues are detected.
• Centralized Results: All test results are collected in QIT for easy access and analysis.
• Built on Custom Tests: Leverages existing frameworks for efficiency and easier maintenance.

Backwards Compatibility

• Seamless Integration: Designed to work with existing CI workflows without disruptions.
• Minimal Adjustments: No significant changes required for extensions currently using QIT.

Documentation

You can find the documentation for the Activation Test type here.

For any queries, issues, or suggestions, feel free to visit our Github page at https://github.com/woocommerce/qit-cli/issues. We appreciate your feedback as it helps us to continuously improve our tool.

What's Changed

• Fix --source option by @Luc45 in #196
• Sync 9.2.2 by @zhongruige in #197
• Add self-tests for the Validation test by @Luc45 in #198
• Activation as a custom test by @Luc45 in #200
• 9.3.0-beta.1 sync snapshots by @Luc45 in #201
• Activation/custom test improvements by @Luc45 in #202
• Update snapshots by @Luc45 in #204
• Fix old doc link by @zhongruige in #205
• Update snapshots for validation tests by @Luc45 in #206
• Activation refactor backwards compatibility by @Luc45 in #203
• Support qm logs by @MrJnrman in #195

Full Changelog: 0.5.4...0.6.0

0.5.4

QIT CLI 0.5.4 Release Notes

• Adds support for a QIT_RESULTS_DIR env var which specifies where to store the e2e results when running run:e2e command.

For any queries, issues, or suggestions, feel free to visit our Github page at https://github.com/woocommerce/qit-cli/issues. We appreciate your feedback as it helps us to continuously improve our tool.

What's Changed

• Update snapshots for Security Tests by @Luc45 in #179
• 9.0.0 sync by @Luc45 in #181
• Snapshot updates for WC 9.0.1 by @zhongruige in #182
• Sync e2e snapshots for WC 9.1.0 and 9.0.2 tests by @zhongruige in #183
• Sync WC 9.1.0 and 9.1.1 by @zhongruige in #184
• Wait 10 seconds between each self-test test by @Luc45 in #185
• Configurable poll intervall to prevent 429 by @Luc45 in #186
• Temporary test environment tweaks by @Luc45 in #190
• Sync 9.2.0-beta.1 by @Luc45 in #191
• Fix typo in docker compose version by @Luc45 in #192
• Set Result Dir using ENV vat by @MrJnrman in #194

Full Changelog: 0.5.3...0.5.4

0.5.3

QIT CLI 0.5.3 Release Notes

• This is a follow-up to the 0.5.2 release. Fixes an issue with using "--woo" parameter that would cause a Custom E2E test files to not be found.

For any queries, issues, or suggestions, feel free to visit our Github page at https://github.com/woocommerce/qit-cli/issues. We appreciate your feedback as it helps us to continuously improve our tool.

What's Changed

• Use Input instead of argv, activate Woo instead of bootstrapping when using "--woo" parameter by @Luc45 in #178

Full Changelog: 0.5.2...0.5.3

0.5.2

QIT CLI 0.5.2 Release Notes

• Fix an error that can happen when using both "--woo" and "--plugin=woocommerce" at the same time when running a Custom E2E Test.
• Many fixes and improvements

For any queries, issues, or suggestions, feel free to visit our Github page at https://github.com/woocommerce/qit-cli/issues. We appreciate your feedback as it helps us to continuously improve our tool.

What's Changed

• 9.0.0-RC1 snapshots with skips by @Luc45 in #172
• Custom tests snapshots by @Luc45 in #173
• Increase mass test timeout by @Luc45 in #175
• Sync Woo E2E 8.9.2 snapshots by @Luc45 in #174
• Update snapshots by @Luc45 in #176
• Fix issue with "--plugins=woocommerce" and "--woo" parameter by @Luc45 in #177

Full Changelog: 0.5.1...0.5.2

0.5.1

QIT CLI 0.5.1 Release Notes

• Register a test run in QIT for local custom test runs, with an embedded Allure Report with support for screenshots and videos of failures in E2E. You can share a link to that test report with others.
• Many fixes and improvements

For any queries, issues, or suggestions, feel free to visit our Github page at https://github.com/woocommerce/qit-cli/issues. We appreciate your feedback as it helps us to continuously improve our tool.

What's Changed

• Add woo:validate-zip <path> command by @rcstr in #163
• Upload test artifacts to debug self-tests flakiness by @Luc45 in #167
• Sync WC 8.9.0 by @zhongruige in #168
• 9.0.0-beta.2 sync by @Luc45 in #171
• Send custom tests results to Manager by @Luc45 in #170

New Contributors

• @rcstr made their first contribution in #163

Full Changelog: 0.5.0...0.5.1

0.5.0

QIT CLI 0.5.0 Release Notes

• New Feature: Local Test Environments: Added the ability to create Local Test Environments with a single command. These environments are disposable and fully customizable, making automated testing much simpler. (Read More: Test Environment Documentation)
• New Feature: Custom Tests: Added support for Custom E2E tests, allowing developers to write, share, and run personalized automated tests easily. (Read more: Custom Tests Documentation)
• Many fixes and improvements

For any queries, issues, or suggestions, feel free to visit our Github page at https://github.com/woocommerce/qit-cli/issues. We appreciate your feedback as it helps us to continuously improve our tool.

What's Changed

• Use Composer CA Bundle instead of custom implementation by @Luc45 in #139
• When doing a retry, throw some randomness into the mix by @Luc45 in #140
• Hide deprecated options (except on CI) by @Luc45 in #141
• Xdebug on QIT tests by @Luc45 in #142
• PHPCS Develop by @Luc45 in #145
• Add self-tests for WordPress.DB in Security Tests by @Luc45 in #146
• Custom E2E Tests 1 - Renaming Environment to Backend by @Luc45 in #133
• Custom E2E Tests 2 - Local Test Environment by @Luc45 in #130
• Local test env tweaks by @Luc45 in #147
• Update malware tests by @MrJnrman in #150
• Update snapshots for 8.7.0 sync by @Luc45 in #149
• Add empty extension list exception by @zhongruige in #152
• 8.8.0-beta.1 sync by @Luc45 in #155
• Added nonJsonOutput property by @zhongruige in #153
• Port Fast PHPCS PR to CLI by @Luc45 in #156
• Update WC 8.8 rc test snapshots by @zhongruige in #158
• Readme suggestions by @Luc45 in #159
• Sync 8.8.2 by @Luc45 in #161
• Rename woo.com to woocommerce.com by @zhongruige in #157
• Sync 8.9.0 RC by @zhongruige in #164
• Custom Tests by @Luc45 in #148
• Add more badges to the readme by @Luc45 in #165
• Tweaks to release process by @Luc45 in #166

Full Changelog: 0.4.2...0.5.0

0.4.2

QIT CLI 0.4.2 Release Notes

• When encountering "429 - Too Many Requests" when starting tests, respect the "Retry-After" header if it's present, otherwise do an exponential backoff.
• When running on Windows without a Certificate Authority file, ask the user if they want to download a CA file to be used in QIT, related to php/php-src#13601.

For any queries, issues, or suggestions, feel free to visit our Github page at https://github.com/woocommerce/qit-cli/issues. We appreciate your feedback as it helps us to continuously improve our tool.

What's Changed

• Update snapshots 8.7.0-beta.2 by @Luc45 in #135
• Retry after 429 by @Luc45 in #137
• Windows SSL by @Luc45 in #138

Full Changelog: 0.4.1...0.4.2

0.4.1

QIT CLI 0.4.1 Release Notes

• Fix the config path on native Windows.
• Groundwork for upcoming local test environments and integrated custom E2E tests for wide-scale compatibility testing.

For any queries, issues, or suggestions, feel free to visit our Github page at https://github.com/woocommerce/qit-cli/issues. We appreciate your feedback as it helps us to continuously improve our tool.

What's Changed

• Update docs by @Luc45 in #119
• Update snapshots for Woo 8.6.0-beta.1 E2E test sync by @Luc45 in #120
• Remove analytics-data.spec.js by @Luc45 in #121
• Allow retry on get request by @Luc45 in #123
• Test snapshots for 8.6.0 rc.1 by @Luc45 in #124
• Use dev version of symfony/process by @Luc45 in #126
• Rename "api" to "woo-api" and "e2e" to "woo-e2e" by @Luc45 in #125
• Minor tweak by @Luc45 in #128
• Update E2E snapshots for Woo 8.6.1 by @Luc45 in #129
• Add self-tests for Activation Tests loopback by @Luc45 in #131
• Fix QIT config path in Windows by @Luc45 in #134

Full Changelog: 0.4.0...0.4.1

0.4.0

QIT CLI 0.4.0 Release Notes

This is a strongly recommended update.

• Updates QIT URL to https://qit.woo.com.
• Renames "authentication_password" to "qit_token" for clarity.
• Bug fixes and improvements.

For any queries, issues, or suggestions, feel free to visit our Github page at https://github.com/woocommerce/qit-cli/issues. We appreciate your feedback as it helps us to continuously improve our tool.

What's Changed

• Ignore unescaped exceptions by @Luc45 in #113
• Add self-tests for PHPCompatibility with PHP 8.3 by @Luc45 in #112
• Debug failed self-tests by @Luc45 in #114
• Add self-test for PHPCompatibility to not flag attribute usage by @Luc45 in #117
• Update QIT CLI to the new domain by @Luc45 in #116
• Rename "Application Password" to "QIT Token" and other improvements by @Luc45 in #118

Full Changelog: 0.3.5...0.4.0

0.3.5

QIT CLI 0.3.5 Release Notes

This is an optional update.

• Return async response when using --json flag without waiting for the test completion (with --wait flag).
• Minor styling and consistency improvements.
• Bug fixes and improvements.

For any queries, issues, or suggestions, feel free to visit our Github page at https://github.com/woocommerce/qit-cli/issues. We appreciate your feedback as it helps us to continuously improve our tool.

What's Changed

• Update self-tests to use RC instead of a fixed version by @Luc45 in #48
• Update snapshots by @Luc45 in #49
• Add self-tests to PHPStan ignoring vendor directory by @Luc45 in #51
• Update self tests for 7.8.2 by @Luc45 in #53
• Allow to run multiple self-test suites separated by comma by @Luc45 in #55
• Update gitignore with common files by @zhongruige in #59
• Update self tests snapshots by @zhongruige in #63
• Update self-test snapshots for custom DB by @Luc45 in #65
• Update self tests snapshots by @zhongruige in #67
• Improvements to validate context by @zhongruige in #68
• Update self tests snapshots by @zhongruige in #72
• Update self tests snapshots by @zhongruige in #74
• Update to checkout@v4 by @Luc45 in #75
• Sync tests for 8.1.0-a.5 by @Luc45 in #76
• Run self-tests for Stable WooCommerce version as well by @Luc45 in #79
• Update self tests snapshots by @zhongruige in #82
• Add self-tests for allowing to require nonce checks in security tests by @Luc45 in #83
• Normalize debug log in self tests by @Luc45 in #84
• Update self tests snapshots by @zhongruige in #86
• Update self tests snapshots by @zhongruige in #87
• Self-tests for PHPCompatibility Test by @Luc45 in #80
• Update self tests snapshots by @zhongruige in #89
• Security Tests - Redirect rule tests by @Luc45 in #78
• Update instances of WooCommerce.com to Woo.com by @zhongruige in #90
• Update snapshots by @Luc45 in #91
• Update self tests snapshots by @zhongruige in #92
• Update self tests snapshots by @zhongruige in #95
• Update self tests snapshots by @zhongruige in #96
• Added malware self tests by @MrJnrman in #94
• Allow to use slug to list tests, color successes by @Luc45 in #99
• Sync Woo 8.4.0 by @Luc45 in #98
• Added Theme self tests by @MrJnrman in #97
• Update self-tests by @Luc45 in #100
• Improve self-test output, allow bigger variance of debug_logs by @Luc45 in #102
• Use zip from docker by @Luc45 in #103
• Add jetpack scan self test by @MrJnrman in #101
• Sync 8.5.0 by @Luc45 in #105
• Do not show onboarding on CI by @Luc45 in #107
• Update snapshots for WPCS 3.0.1 by @Luc45 in #110
• Unify run_id header in outputs. by @tomalec in #109

New Contributors

• @tomalec made their first contribution in #109

Full Changelog: 0.3.4...0.3.5