[CI] Build and verify release artifacts for plugins for each OS

[kavilla]

Is your feature request related to a problem? Please describe.

OpenSearch Dashboards verifies release builds and artifacts with BWC tests being able to execute. But this process should be easy for plugins to execute and very prior to a release.

For example,
opensearch-project/dashboards-search-relevance#184 works on local development and builds release artifacts successful. But once compiled down it would have run-time errors.

Describe the solution you'd like

Verification of the artifact compiled down.

Script that can be housed in the core repo in the OSD Plugin Helpers package or within the SDK. Script will be responsible for building the plugin and verifying the successful installation and runtime of a plugin within an OpenSearch Dashboards release distribution.

The goal to being to provide a generic solution that reduces the redundancy of having every plugin had a build workflow that installs the plugin on the release build and pings the health of OpenSearch Dashboards. This solution should still be verified with a workflow but the script will be something like yarn plugin-helpers verify that exits 0 on success of installation and runtime or exits 1 on failure installation and runtime.

Describe alternatives you've considered

Plugins add a workflow to pull down release version of OpenSearch Dashboards build the zip of their plugin and install the plugin. Then run some basic sanity tests on OpenSearch Dashboards after the application has started.

Additional context

n/a

[ashwin-pc]

@kavilla can you add more context to this issue. I dont think i quite understand what this issue is asking for. Isnt this what the BWC tests already do? or are you suggesting that we build and verify artifacts with plugins included in our CI? I would strongly reccomend against that since that validation should not happen in on the platform repo. Each plugin should be responsible for building dashboards with their plugin installed and validating against that in the plugin repo directly.

[kavilla]

@kavilla can you add more context to this issue. I dont think i quite understand what this issue is asking for. Isnt this what the BWC tests already do? or are you suggesting that we build and verify artifacts with plugins included in our CI? I would strongly reccomend against that since that validation should not happen in on the platform repo. Each plugin should be responsible for building dashboards with their plugin installed and validating against that in the plugin repo directly.

@ashwin-pc sorry about the delay. Updated. To sum up, yes I think validation should happen in every repo but this proposal being a generic script that plugins can call something like yarn plugin-helpers verify.

[ashwin-pc]

Thanks for the context, this makes sense 👍 Would it make sense to also have this do something similar to the BWC tests? so that the plugins can also write BWC tests for their features and use this script to run them. Each plugin will likely have their own sanity and bwc flows that we cannot generalize, but by giving them a script that can point to flows version controlled in their repo, we can extend this to be more through. If this was already planned a s a part of this script, ignore my suggestion :)