[BUG] Security plugin interfering with Performance Analyzer metric collection

[Tjofil]

What is the bug?
Performance Analyzer, for a subset of it's available metrics, uses transport channels to get notified of certain events from core Opensearch functionalities. These events are usually paired in start and finish event emissions, i.e. ShardBulk start and finish events. We recently discovered that ShardBulk finish event is not emitted when running Opensearch with Security plugin installed, while start events function properly, uninstalling Security solves the problem and finish events are present (Issue).Not having finish events prevents metrics for being written altogether.

Issue is certainly related to Transport interceptors interfering with each other when invoked from ShardBulk finish function in core.
More detailed description, screenshots, logs and some assumptions can be found in issue comment.

What is the expected behavior?
Both start and finish events are getting emitted.

How can one reproduce the bug?
Steps to reproduce the behavior can be found in issue description.

[peternied]

@Tjofil Thanks for filing, I just posted to your issue a moment ago. My intuition says that these actions should be triggered with out the user credentials, but I would need a more detailed workflow to know for certain. See how this is done in the following issue. Unfortunately, we do not have an more security alternative for plugins at this time.

• [FEATURE] Deprecate the usage of ThreadContext.stashContext in plugins #2487

[peternied]

Please let me know if you believe this issue is still outstanding.

[peternied]

Context on by design: This is the purpose of the security plugin to block transport actions that are not authorized. We should not weaken/remove these checks within the security plugin as it would expose clusters without the consent of cluster administrators or the intention of the plugin.