Plugins no longer found following logstash 8.x update with yum

[mattBaumBeneva]

Describe the bug
We are running Logstash 8.x installed with YUM (https://www.elastic.co/guide/en/logstash/current/installing-logstash.html#_yum) on Redhat Linux and using the Dynatrace output plugin (https://github.com/dynatrace-oss/logstash-output-dynatrace).

Each time we update the Logstash package, we end up with the following error messages in the service logs and our pipeline crashes. This affects the Dynatrace plugin, but we seem to also be experiencing this issue with other plugins, which is why I have posted here :

[2024-08-22T12:27:17,105][ERROR][logstash.plugins.registry] Unable to load plugin. {:type=>"output", :name=>"dynatrace"}
[2024-08-22T12:27:17,112][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (PluginLoadingError) Couldn't find any output plugin named 'dynatrace'. Are you sure this is correct? Trying to load the dynatrace output plugin resulted in this error: Unable to load the requested plugin named dynatrace of type output. The plugin is not installed.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:120)", "org.logstash.execution.AbstractPipelineExt.initialize(AbstractPipelineExt.java:186)", "org.logstash.execution.AbstractPipelineExt$INVOKER$i$initialize.call(AbstractPipelineExt$INVOKER$i$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:847)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1379)", "org.jruby.ir.instructions.InstanceSuperInstr.interpret(InstanceSuperInstr.java:139)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:363)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:128)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:115)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:446)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:92)", "org.jruby.RubyClass.newInstance(RubyClass.java:949)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:446)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:92)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:548)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:363)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:88)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:238)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:225)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:228)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:476)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:293)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:324)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:118)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:66)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.Block.call(Block.java:144)", "org.jruby.RubyProc.call(RubyProc.java:354)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:111)", "java.base/java.lang.Thread.run(Thread.java:1583)"]}

When this occurs, we must reinstall the plugin to fix the issue.
From what we can tell, the file still exists under /usr/share/logstash/vendor/bundle/jruby/3.1.0/gems, but logstash does not appear to load it correctly.

To Reproduce
Steps to reproduce the behavior:

1. Install a previous version of Logstash 8.x with yum on Redhat 8
2. Install the Dynatrace output plugin
3. Update the logstash package to the latest version
4. Restart the logstash systemd service
5. View logs with: journalctl -u logstash
6. See error

Expected behavior
Upgrading logstash should have no effect on previously-installed output plugins

Additional context
It isn't clear if this is a pluging-related issue of if this is a bug in the logstash update process. We see the bug affecting the dynatrace output plugin when we update, but we are getting reports that installations using other plugins present similar issues.

[mattBaumBeneva]

We have also opened a bug with the plugin maintainer here:
dynatrace-oss/logstash-output-dynatrace#37

[mattBaumBeneva]

We have been able to reproduce this in production with the "syslog" output plugin. This seems to be a more general issue with Logstash on Redhat Linux.