NewRelic 9.10.0 stopped reporting background jobs

[edariedl]

Description

After upgrade to 9.10.0, newrelic UI stopped recording Non-Web transactions (background jobs). It started working again after downgrade back to 9.9.0.

Expected Behavior

It should still record background jobs.

Steps to Reproduce

Upgrade the gem to 9.10.0

Your Environment

Ruby 3.3.1 with YJIT
Rails 7.1.3.3
Background jobs: ActiveJob with SolidQueue 0.3.2.

For Maintainers Only or Hero Triaging this bug

Suggested Priority (P1,P2,P3,P4,P5):
Suggested T-Shirt size (S, M, L, XL, Unknown):

[workato-integration]

https://new-relic.atlassian.net/browse/NR-276480

[kaylareopelle]

Hi @edariedl! Thanks for bringing this to our attention.

We have a few ideas about what might be causing this problem:

1. We received a bug report about problems with agent startup when the bootstrap gem was installed alongside version 9.10.0 of newrelic_rpm. We've released 9.10.1 with a fix for this problem. If you have bootstrap in this application, could you try upgrading to the new version?

2. Version 9.10.0 also changed the autostart.denylisted_constants configuration default value. We intended this to mean the agent would not start up when rails runner or rails db commands were run by adding Rails::Command::RakeCommand and Rails::Command::RunnerCommand to the list. However, this might have had unintended consequences. If the first option doesn't solve your reporting issue, could you try manually setting autostart.denylisted_constants to the following in your config/newrelic.yml? (This is just the original list, with the new constants removed)

# config/newrelic.yml
common: &default_settings
  autostart.denylisted_constants: Rails::Command::ConsoleCommand,Rails::Command::CredentialsCommand,Rails::Command::Db::System::ChangeCommand,Rails::Command::DbConsoleCommand,Rails::Command::DestroyCommand,Rails::Command::DevCommand,Rails::Command::EncryptedCommand,Rails::Command::GenerateCommand,Rails::Command::InitializersCommand,Rails::Command::NotesCommand,Rails::Command::RoutesCommand,Rails::Command::SecretsCommand,Rails::Console,Rails::DBConsole

3. If neither of these options solve your problem, let us know and we can continue investigating!

[edariedl]

Hi @kaylareopelle!

It looks like it was the change in the autostart.denylisted_constants that caused the problem with background job recording. Simple upgrade to 9.10.1 didn't help but suggested change in the autostart.denylisted_constants fixed it.

Thank you very much for your help!

[kaylareopelle]

That's great news! Thanks for the update, @edariedl!

Now that we've identified the problem, we've created a few issues to solve this in a longer-term way:

• Revert autostart_denylisted_constants default change and document options #2691
• Spike: Investigate autostart.denylisted_constants #2692

I'm going to close this issue for now. If you have any ideas or suggestions for the issues linked above, please add them directly to those issues.

If anyone comes across this issue and needs a fix, please update the configuration value for autostart.denylisted_constants to:

Rails::Command::ConsoleCommand,Rails::Command::CredentialsCommand,Rails::Command::Db::System::ChangeCommand,Rails::Command::DbConsoleCommand,Rails::Command::DestroyCommand,Rails::Command::DevCommand,Rails::Command::EncryptedCommand,Rails::Command::GenerateCommand,Rails::Command::InitializersCommand,Rails::Command::NotesCommand,Rails::Command::RoutesCommand,Rails::Command::SecretsCommand,Rails::Console,Rails::DBConsole

This removes Rails::Command::RakeCommand and Rails::Command::RunnerCommand from the defaults.

[edariedl]

Thanks a lot for the help!

Now when I am looking at it, it makes some much sense that inclusion of Rails::Command::RakeCommand caused problems with background job instrumentation with SolidQueue. It uses a rake command to start all of its processes.

SolidQueue will be the default background job backend for rails.

[kaylareopelle]

Thanks for the tip, @edariedl!

This helped us decide to remove only Rails::Command::RakeCommand from the default list of constants in autostart.denylisted_constants, since it seems Solid Queue was impacted by the inclusion of this constant and not the other.

This change was included in version 9.10.2, released yesterday!