datadog cookbook constantly failing on windows servers

[benorgil]

Not sure whats causing this, but every few weeks all my windows chef runs start failing because of this datadog cookbook. I have the version pinned to 2.11.0 so it should not be updating the client, but it looks like it does periodically try to reinstall?

This kills me because every time this happens I have to interactively log into each server and run this command for it to force uninstall so the chef run will succeed:
msiexec /qn /i "c:\chef\cache\ddagent-cli.msi" /norestart ALLUSERS=1 /log dd-agent-install-log.txt NOTUPGRADE=1

Is it trying to reinstall the msi? why?

Errors from failed runs:

Recipe: datadog::dd-agent

• ruby_block[datadog-api-key-unset] action run[2018-03-30T11:23:25-04:00] INFO: Processing ruby_block[datadog-api-key-
  unset] action run (datadog::dd-agent line 21)
  (skipped due to only_if)
  Recipe: datadog::_install-windows

• windows_package[Datadog Agent removal] action nothing[2018-03-30T11:23:25-04:00] INFO: Processing windows_package[Da
  tadog Agent removal] action nothing (datadog::_install-windows line 58)
  (skipped due to action :nothing)

• remote_file[c:/chef/cache/ddagent-cli.msi] action create[2018-03-30T11:23:25-04:00] INFO: Processing remote_file[c:/
  chef/cache/ddagent-cli.msi] action create (datadog::_install-windows line 68)
  (up to date)

• windows_package[Datadog Agent] action install[2018-03-30T11:23:26-04:00] INFO: Processing windows_package[Datadog Ag
  ent] action install (datadog::_install-windows line 83)

  ================================================================================
  Error executing action install on resource 'windows_package[Datadog Agent]'

  Mixlib::ShellOut::ShellCommandFailed

  Expected process to exit with [0, 3010], but received '1603'
  ---- Begin output of msiexec /qn /i "c:\chef\cache\ddagent-cli.msi" /norestart ALLUSERS=1 ----
  STDOUT:
  STDERR:
  ---- End output of msiexec /qn /i "c:\chef\cache\ddagent-cli.msi" /norestart ALLUSERS=1 ----
  Ran msiexec /qn /i "c:\chef\cache\ddagent-cli.msi" /norestart ALLUSERS=1 returned 1603

  Resource Declaration:

  In c:/chef/cache/cookbooks/datadog/recipes/_install-windows.rb

  83: windows_package 'Datadog Agent' do # ~FC009
  84: source temp_file
  85: installer_type installer_type
  86: options install_options
  87: action :install
  88: if respond_to?(:returns)
  89: returns [0, 3010]
  90: else
  91: success_codes [0, 3010]
  92: end
  93: end

[olivielpeau]

Hi @benorgil and sorry for the very late reply.

To pin the version of the Agent that's installed by the cookbook, you should make sure agent_package_action is left to its default or set to "install", and use the agent6_version (Agent v6) or agent_version (Agent 5) attribute (see documentation in default attribute file: https://github.com/DataDog/chef-datadog/blob/master/attributes/default.rb, please use the one from your version of the cookbook).

Also, please note that, currently, the cookbook will reinstall the Agent if you clear Chef's cache directory. Unfortunately this was done to support older versions of Chef (<12.6), we'll attempt to drop support for these old versions and improve the logic in v3.0 of the present cookbook.

Does this info help? Which version of Chef are you running?

[bwillif1]

I'm getting the same error code '1603' whenever datadog upgrades the agent. I have datadog pinned to version 6.11.0 in chef, so some of my old clients are upgrading. It removes the agent with no problem but fails to re-install it. I experience the same issue when trying to install the msi manually from the server itself.

Browsing on the datadog site for properly uninstalling the agent https://docs.datadoghq.com/agent/faq/how-do-i-uninstall-the-agent/?tab=agentv6
I see this statement: "It’s important that the original account used to install the Agent is also used to remove it, otherwise it’s possible remnants are left behind and it won’t be cleanly removed."
I'm thinking this is what is contributing to the failing agent installs.

However, the user who previously manually installed an older version of datadog left the company and their account no longer exists. Does this mean I can never install datadog on these servers again? There has to be a better way to handle uninstalls/installs.

[Swampcritter]

We are running the datadog cookbook through the use of a in-house wrapper to enable log aggregation. The Linux side of the cookbook works fine, however it keeps failing on the Windows installs.

chef-client -r recipe[wabtec_datadog]
+---------------------------------------------+
✔ 2 product licenses accepted.
+---------------------------------------------+
Starting Chef Infra Client, version 15.9.17
resolving cookbooks for run list: ["wabtec_datadog"]
Synchronizing Cookbooks:

• chef_handler (3.0.3)
• yum (5.1.0)
• apt (7.3.0)
• datadog (4.3.0)
• wabtec_datadog (0.3.7)
  Installing Cookbook Gems:
  Compiling Cookbooks...
  [2020-05-08T09:49:33-04:00] WARN: Resource chef_handler from the client is overriding the resource from a cookbook. Please upgrade your cookbook or remove the cookbook from your run_list.
  Converging 12 resources
  Recipe: datadog::dd-agent

• ruby_block[datadog-api-key-unset] action run (skipped due to only_if)
  Recipe: datadog::_install-windows

• windows_package[Datadog Agent removal] action nothing (skipped due to action :nothing)

• remote_file[C:\chef\cache/ddagent-cli.msi] action create

  • create new file C:\chef\cache/ddagent-cli.msi
  • update content in file C:\chef\cache/ddagent-cli.msi from none to f250d6
    (file sizes exceed 10000000 bytes, diff output suppressed)

• remote_file[C:\chef\cache/fix_6_14.ps1] action nothing (skipped due to action :nothing)

• powershell_script[datadog_6.14.x_fix] action nothing (skipped due to action :nothing)

• windows_package[Datadog Agent] action install

  ================================================================================
  Error executing action install on resource 'windows_package[Datadog Agent]'

  Mixlib::ShellOut::ShellCommandFailed

  Expected process to exit with [0, 3010], but received '1603'
  ---- Begin output of msiexec /qn /i "c:\chef\cache\ddagent-cli.msi" /norestart ALLUSERS=1 ----
  STDOUT:
  STDERR:
  ---- End output of msiexec /qn /i "c:\chef\cache\ddagent-cli.msi" /norestart ALLUSERS=1 ----
  Ran msiexec /qn /i "c:\chef\cache\ddagent-cli.msi" /norestart ALLUSERS=1 returned 1603

  Resource Declaration:

  In C:/chef/cache/cookbooks/datadog/recipes/_install-windows.rb

  112: windows_package 'Datadog Agent' do # ~FC009
  113: source temp_file
  114: installer_type resolved_installer_type
  115: options install_options
  116: timeout node['datadog']['windows_msi_timeout']
  117: action :install
  118: # Before 3.0.0, we adviced users to use the windows cookbook ~> 1.38.0,
  119: # we should probably keep the compatibilty for some time.
  120: if respond_to?(:returns)
  121: returns [0, 3010]
  122: else
  123: success_codes [0, 3010]
  124: end
  125: not_if do

  Compiled Resource:

  Declared in C:/chef/cache/cookbooks/datadog/recipes/_install-windows.rb:112:in `from_file'

  windows_package("Datadog Agent") do
  package_name "Datadog Agent"
  action [:install]
  default_guard_interpreter :default
  declared_type :windows_package
  cookbook_name "datadog"
  recipe_name "_install-windows"
  timeout 900
  source "c:\chef\cache\ddagent-cli.msi"
  installer_type :msi
  options "/norestart ALLUSERS=1"
  returns [0, 3010]
  version "7.19.1.1"
  not_if { #code block }
  end

  System Info:

  chef_version=15.9.17
  platform=windows
  platform_version=10.0.14393
  ruby=ruby 2.6.6p146 (2020-03-31 revision 67876) [x64-mingw32]
  program_name=C:/opscode/chef/bin/chef-client
  executable=C:/opscode/chef/bin/chef-client

Running handlers:
[2020-05-08T09:53:22-04:00] ERROR: Running exception handlers
Running handlers complete
[2020-05-08T09:53:22-04:00] ERROR: Exception handlers complete
Chef Infra Client failed. 1 resources updated in 03 minutes 58 seconds
[2020-05-08T09:53:22-04:00] FATAL: Stacktrace dumped to C:/chef/cache/chef-stacktrace.out
[2020-05-08T09:53:22-04:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2020-05-08T09:53:22-04:00] FATAL: Mixlib::ShellOut::ShellCommandFailed: windows_package[Datadog Agent] (datadog::_insta
ll-windows line 112) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0, 3010], but received '1603'
---- Begin output of msiexec /qn /i "c:\chef\cache\ddagent-cli.msi" /norestart ALLUSERS=1 ----
STDOUT:
STDERR:
---- End output of msiexec /qn /i "c:\chef\cache\ddagent-cli.msi" /norestart ALLUSERS=1 ----
Ran msiexec /qn /i "c:\chef\cache\ddagent-cli.msi" /norestart ALLUSERS=1 returned 1603