[windows] safely handle buggy versions - 3.x (#652)

* [windows] try to make safer against 6.14.x apocalypse * [metadata] bump version to 2.3.0 * [windows] indentation fix * [windows] re-order powershell script * [windows] multiple fixes * [windows] better raise error message * [windows] fix: use stable PS script * [windows] no install if bad hashsum [windows] avoid ruby block - successive runs made the solution iffy * [docs] updating changelog + readme * [windows] allow silent failures * [docs] updated README instructions. * [README] better versioning references. * [windows][spec] fix tests. * [attribute] better document windows_blacklist_silent_fail * [changelog] adding PR references * Apply suggestions from code review
DataDog · Nov 11, 2019 · 3328b80 · 3328b80
1 parent 4cc5ae2
commit 3328b80
Show file tree

Hide file tree

Showing 7 changed files with 190 additions and 8 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,6 +1,11 @@
 Changes
 =======
 
+# 3.4.0 / Unreleased
+
+* [FEATURE] Blacklist installation of 6.14.0 and 6.14.1. See [#652][] [@truthbk][]
+* [FEATURE] Run fix + sanity check script before agent uninstalls. See [#652][] [@truthbk][]
+
 # 3.3.0 / 2019-09-25
 
 * [FEATURE] Add RHEL8/Fedora 28 support (needs Chef >= 15). See [#641][] [@KSerrania][]
@@ -773,6 +778,7 @@ A fix has gone in to `apt` 2.1.0 that relaxes this condition, and plays well wit
 [#635]: https://github.com/DataDog/chef-datadog/issues/635
 [#639]: https://github.com/DataDog/chef-datadog/issues/639
 [#641]: https://github.com/DataDog/chef-datadog/issues/641
+[#652]: https://github.com/DataDog/chef-datadog/issues/652
 [@ABrehm264]: https://github.com/ABrehm264
 [@AlexBevan]: https://github.com/AlexBevan
 [@BrentOnRails]: https://github.com/BrentOnRails
@@ -911,4 +917,4 @@ A fix has gone in to `apt` 2.1.0 that relaxes this condition, and plays well wit
 [@wolf31o2]: https://github.com/wolf31o2
 [@xt99]: https://github.com/xt99
 [@yannmh]: https://github.com/yannmh
-[@zshenker]: https://github.com/zshenker
+[@zshenker]: https://github.com/zshenker
diff --git a/README.md b/README.md
@@ -69,6 +69,36 @@ dd-handler recipe. The known workaround is to update your dependency to `chef_ha
 the `chef_handler` cookbook which is now shipped as a resource in Chef 14.
 Unfortunately, it will display a deprecation message to Chef 14 and 15 users.
 
+Known Issues
+============
+
+Due to a critical bug in agent versions `6.14.0` and `6.14.1` for Windows, these versions have
+been blacklisted in cookbook versions `>=3.4.0`. `>=2.20.0`.
+
+**PLEASE NOTE:** chef-runs will fail on windows when updating to cookbook versions `>=3.4.0`. `>=2.20.0`
+if:
+- Any of the blacklisted Datadog Agent versions (`6.14.0` or `6.14.1`) has been pinned in `agent6_version`.
+- No Agent version is specified in the `agent6_version` attribute (defaulting to `6.14.1` at the
+time of this writing)
+
+After uploading the latest datadog cookbook to your Chef Server and ensuring the cookbook has been
+propagated to all nodes, upgrade the Agent version to `6.14.2` by setting the `agent6_version` attribute.
+
+If you are updating from **6.14.0 or 6.14.1 on Windows**, we **strongly** recommend following these steps:
+
+1. Install the latest datadog cookbook on your Chef Server (`>=3.4.0` or `>=2.20.0`)
+2. Remove ALL prior versions of the datadog cookbook to ensure only the latest version of the cookbook
+is available to your hosts. Please make sure to backup any modified or forked versions of the datadog
+cookbook using `knife` or `berks` if you need to reference them later.
+3. Set the `agent6_version` attribute to `6.14.2`
+
+**PLEASE NOTE:** It is important to ensure that your hosts ONLY use the latest version of the 
+datadog cookbook (`>=3.4.0` or `>=2.20.0`), before setting the `agent6_version` attribute. This is a
+precaution to ensure older versions of the datadog cookbook do not trigger the uninstall bug.
+
+To learn more about this bug, please read [here](http://dtdg.co/win-614-fix).
+
+
 Recipes
 =======
 
@@ -317,8 +347,12 @@ To upgrade from an already installed Agent v5 to Agent v6, you'll have to set th
   default_attributes(
     'datadog' => {
       'agent6' => true,
-      'agent6_version' => '1:6.10.0-1', # optional but recommended
       'agent6_package_action' => 'install',
+      'agent6_version' => {
+        'debian' => '1:6.14.2-1',
+        'rhel' => '6.14.2-1',
+        'windows' => '6.14.2'
+      }
     }
   )
 ```
@@ -343,7 +377,7 @@ You will need to indicate that you want to setup an Agent v5 instead of v6, pin
 
 1. Add this cookbook to your Chef Server, either by installing with knife or by adding it to your Berksfile:
   ```
-  cookbook 'datadog', '~> 3.0.0'
+  cookbook 'datadog', '~> 3.4.0'
   ```
 2. Add your API Key either:
   * as a node attribute via an `environment` or `role`, or
@@ -365,6 +399,11 @@ You will need to indicate that you want to setup an Agent v5 instead of v6, pin
       'agent6' => true,
       'api_key' => 'api_key',
       'application_key' => 'app_key',
+      'agent6_version' => {
+        'debian' => '1:6.14.2-1',
+        'rhel' => '6.14.2-1',
+        'windows' => '6.14.2'
+      },
       'mongo' => {
         'instances' => [
           {'host' => 'localhost', 'port' => '27017'}
@@ -410,7 +449,7 @@ AWS OpsWorks Chef Deployment
 
 1. Add Chef Custom JSON:
   ```json
-  {"datadog":{"agent6": true, "api_key": "<API_KEY>", "application_key": "<APP_KEY>"}}
+  {"datadog":{"agent6": true, "agent6_version": "1:6.14.2-1",  "api_key": "<API_KEY>", "application_key": "<APP_KEY>"}}
   ```
 
 2. Include the recipe in install-lifecycle recipe:

diff --git a/attributes/default.rb b/attributes/default.rb
@@ -182,6 +182,15 @@
 # of the Agent will be signed with this key.
 default['datadog']['yumrepo_gpgkey_new'] = "#{yum_protocol}://yum.datadoghq.com/DATADOG_RPM_KEY_E09422B3.public"
 
+# Windows Agent Blacklist
+# Attribute to enforce silent failures on agent installs when attempting to install a
+# blacklisted MSI. This attribute is to provide a workaround for users already pinned
+# to a blacklisted windows Agent version who may need to avoid breaking their chef runs.
+# The new blacklisting logic, by default will fail your chef run.
+# Please note that this attribute is no silver bullet, not all failed chef runs due to
+# the blacklist will be resolved enabling this feature.
+default['datadog']['windows_blacklist_silent_fail'] = false
+
 # Agent installer checksum
 # Expected checksum to validate correct agent installer is downloaded (Windows only)
 default['datadog']['windows_agent_checksum'] = nil

diff --git a/metadata.rb b/metadata.rb
@@ -4,7 +4,7 @@
 license          'Apache-2.0'
 description      'Installs/Configures datadog components'
 long_description IO.read(File.join(File.dirname(__FILE__), 'README.md'))
-version          '3.3.0'
+version          '3.4.0'
 chef_version     '>= 12.7'
 source_url       'https://github.com/DataDog/chef-datadog'
 issues_url       'https://github.com/DataDog/chef-datadog/issues'

diff --git a/recipes/_install-windows.rb b/recipes/_install-windows.rb
@@ -74,14 +74,42 @@
 package_retries = node['datadog']['agent_package_retries']
 package_retry_delay = node['datadog']['agent_package_retry_delay']
 
+unsafe_hashsums = [
+  '928b00d2f952219732cda9ae0515351b15f9b9c1ea1d546738f9dc0fda70c336',
+  '78b2bb2b231bcc185eb73dd367bfb6cb8a5d45ba93a46a7890fd607dc9188194'
+]
+fix_message = 'The file downloaded matches a known unsafe MSI - Agent versions 6.14.0/1 have been blacklisted. please use a different release. '\
+        'See http://dtdg.co/win-614-fix'
+
 # Download the installer to a temp location
 remote_file temp_file do
   source node['datadog']['windows_agent_url'] + dd_agent_installer
   checksum node['datadog']['windows_agent_checksum'] if node['datadog']['windows_agent_checksum']
   retries package_retries unless package_retries.nil?
   retry_delay package_retry_delay unless package_retry_delay.nil?
+
+  # validate the downloaded MSI is safe
+  verify do |path|
+    require 'digest'
+
+    unsafe = unsafe_hashsums.include? Digest::SHA256.file(path).hexdigest
+    Chef::Log.error("\n#{fix_message}\n") if unsafe
+
+    # verify will abort update if false
+    !unsafe
+  end unless node['datadog']['windows_blacklist_silent_fail']
+
+  notifies :run, 'powershell_script[datadog_6.14.x_fix]', :immediately
+end
+
+powershell_script 'datadog_6.14.x_fix' do
   # As of v1.37, the windows cookbook doesn't upgrade the package if a newer version is downloaded
   # As a workaround uninstall the package first if a new MSI is downloaded
+  code <<-EOH
+((New-Object System.Net.WebClient).DownloadFile('https://s3.amazonaws.com/ddagent-windows-stable/scripts/fix_6_14.ps1', $env:temp + '\\fix_6_14.ps1')); &$env:temp\\fix_6_14.ps1
+  EOH
+
+  action :nothing
   notifies :remove, 'package[Datadog Agent removal]', :immediately
 end
 
@@ -98,4 +126,12 @@
   else
     success_codes [0, 3010]
   end
+  not_if do
+    require 'digest'
+
+    unsafe = unsafe_hashsums.include? Digest::SHA256.file(temp_file).hexdigest
+    Chef::Log.warn("\n#{fix_message}\nContinuing without installing Datadog Agent.") if unsafe
+
+    unsafe
+  end
 end
diff --git a/spec/dd-agent_spec.rb b/spec/dd-agent_spec.rb
@@ -1,4 +1,5 @@
 require 'spec_helper'
+require 'digest'
 
 module EnvVar
   def set_env_var(name, value)
@@ -139,6 +140,16 @@ def set_env_var(name, value)
         end.converge described_recipe
       end
 
+      temp_file = ::File.join('C:/chef/cache', 'ddagent-cli.msi')
+      mock_digest = Digest::SHA256.new
+
+      before do
+        allow(File).to receive(:open).and_call_original
+        allow(File).to receive(:open).with(temp_file).and_return('foo')
+        allow(Digest::SHA256).to receive(:file).and_call_original
+        allow(Digest::SHA256).to receive(:file).with(temp_file).and_return(mock_digest)
+      end
+
       it_behaves_like 'windows Datadog Agent', :msi
     end
 
@@ -157,6 +168,16 @@ def set_env_var(name, value)
         end.converge described_recipe
       end
 
+      temp_file = ::File.join('C:/chef/cache', 'ddagent-cli.exe')
+      mock_digest = Digest::SHA256.new
+
+      before do
+        allow(File).to receive(:open).and_call_original
+        allow(File).to receive(:open).with(temp_file).and_return('foo')
+        allow(Digest::SHA256).to receive(:file).and_call_original
+        allow(Digest::SHA256).to receive(:file).with(temp_file).and_return(mock_digest)
+      end
+
       it_behaves_like 'windows Datadog Agent', :exe
     end
   end
@@ -206,6 +227,7 @@ def set_env_var(name, value)
     context 'on windows' do
       cached(:chef_run) do
         set_env_var('ProgramData', 'C:\ProgramData')
+
         ChefSpec::SoloRunner.new(
           :platform => 'windows',
           :version => '2012R2',
@@ -220,6 +242,14 @@ def set_env_var(name, value)
       end
 
       temp_file = ::File.join('C:/chef/cache', 'ddagent-cli.msi')
+      mock_digest = Digest::SHA256.new
+
+      before do
+        allow(File).to receive(:open).and_call_original
+        allow(File).to receive(:open).with(temp_file).and_return('foo')
+        allow(Digest::SHA256).to receive(:file).and_call_original
+        allow(Digest::SHA256).to receive(:file).with(temp_file).and_return(mock_digest)
+      end
 
       it_behaves_like 'windows Datadog Agent v5', :msi
       # remote_file source gets converted to an array, so we need to do
@@ -262,6 +292,7 @@ def set_env_var(name, value)
     context 'when windows' do
       cached(:chef_run) do
         set_env_var('ProgramData', 'C:\ProgramData')
+
         ChefSpec::SoloRunner.new(
           :platform => 'windows',
           :version => '2012R2',
@@ -283,6 +314,14 @@ def set_env_var(name, value)
       end
 
       temp_file = ::File.join('C:/chef/cache', 'ddagent-cli.msi')
+      mock_digest = Digest::SHA256.new
+
+      before do
+        allow(File).to receive(:open).and_call_original
+        allow(File).to receive(:open).with(temp_file).and_return('foo')
+        allow(Digest::SHA256).to receive(:file).and_call_original
+        allow(Digest::SHA256).to receive(:file).with(temp_file).and_return(mock_digest)
+      end
 
       it_behaves_like 'windows Datadog Agent', :msi
       # remote_file source gets converted to an array, so we need to do
@@ -321,6 +360,14 @@ def set_env_var(name, value)
       end
 
       temp_file = ::File.join('C:/chef/cache', 'ddagent-cli.msi')
+      mock_digest = Digest::SHA256.new
+
+      before do
+        allow(File).to receive(:open).and_call_original
+        allow(File).to receive(:open).with(temp_file).and_return('foo')
+        allow(Digest::SHA256).to receive(:file).and_call_original
+        allow(Digest::SHA256).to receive(:file).with(temp_file).and_return(mock_digest)
+      end
 
       it_behaves_like 'windows Datadog Agent', :msi
       # remote_file source gets converted to an array, so we need to do
@@ -426,6 +473,14 @@ def set_env_var(name, value)
       end
 
       temp_file = ::File.join('C:/chef/cache', 'ddagent-cli.msi')
+      mock_digest = Digest::SHA256.new
+
+      before do
+        allow(File).to receive(:open).and_call_original
+        allow(File).to receive(:open).with(temp_file).and_return('foo')
+        allow(Digest::SHA256).to receive(:file).and_call_original
+        allow(Digest::SHA256).to receive(:file).with(temp_file).and_return(mock_digest)
+      end
 
       it_behaves_like 'windows Datadog Agent v5', :msi
       # remote_file source gets converted to an array, so we need to do
@@ -1001,13 +1056,24 @@ def set_env_var(name, value)
             set_env_var('ProgramData', 'C:\ProgramData')
             ChefSpec::SoloRunner.new(
               platform: 'windows',
-              version: '2012R2'
+              version: '2012R2',
+              file_cache_path: 'C:/chef/cache'
             ) do |node|
               node.name 'chef-nodename' # expected to be used as the hostname in `datadog.yaml`
               node.normal['datadog'] = { 'api_key' => 'somethingnotnil', 'agent6' => true }
             end.converge described_recipe
           end
 
+          temp_file = ::File.join('C:/chef/cache', 'ddagent-cli.msi')
+          mock_digest = Digest::SHA256.new
+
+          before do
+            allow(File).to receive(:open).and_call_original
+            allow(File).to receive(:open).with(temp_file).and_return('foo')
+            allow(Digest::SHA256).to receive(:file).and_call_original
+            allow(Digest::SHA256).to receive(:file).with(temp_file).and_return(mock_digest)
+          end
+
           it 'is created' do
             expect(chef_run).to create_template('C:\ProgramData/Datadog/datadog.yaml')
           end
@@ -1067,6 +1133,16 @@ def set_env_var(name, value)
           end.converge described_recipe
         end
 
+        temp_file = ::File.join('C:/chef/cache', 'ddagent-cli.msi')
+        mock_digest = Digest::SHA256.new
+
+        before do
+          allow(File).to receive(:open).and_call_original
+          allow(File).to receive(:open).with(temp_file).and_return('foo')
+          allow(Digest::SHA256).to receive(:file).and_call_original
+          allow(Digest::SHA256).to receive(:file).with(temp_file).and_return(mock_digest)
+        end
+
         it 'is created' do
           expect(chef_run).to create_template('/etc/datadog-agent/datadog.yaml')
         end
@@ -1128,6 +1204,14 @@ def set_env_var(name, value)
         end
 
         temp_file = ::File.join('C:/chef/cache', 'ddagent-cli.msi')
+        mock_digest = Digest::SHA256.new
+
+        before do
+          allow(File).to receive(:open).and_call_original
+          allow(File).to receive(:open).with(temp_file).and_return('foo')
+          allow(Digest::SHA256).to receive(:file).and_call_original
+          allow(Digest::SHA256).to receive(:file).with(temp_file).and_return(mock_digest)
+        end
 
         it 'installs Datadog Agent' do
           expect(chef_run).to install_windows_package('Datadog Agent').with(installer_type: :msi)

diff --git a/spec/shared_examples.rb b/spec/shared_examples.rb
@@ -130,7 +130,11 @@
   end
 
   it 'notifies the removal of the Datadog Agent when a remote file is downloaded' do
-    expect(chef_run.remote_file(agent_installer)).to notify('package[Datadog Agent removal]').to(:remove)
+    expect(chef_run.powershell_script('datadog_6.14.x_fix')).to notify('package[Datadog Agent removal]').to(:remove)
+  end
+
+  it 'notifies the execution of the powershell fix when a remote file is downloaded' do
+    expect(chef_run.remote_file(agent_installer)).to notify('powershell_script[datadog_6.14.x_fix]').to(:run)
   end
 
   it 'installs Datadog Agent' do
@@ -153,7 +157,11 @@
   end
 
   it 'notifies the removal of the Datadog Agent when a remote file is downloaded' do
-    expect(chef_run.remote_file(agent_installer)).to notify('package[Datadog Agent removal]').to(:remove)
+    expect(chef_run.powershell_script('datadog_6.14.x_fix')).to notify('package[Datadog Agent removal]').to(:remove)
+  end
+
+  it 'notifies the execution of the powershell fix when a remote file is downloaded' do
+    expect(chef_run.remote_file(agent_installer)).to notify('powershell_script[datadog_6.14.x_fix]').to(:run)
   end
 
   it 'installs Datadog Agent' do