From 092885e447a88a9b81af200aff0e857273034584 Mon Sep 17 00:00:00 2001 From: Arnaud Lachaume Date: Mon, 31 May 2021 11:06:36 +0200 Subject: [PATCH] prepare v0.12.rc10 --- CHANGELOG.md | 7 +- docs/BATCH_JOBS.md | 34 +++- examples/rails/Gemfile.lock | 158 ++++++++++--------- gemfiles/google_cloud_tasks_1.0.gemfile.lock | 8 +- gemfiles/google_cloud_tasks_1.1.gemfile.lock | 8 +- gemfiles/google_cloud_tasks_1.2.gemfile.lock | 8 +- gemfiles/google_cloud_tasks_1.3.gemfile.lock | 8 +- gemfiles/rails_5.2.gemfile.lock | 10 +- gemfiles/rails_6.0.gemfile.lock | 10 +- lib/cloudtasker/version.rb | 2 +- 10 files changed, 151 insertions(+), 102 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 880414b8..e2f316c2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,8 +1,8 @@ # Changelog -## Latest RC [v0.12.rc8](https://github.com/keypup-io/cloudtasker/tree/v0.12.rc8) (2021-04-06) +## Latest RC [v0.12.rc10](https://github.com/keypup-io/cloudtasker/tree/v0.12.rc10) (2021-05-31) -[Full Changelog](https://github.com/keypup-io/cloudtasker/compare/v0.11.0...v0.12.rc8) +[Full Changelog](https://github.com/keypup-io/cloudtasker/compare/v0.11.0...v0.12.rc10) **Improvements:** - ActiveJob: do not double log errors (ActiveJob has its own error logging) @@ -10,11 +10,14 @@ - Batch state: use native Redis hashes to store batch state instead of a serialized hash in a string key - Batch progress: restrict calculation to direct children by default. Allow depth to be specified. Calculating progress using all tree jobs created significant delays on large batches. - Batch redis usage: cleanup batches as they get completed or become dead to avoid excessive redis usage with large batches. +- Batch expansion: Inject `parent_batch` in jobs. Can be used to expand the parent batch the job is in. - Configuration: allow configuration of Cloud Tasks `dispatch deadline` at global and worker level - Cron jobs: Use Redis Sets instead of key pattern matching for resource listing - Error logging: Use worker logger so as to include context (job args etc.) - Error logging: Do not log exception and stack trace separately, combine them instead. - Local server: Use Redis Sets instead of key pattern matching for resource listing +- Local server: Guard against nil tasks to prevent job daemon failures +- Performance: remove use of redis locks and rely on atomic transactions instead for Batch and Unique Job. - Worker: raise DeadWorkerError instead of MissingWorkerArgumentsError when arguments are missing. This is more consistent with what middlewares expect. - Worker redis usage: delete redis payload storage once the job is successful or dead instead of expiring the key. diff --git a/docs/BATCH_JOBS.md b/docs/BATCH_JOBS.md index 228f3934..48b15f7d 100644 --- a/docs/BATCH_JOBS.md +++ b/docs/BATCH_JOBS.md @@ -18,7 +18,7 @@ Cloudtasker.configure do |config| end ``` -## Example +## Example: Creating a new batch The following example defines a worker that adds itself to the batch with different arguments then monitors the success of the batch. @@ -47,6 +47,38 @@ class BatchWorker end ``` +## Example: Expanding the parent batch +**Note**: `parent_batch` is available since `0.12.rc10` + +```ruby +# All the jobs will be attached to the top parent batch. +class BatchWorker + include Cloudtasker::Worker + + def perform(level, instance) + # Use existing parent_batch or create a new one + current_batch = parent_batch || batch + + 3.times { |n| current_batch.add(self.class, level + 1, n) } if level < 2 + end + + # Invoked when any descendant (e.g. sub-sub job) is complete + def on_batch_node_complete(child) + logger.info("Direct or Indirect child complete: #{child.job_id}") + end + + # Invoked when a direct descendant is complete + def on_child_complete(child) + logger.info("Direct child complete: #{child.job_id}") + end + + # Invoked when all chidren have finished + def on_batch_complete + Rails.logger.info("Batch complete") + end +end +``` + ## Available callbacks The following callbacks are available on your workers to track the progress of the batch: diff --git a/examples/rails/Gemfile.lock b/examples/rails/Gemfile.lock index 46c34feb..c8b3fb38 100644 --- a/examples/rails/Gemfile.lock +++ b/examples/rails/Gemfile.lock @@ -1,7 +1,7 @@ PATH remote: ../.. specs: - cloudtasker (0.12.rc7) + cloudtasker (0.12.rc10) activesupport connection_pool fugit @@ -13,56 +13,56 @@ PATH GEM remote: https://rubygems.org/ specs: - actioncable (6.0.3.5) - actionpack (= 6.0.3.5) + actioncable (6.0.3.7) + actionpack (= 6.0.3.7) nio4r (~> 2.0) websocket-driver (>= 0.6.1) - actionmailbox (6.0.3.5) - actionpack (= 6.0.3.5) - activejob (= 6.0.3.5) - activerecord (= 6.0.3.5) - activestorage (= 6.0.3.5) - activesupport (= 6.0.3.5) + actionmailbox (6.0.3.7) + actionpack (= 6.0.3.7) + activejob (= 6.0.3.7) + activerecord (= 6.0.3.7) + activestorage (= 6.0.3.7) + activesupport (= 6.0.3.7) mail (>= 2.7.1) - actionmailer (6.0.3.5) - actionpack (= 6.0.3.5) - actionview (= 6.0.3.5) - activejob (= 6.0.3.5) + actionmailer (6.0.3.7) + actionpack (= 6.0.3.7) + actionview (= 6.0.3.7) + activejob (= 6.0.3.7) mail (~> 2.5, >= 2.5.4) rails-dom-testing (~> 2.0) - actionpack (6.0.3.5) - actionview (= 6.0.3.5) - activesupport (= 6.0.3.5) + actionpack (6.0.3.7) + actionview (= 6.0.3.7) + activesupport (= 6.0.3.7) rack (~> 2.0, >= 2.0.8) rack-test (>= 0.6.3) rails-dom-testing (~> 2.0) rails-html-sanitizer (~> 1.0, >= 1.2.0) - actiontext (6.0.3.5) - actionpack (= 6.0.3.5) - activerecord (= 6.0.3.5) - activestorage (= 6.0.3.5) - activesupport (= 6.0.3.5) + actiontext (6.0.3.7) + actionpack (= 6.0.3.7) + activerecord (= 6.0.3.7) + activestorage (= 6.0.3.7) + activesupport (= 6.0.3.7) nokogiri (>= 1.8.5) - actionview (6.0.3.5) - activesupport (= 6.0.3.5) + actionview (6.0.3.7) + activesupport (= 6.0.3.7) builder (~> 3.1) erubi (~> 1.4) rails-dom-testing (~> 2.0) rails-html-sanitizer (~> 1.1, >= 1.2.0) - activejob (6.0.3.5) - activesupport (= 6.0.3.5) + activejob (6.0.3.7) + activesupport (= 6.0.3.7) globalid (>= 0.3.6) - activemodel (6.0.3.5) - activesupport (= 6.0.3.5) - activerecord (6.0.3.5) - activemodel (= 6.0.3.5) - activesupport (= 6.0.3.5) - activestorage (6.0.3.5) - actionpack (= 6.0.3.5) - activejob (= 6.0.3.5) - activerecord (= 6.0.3.5) - marcel (~> 0.3.1) - activesupport (6.0.3.5) + activemodel (6.0.3.7) + activesupport (= 6.0.3.7) + activerecord (6.0.3.7) + activemodel (= 6.0.3.7) + activesupport (= 6.0.3.7) + activestorage (6.0.3.7) + actionpack (= 6.0.3.7) + activejob (= 6.0.3.7) + activerecord (= 6.0.3.7) + marcel (~> 1.0.0) + activesupport (6.0.3.7) concurrent-ruby (~> 1.0, >= 1.0.2) i18n (>= 0.7, < 2) minitest (~> 5.1) @@ -72,17 +72,25 @@ GEM public_suffix (>= 2.0.2, < 5.0) builder (3.2.4) concurrent-ruby (1.1.8) - connection_pool (2.2.3) + connection_pool (2.2.5) crass (1.0.6) erubi (1.10.0) et-orbi (1.2.4) tzinfo - faraday (1.3.0) + faraday (1.4.2) + faraday-em_http (~> 1.0) + faraday-em_synchrony (~> 1.0) + faraday-excon (~> 1.1) faraday-net_http (~> 1.0) + faraday-net_http_persistent (~> 1.1) multipart-post (>= 1.2, < 3) - ruby2_keywords + ruby2_keywords (>= 0.0.4) + faraday-em_http (1.0.0) + faraday-em_synchrony (1.0.0) + faraday-excon (1.1.0) faraday-net_http (1.0.1) - fugit (1.4.4) + faraday-net_http_persistent (1.1.0) + fugit (1.4.5) et-orbi (~> 1.1, >= 1.1.8) raabro (~> 1.4) globalid (0.4.2) @@ -98,49 +106,53 @@ GEM googleauth (~> 0.9) grpc (~> 1.24) rly (~> 0.2.3) - google-protobuf (3.15.7) + google-protobuf (3.17.1) googleapis-common-protos (1.3.11) google-protobuf (~> 3.14) googleapis-common-protos-types (>= 1.0.6, < 2.0) grpc (~> 1.27) googleapis-common-protos-types (1.0.6) google-protobuf (~> 3.14) - googleauth (0.16.1) + googleauth (0.16.2) faraday (>= 0.17.3, < 2.0) jwt (>= 1.4, < 3.0) memoist (~> 0.16) multi_json (~> 1.11) os (>= 0.9, < 2.0) signet (~> 0.14) - grpc (1.36.0) - google-protobuf (~> 3.14) + grpc (1.38.0) + google-protobuf (~> 3.15) googleapis-common-protos-types (~> 1.0) grpc-google-iam-v1 (0.6.11) google-protobuf (~> 3.14) googleapis-common-protos (>= 1.3.11, < 2.0) grpc (~> 1.27) - i18n (1.8.9) + i18n (1.8.10) concurrent-ruby (~> 1.0) - jwt (2.2.2) - loofah (2.9.0) + io-wait (0.1.0) + jwt (2.2.3) + loofah (2.9.1) crass (~> 1.0.2) nokogiri (>= 1.5.9) mail (2.7.1) mini_mime (>= 0.1.1) - marcel (0.3.3) - mimemagic (~> 0.3.2) + marcel (1.0.1) memoist (0.16.2) method_source (1.0.0) - mimemagic (0.3.10) - nokogiri (~> 1) - rake - mini_mime (1.0.2) - mini_portile2 (2.5.1) + mini_mime (1.1.0) + mini_portile2 (2.5.2) + net-ftp (~> 0.1) minitest (5.14.4) multi_json (1.15.0) multipart-post (2.1.1) + net-ftp (0.1.2) + net-protocol + time + net-protocol (0.1.1) + io-wait + timeout nio4r (2.5.7) - nokogiri (1.11.5) + nokogiri (1.11.6) mini_portile2 (~> 2.5.0) racc (~> 1.4) os (1.1.1) @@ -150,29 +162,29 @@ GEM rack (2.2.3) rack-test (1.1.0) rack (>= 1.0, < 3) - rails (6.0.3.5) - actioncable (= 6.0.3.5) - actionmailbox (= 6.0.3.5) - actionmailer (= 6.0.3.5) - actionpack (= 6.0.3.5) - actiontext (= 6.0.3.5) - actionview (= 6.0.3.5) - activejob (= 6.0.3.5) - activemodel (= 6.0.3.5) - activerecord (= 6.0.3.5) - activestorage (= 6.0.3.5) - activesupport (= 6.0.3.5) + rails (6.0.3.7) + actioncable (= 6.0.3.7) + actionmailbox (= 6.0.3.7) + actionmailer (= 6.0.3.7) + actionpack (= 6.0.3.7) + actiontext (= 6.0.3.7) + actionview (= 6.0.3.7) + activejob (= 6.0.3.7) + activemodel (= 6.0.3.7) + activerecord (= 6.0.3.7) + activestorage (= 6.0.3.7) + activesupport (= 6.0.3.7) bundler (>= 1.3.0) - railties (= 6.0.3.5) + railties (= 6.0.3.7) sprockets-rails (>= 2.0.0) rails-dom-testing (2.0.3) activesupport (>= 4.2.0) nokogiri (>= 1.6) rails-html-sanitizer (1.3.0) loofah (~> 2.3) - railties (6.0.3.5) - actionpack (= 6.0.3.5) - activesupport (= 6.0.3.5) + railties (6.0.3.7) + actionpack (= 6.0.3.7) + activesupport (= 6.0.3.7) method_source rake (>= 0.8.7) thor (>= 0.20.3, < 2.0) @@ -196,9 +208,11 @@ GEM sqlite3 (1.4.2) thor (1.1.0) thread_safe (0.3.6) + time (0.1.0) + timeout (0.1.1) tzinfo (1.2.9) thread_safe (~> 0.1) - websocket-driver (0.7.3) + websocket-driver (0.7.4) websocket-extensions (>= 0.1.0) websocket-extensions (0.1.5) zeitwerk (2.4.2) diff --git a/gemfiles/google_cloud_tasks_1.0.gemfile.lock b/gemfiles/google_cloud_tasks_1.0.gemfile.lock index 1c0aaf50..ad4f2ba1 100644 --- a/gemfiles/google_cloud_tasks_1.0.gemfile.lock +++ b/gemfiles/google_cloud_tasks_1.0.gemfile.lock @@ -1,7 +1,7 @@ PATH remote: .. specs: - cloudtasker (0.12.rc9) + cloudtasker (0.12.rc10) activesupport connection_pool fugit @@ -121,7 +121,7 @@ GEM faraday-net_http (1.0.1) faraday-net_http_persistent (1.1.0) fiber-local (1.0.0) - fugit (1.4.4) + fugit (1.4.5) et-orbi (~> 1.1, >= 1.1.8) raabro (~> 1.4) github_changelog_generator (1.16.1) @@ -146,7 +146,7 @@ GEM googleauth (~> 0.9) grpc (~> 1.24) rly (~> 0.2.3) - google-protobuf (3.15.8) + google-protobuf (3.17.1) googleapis-common-protos (1.3.11) google-protobuf (~> 3.14) googleapis-common-protos-types (>= 1.0.6, < 2.0) @@ -160,7 +160,7 @@ GEM multi_json (~> 1.11) os (>= 0.9, < 2.0) signet (~> 0.14) - grpc (1.37.0) + grpc (1.38.0) google-protobuf (~> 3.15) googleapis-common-protos-types (~> 1.0) grpc-google-iam-v1 (0.6.11) diff --git a/gemfiles/google_cloud_tasks_1.1.gemfile.lock b/gemfiles/google_cloud_tasks_1.1.gemfile.lock index a96ca240..02daf565 100644 --- a/gemfiles/google_cloud_tasks_1.1.gemfile.lock +++ b/gemfiles/google_cloud_tasks_1.1.gemfile.lock @@ -1,7 +1,7 @@ PATH remote: .. specs: - cloudtasker (0.12.rc9) + cloudtasker (0.12.rc10) activesupport connection_pool fugit @@ -121,7 +121,7 @@ GEM faraday-net_http (1.0.1) faraday-net_http_persistent (1.1.0) fiber-local (1.0.0) - fugit (1.4.4) + fugit (1.4.5) et-orbi (~> 1.1, >= 1.1.8) raabro (~> 1.4) github_changelog_generator (1.16.1) @@ -146,7 +146,7 @@ GEM googleauth (~> 0.9) grpc (~> 1.24) rly (~> 0.2.3) - google-protobuf (3.15.8) + google-protobuf (3.17.1) googleapis-common-protos (1.3.11) google-protobuf (~> 3.14) googleapis-common-protos-types (>= 1.0.6, < 2.0) @@ -160,7 +160,7 @@ GEM multi_json (~> 1.11) os (>= 0.9, < 2.0) signet (~> 0.14) - grpc (1.37.0) + grpc (1.38.0) google-protobuf (~> 3.15) googleapis-common-protos-types (~> 1.0) grpc-google-iam-v1 (0.6.11) diff --git a/gemfiles/google_cloud_tasks_1.2.gemfile.lock b/gemfiles/google_cloud_tasks_1.2.gemfile.lock index 8f33f49f..92a5f8bf 100644 --- a/gemfiles/google_cloud_tasks_1.2.gemfile.lock +++ b/gemfiles/google_cloud_tasks_1.2.gemfile.lock @@ -1,7 +1,7 @@ PATH remote: .. specs: - cloudtasker (0.12.rc9) + cloudtasker (0.12.rc10) activesupport connection_pool fugit @@ -121,7 +121,7 @@ GEM faraday-net_http (1.0.1) faraday-net_http_persistent (1.1.0) fiber-local (1.0.0) - fugit (1.4.4) + fugit (1.4.5) et-orbi (~> 1.1, >= 1.1.8) raabro (~> 1.4) github_changelog_generator (1.16.1) @@ -146,7 +146,7 @@ GEM googleauth (~> 0.9) grpc (~> 1.24) rly (~> 0.2.3) - google-protobuf (3.15.8) + google-protobuf (3.17.1) googleapis-common-protos (1.3.11) google-protobuf (~> 3.14) googleapis-common-protos-types (>= 1.0.6, < 2.0) @@ -160,7 +160,7 @@ GEM multi_json (~> 1.11) os (>= 0.9, < 2.0) signet (~> 0.14) - grpc (1.37.0) + grpc (1.38.0) google-protobuf (~> 3.15) googleapis-common-protos-types (~> 1.0) grpc-google-iam-v1 (0.6.11) diff --git a/gemfiles/google_cloud_tasks_1.3.gemfile.lock b/gemfiles/google_cloud_tasks_1.3.gemfile.lock index c9e92dce..1c1c537a 100644 --- a/gemfiles/google_cloud_tasks_1.3.gemfile.lock +++ b/gemfiles/google_cloud_tasks_1.3.gemfile.lock @@ -1,7 +1,7 @@ PATH remote: .. specs: - cloudtasker (0.12.rc9) + cloudtasker (0.12.rc10) activesupport connection_pool fugit @@ -121,7 +121,7 @@ GEM faraday-net_http (1.0.1) faraday-net_http_persistent (1.1.0) fiber-local (1.0.0) - fugit (1.4.4) + fugit (1.4.5) et-orbi (~> 1.1, >= 1.1.8) raabro (~> 1.4) github_changelog_generator (1.16.1) @@ -147,7 +147,7 @@ GEM googleauth (~> 0.9) grpc (~> 1.24) rly (~> 0.2.3) - google-protobuf (3.15.8) + google-protobuf (3.17.1) googleapis-common-protos (1.3.11) google-protobuf (~> 3.14) googleapis-common-protos-types (>= 1.0.6, < 2.0) @@ -161,7 +161,7 @@ GEM multi_json (~> 1.11) os (>= 0.9, < 2.0) signet (~> 0.14) - grpc (1.37.0) + grpc (1.38.0) google-protobuf (~> 3.15) googleapis-common-protos-types (~> 1.0) grpc-google-iam-v1 (0.6.11) diff --git a/gemfiles/rails_5.2.gemfile.lock b/gemfiles/rails_5.2.gemfile.lock index a5cdaaf6..712baaec 100644 --- a/gemfiles/rails_5.2.gemfile.lock +++ b/gemfiles/rails_5.2.gemfile.lock @@ -1,7 +1,7 @@ PATH remote: .. specs: - cloudtasker (0.12.rc9) + cloudtasker (0.12.rc10) activesupport connection_pool fugit @@ -104,7 +104,7 @@ GEM faraday-net_http (1.0.1) faraday-net_http_persistent (1.1.0) fiber-local (1.0.0) - fugit (1.4.4) + fugit (1.4.5) et-orbi (~> 1.1, >= 1.1.8) raabro (~> 1.4) github_changelog_generator (1.16.1) @@ -130,21 +130,21 @@ GEM googleauth (~> 0.9) grpc (~> 1.24) rly (~> 0.2.3) - google-protobuf (3.15.8) + google-protobuf (3.17.1) googleapis-common-protos (1.3.11) google-protobuf (~> 3.14) googleapis-common-protos-types (>= 1.0.6, < 2.0) grpc (~> 1.27) googleapis-common-protos-types (1.0.6) google-protobuf (~> 3.14) - googleauth (0.16.1) + googleauth (0.16.2) faraday (>= 0.17.3, < 2.0) jwt (>= 1.4, < 3.0) memoist (~> 0.16) multi_json (~> 1.11) os (>= 0.9, < 2.0) signet (~> 0.14) - grpc (1.37.0) + grpc (1.38.0) google-protobuf (~> 3.15) googleapis-common-protos-types (~> 1.0) grpc-google-iam-v1 (0.6.11) diff --git a/gemfiles/rails_6.0.gemfile.lock b/gemfiles/rails_6.0.gemfile.lock index 70f65dd4..7bf0be6c 100644 --- a/gemfiles/rails_6.0.gemfile.lock +++ b/gemfiles/rails_6.0.gemfile.lock @@ -1,7 +1,7 @@ PATH remote: .. specs: - cloudtasker (0.12.rc9) + cloudtasker (0.12.rc10) activesupport connection_pool fugit @@ -117,7 +117,7 @@ GEM faraday-net_http (1.0.1) faraday-net_http_persistent (1.1.0) fiber-local (1.0.0) - fugit (1.4.4) + fugit (1.4.5) et-orbi (~> 1.1, >= 1.1.8) raabro (~> 1.4) github_changelog_generator (1.16.1) @@ -143,21 +143,21 @@ GEM googleauth (~> 0.9) grpc (~> 1.24) rly (~> 0.2.3) - google-protobuf (3.15.8) + google-protobuf (3.17.1) googleapis-common-protos (1.3.11) google-protobuf (~> 3.14) googleapis-common-protos-types (>= 1.0.6, < 2.0) grpc (~> 1.27) googleapis-common-protos-types (1.0.6) google-protobuf (~> 3.14) - googleauth (0.16.1) + googleauth (0.16.2) faraday (>= 0.17.3, < 2.0) jwt (>= 1.4, < 3.0) memoist (~> 0.16) multi_json (~> 1.11) os (>= 0.9, < 2.0) signet (~> 0.14) - grpc (1.37.0) + grpc (1.38.0) google-protobuf (~> 3.15) googleapis-common-protos-types (~> 1.0) grpc-google-iam-v1 (0.6.11) diff --git a/lib/cloudtasker/version.rb b/lib/cloudtasker/version.rb index ffd2548d..4f9b3f89 100644 --- a/lib/cloudtasker/version.rb +++ b/lib/cloudtasker/version.rb @@ -1,5 +1,5 @@ # frozen_string_literal: true module Cloudtasker - VERSION = '0.12.rc9' + VERSION = '0.12.rc10' end