prepare v0.12.rc10

alachaum · May 31, 2021 · 092885e · 092885e
1 parent 76ea4c9
commit 092885e
Show file tree

Hide file tree

Showing 10 changed files with 151 additions and 102 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,20 +1,23 @@
 # Changelog
 
-## Latest RC [v0.12.rc8](https://github.com/keypup-io/cloudtasker/tree/v0.12.rc8) (2021-04-06)
+## Latest RC [v0.12.rc10](https://github.com/keypup-io/cloudtasker/tree/v0.12.rc10) (2021-05-31)
 
-[Full Changelog](https://github.com/keypup-io/cloudtasker/compare/v0.11.0...v0.12.rc8)
+[Full Changelog](https://github.com/keypup-io/cloudtasker/compare/v0.11.0...v0.12.rc10)
 
 **Improvements:**
 - ActiveJob: do not double log errors (ActiveJob has its own error logging)
 - Batch callbacks: Retry jobs when completion callback fails
 - Batch state: use native Redis hashes to store batch state instead of a serialized hash in a string key
 - Batch progress: restrict calculation to direct children by default. Allow depth to be specified. Calculating progress using all tree jobs created significant delays on large batches.
 - Batch redis usage: cleanup batches as they get completed or become dead to avoid excessive redis usage with large batches.
+- Batch expansion: Inject `parent_batch` in jobs. Can be used to expand the parent batch the job is in.
 - Configuration: allow configuration of Cloud Tasks `dispatch deadline` at global and worker level
 - Cron jobs: Use Redis Sets instead of key pattern matching for resource listing
 - Error logging: Use worker logger so as to include context (job args etc.)
 - Error logging: Do not log exception and stack trace separately, combine them instead.
 - Local server: Use Redis Sets instead of key pattern matching for resource listing
+- Local server: Guard against nil tasks to prevent job daemon failures
+- Performance: remove use of redis locks and rely on atomic transactions instead for Batch and Unique Job.
 - Worker: raise DeadWorkerError instead of MissingWorkerArgumentsError when arguments are missing. This is more consistent with what middlewares expect.
 - Worker redis usage: delete redis payload storage once the job is successful or dead instead of expiring the key.
 

diff --git a/docs/BATCH_JOBS.md b/docs/BATCH_JOBS.md
@@ -18,7 +18,7 @@ Cloudtasker.configure do |config|
 end
 ```
 
-## Example
+## Example: Creating a new batch
 
 The following example defines a worker that adds itself to the batch with different arguments then monitors the success of the batch.
 
@@ -47,6 +47,38 @@ class BatchWorker
 end
 ```
 
+## Example: Expanding the parent batch
+**Note**: `parent_batch` is available since `0.12.rc10`
+
+```ruby
+# All the jobs will be attached to the top parent batch.
+class BatchWorker
+  include Cloudtasker::Worker
+
+  def perform(level, instance)
+    # Use existing parent_batch or create a new one
+    current_batch = parent_batch || batch
+
+    3.times { |n| current_batch.add(self.class, level + 1, n) } if level < 2
+  end
+
+  # Invoked when any descendant (e.g. sub-sub job) is complete
+  def on_batch_node_complete(child)
+    logger.info("Direct or Indirect child complete: #{child.job_id}")
+  end
+
+  # Invoked when a direct descendant is complete
+  def on_child_complete(child)
+    logger.info("Direct child complete: #{child.job_id}")
+  end
+
+  # Invoked when all chidren have finished
+  def on_batch_complete
+    Rails.logger.info("Batch complete")
+  end
+end
+```
+
 ## Available callbacks
 
 The following callbacks are available on your workers to track the progress of the batch:

diff --git a/examples/rails/Gemfile.lock b/examples/rails/Gemfile.lock
@@ -1,7 +1,7 @@
 PATH
   remote: ../..
   specs:
-    cloudtasker (0.12.rc7)
+    cloudtasker (0.12.rc10)
       activesupport
       connection_pool
       fugit
@@ -13,56 +13,56 @@ PATH
 GEM
   remote: https://rubygems.org/
   specs:
-    actioncable (6.0.3.5)
-      actionpack (= 6.0.3.5)
+    actioncable (6.0.3.7)
+      actionpack (= 6.0.3.7)
       nio4r (~> 2.0)
       websocket-driver (>= 0.6.1)
-    actionmailbox (6.0.3.5)
-      actionpack (= 6.0.3.5)
-      activejob (= 6.0.3.5)
-      activerecord (= 6.0.3.5)
-      activestorage (= 6.0.3.5)
-      activesupport (= 6.0.3.5)
+    actionmailbox (6.0.3.7)
+      actionpack (= 6.0.3.7)
+      activejob (= 6.0.3.7)
+      activerecord (= 6.0.3.7)
+      activestorage (= 6.0.3.7)
+      activesupport (= 6.0.3.7)
       mail (>= 2.7.1)
-    actionmailer (6.0.3.5)
-      actionpack (= 6.0.3.5)
-      actionview (= 6.0.3.5)
-      activejob (= 6.0.3.5)
+    actionmailer (6.0.3.7)
+      actionpack (= 6.0.3.7)
+      actionview (= 6.0.3.7)
+      activejob (= 6.0.3.7)
       mail (~> 2.5, >= 2.5.4)
       rails-dom-testing (~> 2.0)
-    actionpack (6.0.3.5)
-      actionview (= 6.0.3.5)
-      activesupport (= 6.0.3.5)
+    actionpack (6.0.3.7)
+      actionview (= 6.0.3.7)
+      activesupport (= 6.0.3.7)
       rack (~> 2.0, >= 2.0.8)
       rack-test (>= 0.6.3)
       rails-dom-testing (~> 2.0)
       rails-html-sanitizer (~> 1.0, >= 1.2.0)
-    actiontext (6.0.3.5)
-      actionpack (= 6.0.3.5)
-      activerecord (= 6.0.3.5)
-      activestorage (= 6.0.3.5)
-      activesupport (= 6.0.3.5)
+    actiontext (6.0.3.7)
+      actionpack (= 6.0.3.7)
+      activerecord (= 6.0.3.7)
+      activestorage (= 6.0.3.7)
+      activesupport (= 6.0.3.7)
       nokogiri (>= 1.8.5)
-    actionview (6.0.3.5)
-      activesupport (= 6.0.3.5)
+    actionview (6.0.3.7)
+      activesupport (= 6.0.3.7)
       builder (~> 3.1)
       erubi (~> 1.4)
       rails-dom-testing (~> 2.0)
       rails-html-sanitizer (~> 1.1, >= 1.2.0)
-    activejob (6.0.3.5)
-      activesupport (= 6.0.3.5)
+    activejob (6.0.3.7)
+      activesupport (= 6.0.3.7)
       globalid (>= 0.3.6)
-    activemodel (6.0.3.5)
-      activesupport (= 6.0.3.5)
-    activerecord (6.0.3.5)
-      activemodel (= 6.0.3.5)
-      activesupport (= 6.0.3.5)
-    activestorage (6.0.3.5)
-      actionpack (= 6.0.3.5)
-      activejob (= 6.0.3.5)
-      activerecord (= 6.0.3.5)
-      marcel (~> 0.3.1)
-    activesupport (6.0.3.5)
+    activemodel (6.0.3.7)
+      activesupport (= 6.0.3.7)
+    activerecord (6.0.3.7)
+      activemodel (= 6.0.3.7)
+      activesupport (= 6.0.3.7)
+    activestorage (6.0.3.7)
+      actionpack (= 6.0.3.7)
+      activejob (= 6.0.3.7)
+      activerecord (= 6.0.3.7)
+      marcel (~> 1.0.0)
+    activesupport (6.0.3.7)
       concurrent-ruby (~> 1.0, >= 1.0.2)
       i18n (>= 0.7, < 2)
       minitest (~> 5.1)
@@ -72,17 +72,25 @@ GEM
       public_suffix (>= 2.0.2, < 5.0)
     builder (3.2.4)
     concurrent-ruby (1.1.8)
-    connection_pool (2.2.3)
+    connection_pool (2.2.5)
     crass (1.0.6)
     erubi (1.10.0)
     et-orbi (1.2.4)
       tzinfo
-    faraday (1.3.0)
+    faraday (1.4.2)
+      faraday-em_http (~> 1.0)
+      faraday-em_synchrony (~> 1.0)
+      faraday-excon (~> 1.1)
       faraday-net_http (~> 1.0)
+      faraday-net_http_persistent (~> 1.1)
       multipart-post (>= 1.2, < 3)
-      ruby2_keywords
+      ruby2_keywords (>= 0.0.4)
+    faraday-em_http (1.0.0)
+    faraday-em_synchrony (1.0.0)
+    faraday-excon (1.1.0)
     faraday-net_http (1.0.1)
-    fugit (1.4.4)
+    faraday-net_http_persistent (1.1.0)
+    fugit (1.4.5)
       et-orbi (~> 1.1, >= 1.1.8)
       raabro (~> 1.4)
     globalid (0.4.2)
@@ -98,49 +106,53 @@ GEM
       googleauth (~> 0.9)
       grpc (~> 1.24)
       rly (~> 0.2.3)
-    google-protobuf (3.15.7)
+    google-protobuf (3.17.1)
     googleapis-common-protos (1.3.11)
       google-protobuf (~> 3.14)
       googleapis-common-protos-types (>= 1.0.6, < 2.0)
       grpc (~> 1.27)
     googleapis-common-protos-types (1.0.6)
       google-protobuf (~> 3.14)
-    googleauth (0.16.1)
+    googleauth (0.16.2)
       faraday (>= 0.17.3, < 2.0)
       jwt (>= 1.4, < 3.0)
       memoist (~> 0.16)
       multi_json (~> 1.11)
       os (>= 0.9, < 2.0)
       signet (~> 0.14)
-    grpc (1.36.0)
-      google-protobuf (~> 3.14)
+    grpc (1.38.0)
+      google-protobuf (~> 3.15)
       googleapis-common-protos-types (~> 1.0)
     grpc-google-iam-v1 (0.6.11)
       google-protobuf (~> 3.14)
       googleapis-common-protos (>= 1.3.11, < 2.0)
       grpc (~> 1.27)
-    i18n (1.8.9)
+    i18n (1.8.10)
       concurrent-ruby (~> 1.0)
-    jwt (2.2.2)
-    loofah (2.9.0)
+    io-wait (0.1.0)
+    jwt (2.2.3)
+    loofah (2.9.1)
       crass (~> 1.0.2)
       nokogiri (>= 1.5.9)
     mail (2.7.1)
       mini_mime (>= 0.1.1)
-    marcel (0.3.3)
-      mimemagic (~> 0.3.2)
+    marcel (1.0.1)
     memoist (0.16.2)
     method_source (1.0.0)
-    mimemagic (0.3.10)
-      nokogiri (~> 1)
-      rake
-    mini_mime (1.0.2)
-    mini_portile2 (2.5.1)
+    mini_mime (1.1.0)
+    mini_portile2 (2.5.2)
+      net-ftp (~> 0.1)
     minitest (5.14.4)
     multi_json (1.15.0)
     multipart-post (2.1.1)
+    net-ftp (0.1.2)
+      net-protocol
+      time
+    net-protocol (0.1.1)
+      io-wait
+      timeout
     nio4r (2.5.7)
-    nokogiri (1.11.5)
+    nokogiri (1.11.6)
       mini_portile2 (~> 2.5.0)
       racc (~> 1.4)
     os (1.1.1)
@@ -150,29 +162,29 @@ GEM
     rack (2.2.3)
     rack-test (1.1.0)
       rack (>= 1.0, < 3)
-    rails (6.0.3.5)
-      actioncable (= 6.0.3.5)
-      actionmailbox (= 6.0.3.5)
-      actionmailer (= 6.0.3.5)
-      actionpack (= 6.0.3.5)
-      actiontext (= 6.0.3.5)
-      actionview (= 6.0.3.5)
-      activejob (= 6.0.3.5)
-      activemodel (= 6.0.3.5)
-      activerecord (= 6.0.3.5)
-      activestorage (= 6.0.3.5)
-      activesupport (= 6.0.3.5)
+    rails (6.0.3.7)
+      actioncable (= 6.0.3.7)
+      actionmailbox (= 6.0.3.7)
+      actionmailer (= 6.0.3.7)
+      actionpack (= 6.0.3.7)
+      actiontext (= 6.0.3.7)
+      actionview (= 6.0.3.7)
+      activejob (= 6.0.3.7)
+      activemodel (= 6.0.3.7)
+      activerecord (= 6.0.3.7)
+      activestorage (= 6.0.3.7)
+      activesupport (= 6.0.3.7)
       bundler (>= 1.3.0)
-      railties (= 6.0.3.5)
+      railties (= 6.0.3.7)
       sprockets-rails (>= 2.0.0)
     rails-dom-testing (2.0.3)
       activesupport (>= 4.2.0)
       nokogiri (>= 1.6)
     rails-html-sanitizer (1.3.0)
       loofah (~> 2.3)
-    railties (6.0.3.5)
-      actionpack (= 6.0.3.5)
-      activesupport (= 6.0.3.5)
+    railties (6.0.3.7)
+      actionpack (= 6.0.3.7)
+      activesupport (= 6.0.3.7)
       method_source
       rake (>= 0.8.7)
       thor (>= 0.20.3, < 2.0)
@@ -196,9 +208,11 @@ GEM
     sqlite3 (1.4.2)
     thor (1.1.0)
     thread_safe (0.3.6)
+    time (0.1.0)
+    timeout (0.1.1)
     tzinfo (1.2.9)
       thread_safe (~> 0.1)
-    websocket-driver (0.7.3)
+    websocket-driver (0.7.4)
       websocket-extensions (>= 0.1.0)
     websocket-extensions (0.1.5)
     zeitwerk (2.4.2)

diff --git a/gemfiles/google_cloud_tasks_1.0.gemfile.lock b/gemfiles/google_cloud_tasks_1.0.gemfile.lock
@@ -1,7 +1,7 @@
 PATH
   remote: ..
   specs:
-    cloudtasker (0.12.rc9)
+    cloudtasker (0.12.rc10)
       activesupport
       connection_pool
       fugit
@@ -121,7 +121,7 @@ GEM
     faraday-net_http (1.0.1)
     faraday-net_http_persistent (1.1.0)
     fiber-local (1.0.0)
-    fugit (1.4.4)
+    fugit (1.4.5)
       et-orbi (~> 1.1, >= 1.1.8)
       raabro (~> 1.4)
     github_changelog_generator (1.16.1)
@@ -146,7 +146,7 @@ GEM
       googleauth (~> 0.9)
       grpc (~> 1.24)
       rly (~> 0.2.3)
-    google-protobuf (3.15.8)
+    google-protobuf (3.17.1)
     googleapis-common-protos (1.3.11)
       google-protobuf (~> 3.14)
       googleapis-common-protos-types (>= 1.0.6, < 2.0)
@@ -160,7 +160,7 @@ GEM
       multi_json (~> 1.11)
       os (>= 0.9, < 2.0)
       signet (~> 0.14)
-    grpc (1.37.0)
+    grpc (1.38.0)
       google-protobuf (~> 3.15)
       googleapis-common-protos-types (~> 1.0)
     grpc-google-iam-v1 (0.6.11)