Skip to content

Commit

Permalink
fix: Change set arithmetic for object store sensor
Browse files Browse the repository at this point in the history
Update the set arithmetic for object store sensors to use the `-` operator to ensure
that only keys that are present in the bucket but not the cursor are retrieved. Also
ensure that the cursor is loaded as Python by deserializing from JSON.
  • Loading branch information
blarghmatey committed Oct 2, 2023
1 parent fc5ab07 commit 083cdae
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions src/ol_orchestrate/sensors/sync_gcs_to_s3.py
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ def check_new_gcs_assets_sensor( # noqa: PLR0913
for file in storage_client.list_blobs(bucket, prefix=bucket_prefix)
if object_filter_fn(file.name)
}
new_files: set[str] = bucket_files.difference(set(context.cursor or []))
new_files: set[str] = bucket_files - set(json.loads(context.cursor or "[]"))
if new_files:
context.update_cursor(json.dumps(list(bucket_files)))
yield RunRequest(
Expand Down Expand Up @@ -75,7 +75,7 @@ def check_new_s3_assets_sensor( # noqa: PLR0913
)
if object_filter_fn(obj["Key"])
}
new_files: set[str] = bucket_files.difference(set(context.cursor or []))
new_files: set[str] = bucket_files - set(json.loads(context.cursor or "[]"))
if new_files:
context.update_cursor(json.dumps(list(bucket_files)))
yield RunRequest(
Expand Down

0 comments on commit 083cdae

Please sign in to comment.