Skip to content

Latest commit

 

History

History
630 lines (551 loc) · 52.9 KB

CHANGELOG.md

File metadata and controls

630 lines (551 loc) · 52.9 KB

Changelog

For 1.x release notes, please see v1.x/CHANGELOG.md


Release 2.6.0 (2024-05-01)

  • #1317 Add enablePriorityLeaseAssignment config
  • #1320 Update lease taker to get unassigned leases
  • #1318 Internally construct and use stream ARNs for all streams in multi-stream mode
  • #1291 Update RetrievalFactory implementations to utilize the StreamIdentifier field of StreamConfig
  • #1308 Move shutdownComplete call to ShardConsumer
  • #1313 Add additional integration tests for multi-stream and cross account access
  • #1273 Optimize currentStreamConfigMap by cleaning up lingering streams
  • #1302 Fix gracefulShutdown behavior in Scheduler

Release 2.5.8 (2024-03-27)

  • #1278 Upgrade awssdk.version from 2.25.3 to 2.25.11
  • #1279 Upgrade org.apache.maven.plugins:maven-gpg-plugin from 3.1.0 to 3.2.1
  • #1280 Upgrade org.apache.commons:commons-lang3 from 3.12.0 to 3.14.0
  • #1282 Upgrade org.apache.maven.plugins:maven-javadoc-plugin from 3.5.0 to 3.6.3
  • #1277 Reuse 'ShardSyncTaskManager' instance for existing stream to avoid duplicate enqueue of 'ShardSyncTask'

Release 2.5.7 (2024-03-19)

  • #1275 Update PollingConfig maxRecords method to return PollingConfig
  • #1236 Upgrade commons-io:commons-io from 2.11.0 to 2.15.1
  • #1189 Upgrade org.apache.maven.plugins:maven-resources-plugin from 3.3.0 to 3.3.1
  • #1139 Upgrade maven-surefire-plugin from 2.22.2 to 3.1.2
  • #1138 Upgrade maven-failsafe-plugin from 2.22.2 to 3.1.2
  • #1125 Upgrade maven-gpg-plugin from 3.0.1 to 3.1.0

Release 2.5.6 (2024-03-08)

  • #1271 Adding snapshot for 2.5.6-SNAPSHOT
  • #1268 Upgrade ch.qos.logback:logback-classic dependency from 1.3.12 to 1.3.14
  • #1268 Upgrade awssdk.version from 2.20.43 to 2.25.3
  • #1268 Upgrade aws-java-sdk.version from 1.12.405 to 1.12.668
  • #1268 Upgrade gsr.version from 1.1.17 to 1.1.19

Release 2.5.5 (2024-02-22)

  • #1257 Prevent improper error logging during worker shutdown
  • #1260 Add Deletion protection config
  • #1258 Fix issue in configuring metricsEnabledDimensions
  • #1259 Add snapshot to version

Release 2.5.4 (December 12, 2023)

  • #1232 Upgrade ch.qos.logback:logback-classic dependency from 1.3.0 to 1.3.12 in /amazon-kinesis-client
  • #1233 Upgrade ch.qos.logback:logback-classic dependency from 1.3.0 to 1.3.12 in /amazon-kinesis-client-multilang
  • #1230 Bug fix which now allows MultiLangDaemon to configure idleTimeBetweenReadsInMillis
  • #1229 Added link to javadoc.io-hosted Javadoc in the README
  • #1218 Added doc for leases and the lease lifecycle to help explain lease lifecycle logic.
  • #1226 Upgraded KCL from 2.5.3 to 2.5.4-SNAPSHOT

Release 2.5.3 (November 8, 2023)

  • #1219 Provided streamArn in getRecords request
  • #1216 Updated AWS Glue Schema Registry from version 1.1.14 to 1.1.17.
  • #1205 Updated the FAQ with impact of changing default checkpoint.
  • #1203 Added links from README.md to FAQ and doc folder.
  • #1202 Introduced a FAQ for Kinesis Client Library
  • #1200 Added test case for StreamIdentifier serialization.

Release 2.5.2 (August 7, 2023)

  • #1184 [#367] Enhanced multi-lang AWSCredentialsProvider=... decoder and c…
  • #1186 Provided documentation for multilang's new NestedPropertyKey enhancement.
  • #1181 CVE-2023-2976: Upgrade Google Guava dependency version from 32.0.0-jre to 32.1.1-jre
  • #1159 Bug fix in lease refresher integration test with occasional failures
  • #1157 Fix NPE on graceful shutdown before DDB LeaseCoordinator starts.
  • #1152 Adding resharding integration tests and changing ITs to not run by default
  • #1162 Only deleting resource created by ITs
  • #1158 Checkstyle: tightened LineLength restriction from 170 to 150.
  • #1151 Modified dependabot.yml to set the correct v[1|2].x label.
  • #1164 Upgraded KCL Version from 2.5.1 to 2.5.2-SNAPSHOT

Release 2.5.1 (June 27, 2023)

  • #1143 Upgrade MultiLangDaemon to support StreamARN
  • #1145 Introduced GitHub actions to trigger Maven builds during merge/pull requests
  • #1136 Added testing architecture and KCL 2.x basic polling/streaming tests
  • #1153 Checkstyle: added UnusedImports check.
  • #1150 Enabled Checkstyle validation of test resources.
  • #1149 Bound Checkstyle to validate goal for automated enforcement.
  • #1148 Code cleanup to faciliate Checkstyle enforcement.
  • #1142 Upgrade Google Guava dependency version from 31.1-jre to 32.0.0-jre
  • #1115 Update KCL version from 2.5.0 to 2.5.1-SNAPSHOT

Release 2.5.0 (May 19, 2023)

  • #1109 Add support for stream ARNs
  • #1065 Allow tags to be added when lease table is created
  • #1094 Code cleanup to introduce better testing
  • #1088 Minimize race in PSSM to optimize shard sync calls
  • #1086 Add additional SingleStreamTracker constructor with stream position parameter
  • #1084 More consistent testing behavior with restartAfterRequestTimerExpires
  • #1066 More consistent testing behavior with HashRangesAreAlwaysComplete
  • #1072 Upgrade nexus-staging-maven-plugin from 1.6.8 to 1.6.13
  • #1073 Upgrade slf4j-api from 2.0.6 to 2.0.7
  • #1090 Upgrade awssdk.version from 2.20.8 to 2.20.43
  • #1071 Upgrade maven-compiler-plugin from 3.8.1 to 3.11.0

Release 2.4.8 (March 21, 2023)

  • #1080 Added metric in ShutdownTask for scenario when parent leases are missing.
  • #1077 Reverted changes to pom property
  • #1069 Fixed flaky InitializationWaitsWhenLeaseTableIsEmpty test

Release 2.4.7 (March 17, 2023)

  • NOTE: Due to an issue during the release process, the 2.4.7 published artifacts are incomplete and non-viable. Please use 2.4.8 or later.
  • #1063 Allow leader to learn new leases upon re-election to avoid unnecessary shardSyncs
  • #1060 Add new metric to be emitted on lease creation
  • #1057 Added more logging in Scheduler w.r.t. StreamConfigs.
  • #1059 DRY: simplification of HierarchicalShardSyncerTest.
  • #1062 Fixed retry storm in PrefetchRecordsPublisher.
  • #1061 Fixed NPE in LeaseCleanupManager.
  • #1056 Clean up in-memory state of deleted kinesis stream in MultiStreamMode
  • #1058 Documentation: added <pre> tags so fixed-format diagrams aren't garbled.
  • #1053 Exposed convenience method of ExtendedSequenceNumber#isSentinelCheckpoint()
  • #1043 Removed a .swp file, and updated .gitignore.
  • #1047 Upgrade awssdk.version from 2.19.31 to 2.20.8
  • #1046 Upgrade maven-javadoc-plugin from 3.3.1 to 3.5.0
  • #1038 Upgrade gsr.version from 1.1.13 to 1.1.14
  • #1037 Upgrade aws-java-sdk.version from 1.12.370 to 1.12.405

Release 2.4.6 (February 21, 2023)

  • #1041 Minor optimizations (e.g., calculate-once, put instead of get+put)
  • #1035 Release Note updates to avoid duplication and bitrot (e.g., 1.x release
  • #935 Pass isAtShardEnd correctly to processRecords call
  • #1040 Increased logging verbosity around lease management
  • #1024 Added logging w.r.t. StreamConfig handling.
  • #1034 Optimization: 9~15% improvement in KinesisDataFetcher wall-time
  • #1045 Fixed duplication of project version in children pom.xml
  • #956 Fixed warning message typos
  • #795 Fixed log message spacing
  • #740 Fixed typo in Comment
  • #1028 Refactored MultiStreamTracker to provide and enhance OOP for both
  • #1027 Removed CHECKSTYLE:OFF toggles which can invite/obscure sub-par code.
  • #1032 Upgrade rxjava from 3.1.5 to 3.1.6
  • #1030 Upgrade awssdk.version from 2.19.2 to 2.19.31
  • #1029 Upgrade slf4j-api from 2.0.0 to 2.0.6
  • #1015 Upgrade protobuf-java from 3.21.5 to 3.21.12

Release 2.4.5 (January 04, 2023)

  • #1014 Use AFTER_SEQUENCE_NUMBER iterator type for expired iterator request

Release 2.4.4 (December 23, 2022)

  • #1017 Upgrade aws sdk
    • aws-java-sdk.version from 1.12.296 -> 1.12.370
    • awssdk.version from 2.17.268 -> 2.19.2
  • #1020 Correct the KCL version in the main pom

Release 2.4.3 (September 6, 2022)

  • #980 logback-classic: 1.2.9 -> 1.4.0
  • #983
    • protobuf-java: 3.19.2 -> 3.21.5
    • slf4j.version: 1.7.32 -> 2.0.0
    • schema-registry-serde: 1.1.9 -> 1.1.13
  • #984 awssdk.version from 2.17.108 to 2.17.267
  • #987 guava: 31.0.1-jre -> 31.1-jre
  • #988 jcommander: 1.81 to 1.82
  • #990 Upgrade dependencies
    • aws-java-sdk.version: 1.12.130 -> 1.12.296
    • lombok: 1.18.22 -> 1.18.24
    • rxjava: 3.1.3 -> 3.1.5
    • maven-resources-plugin: 2.6 -> 3.3.0
    • logback-classic: 1.4.0 -> 1.3.0
    • awssdk.version: 2.17.267 -> 2.17.268

Release 2.4.2 (August 10, 2022)

  • #972 Upgrade Lombok to version 1.18.24

Latest Release 2.4.1 (March 24, 2022)

Milestone#68

  • #916 Upgrade to rxjava3

Release 2.4.0 (March 2, 2022)

Milestone#67

  • #894 Bump protobuf-java from 3.19.1 to 3.19.2
  • #924 Support Protobuf Data format with Glue Schema Registry.

Latest Release 2.3.10 (January 4, 2022)

Milestone#66

  • #868 Adding a new metric: Application-level MillisBehindLatest
  • #879 Keep dependencies up-to-date
  • #886 Get latest counter before attempting a take to ensure take succeeds
  • #888 Configure dependabot for v1.x branch

Release 2.3.9 (November 22, 2021)

Milestone#65

  • #866 Update logback dependency.

Release 2.3.8 (October 27, 2021)

Milestone#64

  • #860 Upgrade Glue schema registry from 1.1.4 to 1.1.5.
  • #861 Revert PR#847 due to regression for leases without owners and added new tests.

Release 2.3.7 (October 11, 2021)

Milestone#63

  • #842 Fixing typo is debug logs.
  • #846 Fix DynamoDBLeaseTaker logging of available leases
  • #847 Make use of Java 8 to simplify computeLeaseCounts()
  • #853 Add configurable initial position for orphaned stream
  • #854 Create DynamoDB tables on On-Demand billing mode by default.
  • #855 Emit Glue Schema Registry usage metrics
  • #857 Fix to shutdown PrefetchRecordsPublisher in gracefull manner
  • #858 Upgrade AWS SDK version to 2.17.52.

Release 2.3.6 (July 9, 2021)

Milestone#62

  • #836 Upgraded AWS SDK version to 2.16.98
  • #835 Upgraded Glue Schema Registry version to 1.1.1
  • #828 Modified wildcard imports to individual imports.
  • #817 Updated the Worker shutdown logic to make sure that the LeaseCleanupManager also terminates all the threads that it has started.
  • #794 Silence warning when there are no stale streams to delete.

Release 2.3.5 (June 14, 2021)

Milestone#59

  • #824 Upgraded dependencies
    • logback-classic version to 1.2.3
    • AWS Java SDK version to 1.12.3
    • AWS SDK version to 2.16.81
  • #815 Converted Future to CompletableFuture which helps in proper conversion to Scala using Scala Future Converters.
  • #810 Bump commons-io from 2.6 to 2.7
  • #804 Allowing user to specify an initial timestamp in which daemon will process records.
  • #802 Upgraded guava from 26.0-jre to 29.0-jre
  • #801 Fixing a bug that causes to block indefinitely when trying to unlock a lock that isn't locked.
  • #762 Added support for web identity token in multilang

Release 2.3.4 (February 19, 2021)

Milestone#56

  • #788 Fixing a bug that caused paginated ListShards calls with the ShardFilter parameter to fail when the lease table was being initialized.

Release 2.3.3 (December 23, 2020)

Milestone#55

  • Fixing bug in PrefetchRecordsPublisher which was causing retry storms if initial request fails.
  • Fixing bug where idleTimeBetweenReadsInMillis property was ignored in PollingConfig.

Release 2.3.2 (November 19, 2020)

Milestone#54

  • Adding support for Glue Schema Registry. Deserialize and read schemas associated with the records.
  • Updating AWS SDK version to 2.15.31

Release 2.3.1 (October 20, 2020)

Milestone#53

Release 2.3.0 (August 17, 2020)

  • Milestone#52

  • Behavior of shard synchronization is moving from each worker independently learning about all existing shards to workers only discovering the children of shards that each worker owns. This optimizes memory usage, lease table IOPS usage, and number of calls made to kinesis for streams with high shard counts and/or frequent resharding.

  • When bootstrapping an empty lease table, KCL utilizes the ListShard API's filtering option (the ShardFilter optional request parameter) to retrieve and create leases only for a snapshot of shards open at the time specified by the ShardFilter parameter. The ShardFilter parameter enables you to filter out the response of the ListShards API, using the Type parameter. KCL uses the Type filter parameter and the following of its valid values to identify and return a snapshot of open shards that might require new leases.

    • Currently, the following shard filters are supported:
      • AT_TRIM_HORIZON - the response includes all the shards that were open at TRIM_HORIZON.
      • AT_LATEST - the response includes only the currently open shards of the data stream.
      • AT_TIMESTAMP - the response includes all shards whose start timestamp is less than or equal to the given timestamp and end timestamp is greater than or equal to the given timestamp or still open.
    • ShardFilter is used when creating leases for an empty lease table to initialize leases for a snapshot of shards specified at RetrievalConfig#initialPositionInStreamExtended.
    • For more information about ShardFilter, see the official AWS documentation on ShardFilter.
  • Introducing support for the ChildShards response of the GetRecords and the SubscribeToShard APIs to perform lease/shard synchronization that happens at SHARD_END for closed shards, allowing a KCL worker to only create leases for the child shards of the shard it finished processing.

    • For shared throughout consumer applications, this uses the ChildShards response of the GetRecords API. For dedicated throughput (enhanced fan-out) consumer applications, this uses the ChildShards response of the SubscribeToShard API.
    • For more information, see the official AWS Documentation on GetRecords, SubscribeToShard, and ChildShard.
  • KCL now also performs additional periodic shard/lease scans in order to identify any potential holes in the lease table to ensure the complete hash range of the stream is being processed and create leases for them if required. PeriodicShardSyncManager is the new component that is responsible for running periodic lease/shard scans.

    • New configuration options are available to configure PeriodicShardSyncManager in LeaseManagementConfig
    Name Default Description
    leasesRecoveryAuditorExecutionFrequencyMillis 120000 (2 minutes) Frequency (in millis) of the auditor job to scan for partial leases in the lease table. If the auditor detects any hole in the leases for a stream, then it would trigger shard sync based on leasesRecoveryAuditorInconsistencyConfidenceThreshold.
    leasesRecoveryAuditorInconsistencyConfidenceThreshold 3 Confidence threshold for the periodic auditor job to determine if leases for a stream in the lease table is inconsistent. If the auditor finds same set of inconsistencies consecutively for a stream for this many times, then it would trigger a shard sync
    • New CloudWatch metrics are also now emitted to monitor the health of PeriodicShardSyncManager:
    Name Description
    NumStreamsWithPartialLeases Number of streams that had holes in their hash ranges.
    NumStreamsToSync Number of streams which underwent a full shard sync.
  • Introducing deferred lease cleanup. Leases will be deleted asynchronously by LeaseCleanupManager upon reaching SHARD_END, when a shard has either expired past the stream’s retention period or been closed as the result of a resharding operation.

    • New configuration options are available to configure LeaseCleanupManager.
    Name Default Description
    leaseCleanupIntervalMillis 1 minute Interval at which to run lease cleanup thread.
    completedLeaseCleanupIntervalMillis 5 minutes Interval at which to check if a lease is completed or not.
    garbageLeaseCleanupIntervalMillis 30 minutes Interval at which to check if a lease is garbage (i.e trimmed past the stream's retention period) or not.
  • Introducing experimental support for multistreaming, allowing a single KCL application to multiplex processing multiple streams.

    • New configuration options are available to enable multistreaming in RetrievalConfig#appStreamTracker.
  • Fixing a bug in PrefetchRecordsPublisher restarting while it was already running.

  • Including an optimization to HierarchicalShardSyncer to only create leases for one layer of shards.

  • Adding support to prepare and commit lease checkpoints with arbitrary bytes.

    • This allows checkpointing of an arbitrary byte buffer up to the maximum permitted DynamoDB item size (currently 400 KB as of release), and can be used for recovery by passing a serialized byte buffer to RecordProcessorCheckpointer#prepareCheckpoint and RecordProcessorCheckpointer#checkpoint.
  • Upgrading version of AWS SDK to 2.14.0.

  • #725 Allowing KCL to consider lease tables in UPDATING healthy.

Release 2.2.11 (May 28, 2020)

Milestone#51

  • Adjusting HTTP2 initial window size to 512 KB
  • Updating protobuf-java to version 3.11.4
  • Updating the AWS Java SDK to version 2.13.25

Release 2.2.10 (March 26, 2020)

Milestone#48

  • Fixing a bug in DynamoDB billing mode support for special regions.
  • Adding request id logging to ShardConsumerSubscriber.

Release 2.2.9 (Febuary 17, 2020)

Milestone#47

  • Updating the AWS SDK version to 2.10.66.
  • Adding request id logging to SubscribeToShard response.

Release 2.2.8 (January 28, 2020)

Milestone#46

  • Updating the AWS SDK version to 2.10.56.
    • PR#679
    • NOTE: SDK has a known connection teardown issue when multiple H2 streams are used within a connection. This might result in shard consumers sticking to a stale service host and not progressing. If your shard consumer gets stuck, use the following configuration as a workaround. This configuration might result in up to 5X increase in total connections.
    KinesisAsyncClient kinesisClient = KinesisAsyncClient.builder()
                                                         .region(region)
                                                         .httpClientBuilder(NettyNioAsyncHttpClient.builder().maxConcurrency(Integer.MAX_VALUE).http2Configuration(Http2Configuration.builder().maxStreams(1).build())
                                                         .build()
    
  • Making ShardConsumerTest resilient to race conditions.
  • Updating integration test naming.

Release 2.2.7 (December 2, 2019)

Milestone#45

  • Updating the AWS SDK version to 2.10.25
  • Adding a configurable DynamoDB billing mode
    • PR#582
    • NOTE: Billing mode is not available in all regions; if your lease table cannot be created, use the following configuration as a workaround:
    LeaseManagementConfig leaseManagementConfig = builder.leaseManagementConfig().billingMode(null).build();
    

Release 2.2.6 (November 7, 2019)

Milestone#43

  • Updating the SDK version to 2.9.25.
  • Clearing the local cache on a subscription termination, to avoid noisy logs on new subscriptions.
  • Updating the SDK version to 2.10.0 in order to fix the premature H2 stream close issue.
    • PR#649
    • NOTE: SDK has a known connection teardown issue when multiple H2 streams are used within a connection. This might result in shard consumers sticking to a stale service host and not progressing. If your shard consumer gets stuck, use the following configuration as a workaround. This configuration might result in up to 5X increase in total connections.
    KinesisAsyncClient kinesisClient = KinesisAsyncClient.builder()
                                                         .region(region)
                                                         .httpClientBuilder(NettyNioAsyncHttpClient.builder().maxConcurrency(Integer.MAX_VALUE).maxHttp2Streams(1))
                                                         .build()
    

Release 2.2.5 (October 23, 2019)

Milestone#40

  • Updating Sonatype to dedicated AWS endpoint.
  • Introducing a validation step to verify if ShardEnd is reached, to prevent shard consumer stuck scenarios in the event of malformed response from service.

Release 2.2.4 (September 23, 2019)

Milestone#39

  • Making FanoutRecordsPublisher test cases resilient to delayed thread operations
  • Drain delivery queue in the FanoutRecordsPublisher to make slow consumers consume events at their pace
  • Fix to prevent the onNext event going to stale subscription when restart happens in PrefetchRecordsPublisher

Release 2.2.3 (September 04, 2019)

Milestone#38

  • Fix to prevent data loss and stuck shards in the event of failed records delivery in Polling readers

Release 2.2.2 (August 19, 2019)

Milestone#36

  • Fix to prevent invalid ShardConsumer state transitions due to rejected executor service executions.
  • Fixing a bug in which initial subscription failure caused a shard consumer to get stuck.
  • Making CW publish failures visible by executing the async publish calls in a blocking manner and logging on exception.
  • Update shard end checkpoint failure messaging.
  • A fix for resiliency and durability issues that occur in the reduced thread mode - Nonblocking approach.
  • Preventing duplicate delivery due to unacknowledged event, while completing the subscription.

Release 2.2.1 (July 1, 2019)

Milestone#32

  • Add periodic logging for the state of the thread pool executor service. This service executes the async tasks submitted to and by the ShardConsumer.
  • Add logging of failures from RxJava layer.

Release 2.2.0 (April 8, 2019)

Milestone#31

  • Updated License to Apache License 2.0
  • Introducing configuration for suppressing logs from ReadTimeoutExceptions caused while calling SubscribeToShard.
    Suppression can be configured by setting LifecycleConfig#readTimeoutsToIgnoreBeforeWarning(Count).

Release 2.1.3 (March 18, 2019)

Milestone#30

  • Added a message to recommend using KinesisClientUtil when an acquire timeout occurs in the FanOutRecordsPublisher.
  • Added a sleep between retries while waiting for a newly created stream consumer to become active.
  • Added timeouts on all futures returned from the DynamoDB and Kinesis clients.
    The timeouts can be configured by setting LeaseManagementConfig#requestTimeout(Duration) for DynamoDB, and PollingConfig#kinesisRequestTimeout(Duration) for Kinesis.
  • Upgraded to SDK version 2.5.10.
  • Artifacts for the Amazon Kinesis Client for Java are now signed by a new GPG key:
    pub   4096R/86368934 2019-02-14 [expires: 2020-02-14]
    uid                  Amazon Kinesis Tools <[email protected]>
    

Release 2.1.2 (February 18, 2019)

Milestone#29

  • Fixed handling of the progress detection in the ShardConsumer to restart from the last accepted record, instead of the last queued record.
  • Fixed handling of exceptions when using polling so that it will no longer treat SdkExceptions as an unexpected exception.
  • Fixed a case where lease loss would block the Scheduler while waiting for a record processor's processRecords method to complete.

Release 2.1.1 (February 6, 2019)

Milestone#28

  • Introducing SHUT_DOWN_STARTED state for the WorkerStateChangeListener.
  • Fixed a bug with AWSSessionCredentials using AWSSecretID instead of AWSAccessID and vice versa.
  • Upgrading SDK version to 2.4.0, which includes a fix for a possible deadlock when using Enhanced Fan-Out.

Release 2.1.0 (January 14, 2019)

Milestone #27

  • Introducing MultiLangDaemon support for Enhanced Fan-Out.
  • MultiLangDaemon now supports the following command line options.
    • --properties-file: Properties file that the KCL should use to set up the Scheduler.
    • --log-configuration: logback.xml that the KCL should use for logging.
  • Updated AWS SDK dependency to 2.2.0.
  • MultiLangDaemon now uses logback for logging.

Release 2.0.5 (November 12, 2018)

Milestone #26

  • Fixed a deadlock condition that could occur when using the polling model.
    When using the PollingConfig and a slower record processor it was possible to hit a deadlock in the retrieval of records.
  • Adjusted RetrievalConfig, and FanOutConfig to use accessors instead of direct member access.

Release 2.0.4 (October 18, 2018)

Milestone #25

  • Added method to retrieve leases from the LeaseCoordinator and LeaseTaker.
  • Fixed a race condition shutting down the Scheduler before it has completed initialization.
  • Added HierarchicalShardSyncer which replaces the static ShardSyncer.
    HierarchicalShardSyncer removes the contention between multiple instances of the Scheduler when running under a single JVM.
  • Added TaskExecutionListener which allows monitoring of tasks being executed by the ShardConsumer.
    The listener is invoked before and after a task is executed by the ShardConsumer.

Release 2.0.3 (October 8, 2018)

Milestone #23

  • Fixed an issue where the KinesisAsyncClient could be misconfigured to use HTTP 1.1.
    Using HTTP 1.1 with SubscribeToShard is unsupported, and could cause misdelivery of records to the record processor.
  • Lower the severity of ReadTimeout exceptions.
    ReadTimeout exceptions can occur if the client is unable to request data from Kinesis for more than client timeout, which defaults to 30 seconds. This can occur if the record processor blocks for more than the timeout period. ReadTimeout could also occur as part of Issue #391.
  • Added a callback that allows applications to take actions after DynamoDB table creation.
    Applications can now install a callback that is called after creating the DynamoDB table by implementing TableCreatorCallback.
  • Updated the guava dependency to 26.0-jre.
  • Added some additional debug logging around the initialization of the FanOutRecordsPublisher.
  • Upgraded AWS SDK version to 2.0.6

Release 2.0.2 (September 4, 2018)

Milestone #22

  • Fixed an issue where the a warning would be logged every second if logWarningForTaskAfterMillis was set.
    The logging for last time of data arrival now respects the value of logWarningForTaskAfterMillis.
  • Moved creation of WorkerStateChangedListener and GracefulShutdownCoordinator to the CoordinatorConfig. Originally the WorkerStateChangedListener and GracefulShutdownCoordinator were created by methods on the SchedulerCoordinatorFactory, but they should have been configuration options.
    The original methods have been deprecated, and may be removed at a later date.
  • Removed dependency on Apache Commons Lang 2.6.
    The dependency on Apache Commons Lang 2.6 has removed, and all usages updated to use Apache Commons Lang 3.7.
  • Fixed a typo in the MutliLang Daemon shutdown hook.
  • Added method onAllInitializationAttemptsFailed(Throwable) to WorkerStateChangedListener to report when all initialization attempts have failed.
    This method is a default method, and it isn't require to implement the method. This method is only called after all attempts to initialize the Scheduler have failed.

Release 2.0.1 (August 21, 2018)

  • Mark certain internal components with @KinesisClientInternalApi attribute.
    Components marked as internal may be deprecated at a faster rate than public components.
  • Fixed an issue where ResourceNotFoundException on subscription to a shard was not triggering end of shard handling.
    If a lease table contains a shard that is no longer present in the stream attempt to subscribe to that shard will trigger a ResourceNotFoundException. These exception are treated the same as reaching the end of a shard.
  • Fixed an issue where the KCL would not Use the configured DynamoDB IOPs when creating the lease table.
  • Make the maximum number of Scheduler initialization attempts configurable.
    The maximum number of Scheduler initialization attempts can be configured via CoordinatorConfig#maxInitializationAttempts.
  • Fixed an issue where it was possible to get a duplicate record when resubscribing to a shard.
    Subscribe to shard requires periodic resubscribing, and uses a new concept of a continuation sequence number. If the continuation sequence number was equal to the last record that record would be processed a second time. Resubscribing now uses AFTER_SEQUENCE_NUMBER to ensure that only later records are returned.
  • Upgraded to AWS SDK 2.0.1
  • Fixed an issue where time based restart of the subscription wasn't resetting the lastRequestTime.
    If a subscription hasn't delivered any data for more than 30 seconds it will be canceled and restarted. This detection is based of the lastRequestTime which wasn't getting reset after the restart was triggered.
  • Fixed an issue where requesting on the subscription from the FanOutRecordsPublisher could trigger an unexpected failure.
    Due to a race condition the underlying flow in the subscription could be set to something else. The method is now synchronized, and verifies that the subscriber it was created with is still the subscriber in affect.
    This issue generally would only appear when multiple errors were occurring while connecting to Kinesis.
  • Fixed an issue where the number of requested items could exceed the capacity of the RxJava queue.
    There was an off by one issue when determining whether to make a request to the SDK subscription. This changes the calculation to represent the capacity as a queue.

Release 2.0.0 (August 02, 2018)

  • The Maven groupId, along with the version, for the Amazon Kinesis Client has changed from com.amazonaws to software.amazon.kinesis.
    To add a dependency on the new version of the Amazon Kinesis Client:

    <dependency>
        <groupId>software.amazon.kinesis</groupId>
        <artifactId>amazon-kinesis-client</artifactId>
        <version>2.0.0</version>
    </dependency>
  • Added support for Enhanced Fan Out.
    Enhanced Fan Out provides for lower end to end latency, and increased number of consumers per stream.

    • Records are now delivered via streaming, reducing end-to-end latency.
    • The Amazon Kinesis Client will automatically register a new consumer if required.
      When registering a new consumer, the Kinesis Client will default to the application name unless configured otherwise.
    • SubscribeToShard maintains long lived connections with Kinesis, which in the AWS Java SDK 2.0 is limited by default.
      The KinesisClientUtil has been added to assist configuring the maxConcurrency of the KinesisAsyncClient.
      WARNING: The Amazon Kinesis Client may see significantly increased latency, unless the KinesisAsyncClient is configured to have a maxConcurrency high enough to allow all leases plus additional usages of the KinesisAsyncClient.
    • The Amazon Kinesis Client now uses 3 additional Kinesis API's:
      WARNING: If using a restrictive Kinesis IAM policy you may need to add the following API methods to the policy.
    • New configuration options are available to configure Enhanced Fan Out.
    Name Default Description
    consumerArn Unset The ARN for an already created consumer. If this is set, the Kinesis Client will not attempt to create a consumer.
    streamName Unset The name of the stream that a consumer should be create for if necessary
    consumerName Unset The name of the consumer to create. If this is not set the applicationName will be used instead.
    applicationName Unset The name of the application. This is used as the name of the consumer unless consumerName is set.
  • Modular Configuration of the Kinesis Client The Kinesis Client has migrated to a modular configuration system, and the KinesisClientLibConfiguration class has been removed.
    Configuration has been split into 7 classes. Default versions of the configuration can be created from the ConfigsBuilder.
    Please [see the migration guide for more information][migration-guide].

    • CheckpointConfig
    • CoordinatorConfig
    • LeaseManagementConfig
    • LifecycleConfig
    • MetricsConfig
    • ProcessorConfig
    • RetrievalConfig
  • Upgraded to AWS Java SDK 2.0
    The Kinesis Client now uses the AWS Java SDK 2.0. The dependency on AWS Java SDK 1.11 has been removed. All configurations will only accept 2.0 clients.

    • When configuring the KinesisAsyncClient the KinesisClientUtil#createKinesisAsyncClient can be used to configure the Kinesis Client
    • If you need support for AWS Java SDK 1.11 you will need to add a direct dependency.
      When adding a dependency you must ensure that the 1.11 versions of Jackson dependencies are excluded
      [Please see the migration guide for more information][migration-guide]
  • MultiLangDaemon is now a separate module
    The MultiLangDaemon has been separated to its own Maven module and is no longer available in amazon-kinesis-client. To include the MultiLangDaemon, add a dependency on amazon-kinesis-client-multilang.