Allows upgrading in a backward compatible way. This is needed when upgrading from [0.10.0, 1.1] to 2.0+, or when upgrading from [2.0, 2.3] to 2.4+. When upgrading from 3.3 to a newer version it is not required to specify this config. Default is `null`. Accepted values are "0.10.0", "0.10.1", "0.10.2", "0.11.0", "1.0", "1.1", "2.0", "2.1", "2.2", "2.3", "2.4", "2.5", "2.6", "2.7", "2.8", "3.0", "3.1", "3.2", "3.3", "3.4" (for upgrading from the corresponding old version).
+Allows upgrading in a backward compatible way. This is needed when upgrading from [0.10.0, 1.1] to 2.0+, or when upgrading from [2.0, 2.3] to 2.4+. When upgrading from 3.3 to a newer version it is not required to specify this config. Default is `null`. Accepted values are "0.10.0", "0.10.1", "0.10.2", "0.11.0", "1.0", "1.1", "2.0", "2.1", "2.2", "2.3", "2.4", "2.5", "2.6", "2.7", "2.8", "3.0", "3.1", "3.2", "3.3", "3.4", "3.5(for upgrading from the corresponding old version).
Apache Kafka doesn't provide an out-of-the-box RemoteStorageManager implementation. To have a preview of the tiered storage
- feature, the LocalTieredStorage
- implemented for integration test can be used, which will create a temporary directory in local storage to simulate the remote storage.
-
-
-After build successfully, there should be a `kafka-storage-x.x.x-test.jar` file under `storage/build/libs`.
-Next, setting configurations in the broker side to enable tiered storage feature.
+Here is a sample configuration to enable tiered storage feature in broker side:
-# remote.storage.enable=true -> enables tiered storage on the topic
-# local.retention.ms=1000 -> The number of milliseconds to keep the local log segment before it gets deleted.
- Note that a local log segment is eligible for deletion only after it gets uploaded to remote.
-# retention.ms=3600000 -> when segments exceed this time, the segments in remote storage will be deleted
-# segment.bytes=1048576 -> for test only, to speed up the log segment rolling interval
-# file.delete.delay.ms=10000 -> for test only, to speed up the local-log segment file delete delay
-
-bin/kafka-topics.sh --create --topic tieredTopic --bootstrap-server localhost:9092 \
---config remote.storage.enable=true --config local.retention.ms=1000 --config retention.ms=3600000 \
---config segment.bytes=1048576 --config file.delete.delay.ms=1000
+After broker is started, creating a topic with tiered storage enabled, and a small log time retention value to try this feature:
+
bin/kafka-topics.sh --create --topic tieredTopic --bootstrap-server localhost:9092 --config remote.storage.enable=true --config local.retention.ms=1000
-Try to send messages to the `tieredTopic` topic to roll the log segment:
-
-
-bin/kafka-producer-perf-test.sh --topic tieredTopic --num-records 1200 --record-size 1024 --throughput -1 --producer-props bootstrap.servers=localhost:9092
-
-
Then, after the active segment is rolled, the old segment should be moved to the remote storage and get deleted.
- This can be verified by checking the remote log directory configured above. For example:
- > ls /tmp/kafka-remote-storage/kafka-tiered-storage/tieredTopic-0-jF8s79t9SrG_PNqlwv7bAA
-00000000000000000000-knnxbs3FSRyKdPcSAOQC-w.index
-00000000000000000000-knnxbs3FSRyKdPcSAOQC-w.snapshot
-00000000000000000000-knnxbs3FSRyKdPcSAOQC-w.leader_epoch_checkpoint
-00000000000000000000-knnxbs3FSRyKdPcSAOQC-w.timeindex
-00000000000000000000-knnxbs3FSRyKdPcSAOQC-w.log
-
-
-Lastly, we can try to consume some data from the beginning and print offset number, to make sure it will successfully fetch offset 0 from the remote storage.
-
-bin/kafka-console-consumer.sh --topic tieredTopic --from-beginning --max-messages 1 --bootstrap-server localhost:9092 --property print.offset=true
-
-Please note, if you want to disable tiered storage at the cluster level, you should delete the tiered storage enabled topics explicitly.
- Attempting to disable tiered storage at the cluster level without deleting the topics using tiered storage will result in an exception during startup.
-
-bin/kafka-topics.sh --delete --topic tieredTopic --bootstrap-server localhost:9092
-
-After topics are deleted, you're safe to set remote.log.storage.system.enable=false in the broker configuration.
-
While the early access release of Tiered Storage offers the opportunity to try out this new feature, it is important to be aware of the following limitations:
diff --git a/36/streams/developer-guide/dsl-api.html b/36/streams/developer-guide/dsl-api.html
index 08bf2ef8c..ed2afb584 100644
--- a/36/streams/developer-guide/dsl-api.html
+++ b/36/streams/developer-guide/dsl-api.html
@@ -2818,6 +2818,7 @@
See the semantics overview at the bottom of this section for a detailed description.
@@ -2872,6 +2880,7 @@