Explore checkpointing

[alexanderdean]

StreamingContext.setCheckpointDir() or equivalent. Enable that and set it to a well-known S3 location. this comes in handy if the Driver fails and restarts. You would use StreamingContext.getOrCreateContext() in this case, as well.

Here's more info on checkpointing: https://spark.apache.org/docs/latest/streaming-programming-guide.html#checkpointing