Tag Archive
Discover 2 stories with “kafka” Tag.

If you are using Spark Streaming Kafka Direct API and you are storing offsets in Zookeeper, you might want to reuse offsets from Zookeeper and not always rely on checkpoint. KafkaUtils.createDirectStream() has a fromOffsets parameter which you can provide
Read More

If you are using Spark Streaming's new Kafka Direct API you might be missing some things, like monitoring tools that only work with offsets in ZK. For example, Kafka's kafka-consumer-offset-checker.sh which is very helpful to check consumer lag, meaning
Read More