Skip to content

Instantly share code, notes, and snippets.

@nch14

nch14/KafkaOffsetRangeUtil.java

Created December 26, 2017 09:47
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save nch14/785d3a1f738e92b7ebb7034281e46205 to your computer and use it in GitHub Desktop.
Save nch14/785d3a1f738e92b7ebb7034281e46205 to your computer and use it in GitHub Desktop.
Download ZIP
Used to fetch the lowest offset for each partition of a topic.
Raw
KafkaOffsetRangeUtil.java
package cn.sky_data.storeData.util;
import lombok.AllArgsConstructor;
import lombok.Data;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import java.util.Collections;
import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;
public class KafkaOffsetRangeUtil {
public List<TopicPartitionInfo> fetch(Map<String, Object> config, List<String> topics) {
KafkaConsumer<String, byte[]> consumer = new KafkaConsumer<>(config);
consumer.subscribe(topics);
ConsumerRecords<String, byte[]> records = consumer.poll(1000);
return records.partitions().parallelStream().map(topicPartition -> {
consumer.seekToBeginning(Collections.singletonList(topicPartition));
long offset = consumer.position(topicPartition);
return new TopicPartitionInfo(topicPartition.topic(), topicPartition.partition(), offset);
}).collect(Collectors.toList());
}
@Data
@AllArgsConstructor
public static class TopicPartitionInfo {
private String topic;
private int partition;
private long offset;
}
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment