zookeeper

Introduction to Kafka with Clojure

Recently I was trying to study Kafka, but I didn’t find a single resource that would give me a quick introduction and hands-on experience with it and Clojure. So, I’m making my own here! Don’t expect a “too deep introduction” – this is just the quick-and-dirty introduction about the concepts, and then I’ll show some code examples in Clojure

Kafka is a messaging system similar to RabbitMQ and SQS. The great differential is that it’s faster than both solutions, and works very well in cluster mode. Installing Kafka locally is quite complicated so you probably will wants to use the docker-compose.yaml file below:

version: '2'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:5.3.1
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
  kafka:
    image: confluentinc/cp-enterprise-kafka:5.3.1
    hostname: broker
    container_name: broker
    depends_on:
      - zookeeper
    ports:
      - "9092:9092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_METRIC_REPORTERS: io.confluent.metrics.reporter.ConfluentMetricsReporter
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      CONFLUENT_METRICS_REPORTER_BOOTSTRAP_SERVERS: broker:29092
      CONFLUENT_METRICS_REPORTER_ZOOKEEPER_CONNECT: zookeeper:2181
      CONFLUENT_METRICS_REPORTER_TOPIC_REPLICAS: 1
      CONFLUENT_METRICS_ENABLE: 'true'
      CONFLUENT_SUPPORT_CUSTOMER_ID: 'anonymous'

This file will create the Kafka broker (like a single node of the messaging) and will add Zookeeper (that will allow you to coordinate between different Kafkas, decide which node is the leader, and also participate on the node election when the leader goes down, and other things). You will connect into 9092 port, and then listen and send messages from there.
(more…)

By Maurício Szabo, ago