06

Module 6: Kafka Setup & Hands-On

Chapter 6 • Beginner

50 min

Kafka Setup & Hands-On

You now understand Kafka's core concepts and architecture. In this module, you'll get hands-on experience by installing Kafka locally, managing topics, and using producers and consumers from the command line.

This is where Kafka moves from "theory" to "I actually ran it on my machine."

🎯 What You Will Learn

By the end of this module, you will be able to:

  • Set up Kafka locally using manual install, Docker, or KRaft mode
  • Create, list, describe, and delete topics using CLI tools
  • Use console producers and consumers to send and read messages
  • Work with consumer groups and see load balancing in action
  • Understand and modify key Kafka configuration files
  • Use basic monitoring and management commands
  • Troubleshoot common issues you’ll see in real environments

⚙️ Installation Methods

You can run Kafka in different ways depending on your environment and preferences.

✅ Method 1: Manual Installation

Prerequisites

  • Java 8 or higher
  • At least 4GB RAM
  • Disk space for Kafka logs

Steps

  1. Download Kafka
  • Visit: https://kafka.apache.org/downloads
  • Download the latest binary (for example: kafka_2.13-3.5.0.tgz)
  1. Extract and Setup
bash.js
        tar -xzf kafka_2.13-3.5.0.tgz
        cd kafka_2.13-3.5.0
  1. Start Zookeeper (if using Zookeeper mode)
bash.js
        bin/zookeeper-server-start.sh config/zookeeper.properties
  1. Start Kafka Broker
bash.js
        bin/kafka-server-start.sh config/server.properties

✅ Method 2: Docker Installation (Recommended for Dev)

Using Docker simplifies setup and cleanup, especially for local development.

Docker Compose Setup

yaml.js
    version: '3.8'
    services:
      zookeeper:
        image: confluentinc/cp-zookeeper:latest
        environment:
          ZOOKEEPER_CLIENT_PORT: 2181
          ZOOKEEPER_TICK_TIME: 2000
    
      kafka:
        image: confluentinc/cp-kafka:latest
        depends_on:
          - zookeeper
        ports:
          - "9092:9092"
        environment:
          KAFKA_BROKER_ID: 1
          KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
          KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
          KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

Start with Docker

bash.js
    docker-compose up -d

This will bring up Zookeeper + Kafka with minimal effort.

✅ Method 3: KRaft Mode (Kafka 2.8+)

Modern Kafka can run without Zookeeper using KRaft mode.

Start Kafka in KRaft Mode

bash.js
    # Generate cluster ID
    bin/kafka-storage.sh format -t $(bin/kafka-storage.sh random-uuid) -c config/kraft/server.properties
    
    # Start Kafka in KRaft mode
    bin/kafka-server-start.sh config/kraft/server.properties

KRaft mode is recommended for new clusters.

🔧 Basic Kafka Operations

Once Kafka is running, you can interact with it using CLI tools.

1️⃣ Creating Topics

Using Command Line

bash.js
    # Create a topic
    bin/kafka-topics.sh --create --topic user-events --bootstrap-server localhost:9092 --partitions 3 --replication-factor 1
    
    # List topics
    bin/kafka-topics.sh --list --bootstrap-server localhost:9092
    
    # Describe topic
    bin/kafka-topics.sh --describe --topic user-events --bootstrap-server localhost:9092

Topic Configuration Basics

  • Partitions – Number of parallel streams (default: 1)
  • Replication Factor – Number of copies of each partition (default: 1)
  • Retention – How long messages are kept (default: 7 days, unless changed)

2️⃣ Producing Messages

Using Console Producer

bash.js
    bin/kafka-console-producer.sh --topic user-events --bootstrap-server localhost:9092

Type messages and press Enter to send them.

Useful Producer Options

  • Key separator – --property key.separator=: for key:value input
  • Acknowledgments – --property acks=all for durability
  • Batch Size – --property batch.size=16384 for performance tuning

3️⃣ Consuming Messages

Using Console Consumer

bash.js
    # Basic consumer
    bin/kafka-console-consumer.sh --topic user-events --from-beginning --bootstrap-server localhost:9092
    
    # Consumer with consumer group
    bin/kafka-console-consumer.sh --topic user-events --group my-group --bootstrap-server localhost:9092
    
    # Consumer printing keys and values
    bin/kafka-console-consumer.sh --topic user-events --from-beginning --property print.key=true --bootstrap-server localhost:9092

Useful Consumer Options

  • Read from beginning – --from-beginning
  • Use consumer group – --group group-name to enable group coordination
  • Print keys – --property print.key=true to see keys + values

🧪 Hands-On Exercises

These exercises will make you comfortable with real Kafka commands.

📝 Exercise 1: Basic Topic Operations

Goal: Create, inspect, and delete topics.

Tasks:

  1. Create a topic called orders with 3 partitions
  2. Create a topic called payments with 2 partitions
  3. List all topics
  4. Describe the orders topic
  5. Delete the payments topic

Commands:

bash.js
    # 1. Create orders topic
    bin/kafka-topics.sh --create --topic orders --bootstrap-server localhost:9092 --partitions 3 --replication-factor 1
    
    # 2. Create payments topic
    bin/kafka-topics.sh --create --topic payments --bootstrap-server localhost:9092 --partitions 2 --replication-factor 1
    
    # 3. List topics
    bin/kafka-topics.sh --list --bootstrap-server localhost:9092
    
    # 4. Describe orders topic
    bin/kafka-topics.sh --describe --topic orders --bootstrap-server localhost:9092
    
    # 5. Delete payments topic
    bin/kafka-topics.sh --delete --topic payments --bootstrap-server localhost:9092

🧪 Exercise 2: Producer–Consumer Workflow

Goal: Send and receive messages across topics.

Tasks:

  1. Start a consumer in one terminal
  2. Start a producer in another terminal
  3. Send messages with keys
  4. Observe what is received by the consumer

Terminal 1 – Consumer:

bash.js
    bin/kafka-console-consumer.sh --topic orders --from-beginning --property print.key=true --bootstrap-server localhost:9092

Terminal 2 – Producer:

bash.js
    bin/kafka-console-producer.sh --topic orders --property key.separator=: --bootstrap-server localhost:9092

Test Messages (type into producer):

code
    user1:Order placed for laptop
    user2:Order placed for mouse
    user1:Order updated - laptop
    user3:Order placed for keyboard

🧪 Exercise 3: Consumer Groups in Action

Goal: See load balancing and rebalancing in a real group.

Tasks:

  1. Create a topic with 3 partitions
  2. Start multiple consumers using the same group id
  3. Send messages and watch how they are distributed
  4. Add/remove consumers and observe rebalancing

Commands:

bash.js
    # Create topic
    bin/kafka-topics.sh --create --topic user-actions --bootstrap-server localhost:9092 --partitions 3 --replication-factor 1
    
    # Consumer 1
    bin/kafka-console-consumer.sh --topic user-actions --group analytics --bootstrap-server localhost:9092
    
    # Consumer 2 (new terminal)
    bin/kafka-console-consumer.sh --topic user-actions --group analytics --bootstrap-server localhost:9092
    
    # Consumer 3 (new terminal)
    bin/kafka-console-consumer.sh --topic user-actions --group analytics --bootstrap-server localhost:9092

Now produce messages to user-actions and watch how they get split across consumers.

🗂️ Configuration Files

Understanding basic configs helps when you move beyond default setups.

Server Properties – config/server.properties

properties.js
    # Broker ID
    broker.id=0
    
    # Listeners
    listeners=PLAINTEXT://localhost:9092
    
    # Log directories
    log.dirs=/tmp/kafka-logs
    
    # Default replication factor
    default.replication.factor=1
    
    # Default partitions
    num.partitions=1
    
    # Log retention
    log.retention.hours=168
    log.retention.bytes=1073741824
    
    # Zookeeper connection (Zookeeper mode)
    zookeeper.connect=localhost:2181

Producer Properties

properties.js
    # Bootstrap servers
    bootstrap.servers=localhost:9092
    
    # Serializers
    key.serializer=org.apache.kafka.common.serialization.StringSerializer
    value.serializer=org.apache.kafka.common.serialization.StringSerializer
    
    # Acknowledgments
    acks=all
    
    # Retries
    retries=3
    
    # Batch size
    batch.size=16384
    
    # Compression
    compression.type=lz4

Consumer Properties

properties.js
    # Bootstrap servers
    bootstrap.servers=localhost:9092
    
    # Deserializers
    key.deserializer=org.apache.kafka.common.serialization.StringDeserializer
    value.deserializer=org.apache.kafka.common.serialization.StringDeserializer
    
    # Group ID
    group.id=my-consumer-group
    
    # Auto offset reset
    auto.offset.reset=latest
    
    # Enable auto commit
    enable.auto.commit=true
    auto.commit.interval.ms=5000

📈 Monitoring and Management

Kafka includes built-in tools to inspect consumers, topics, and logs.

Consumer Group Management

bash.js
    # List consumer groups
    bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list
    
    # Describe a consumer group
    bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group analytics --describe
    
    # Reset consumer group offsets
    bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group analytics --reset-offsets --to-earliest --topic user-actions --execute

Log and Offset Management

bash.js
    # Check log directories and usage
    bin/kafka-log-dirs.sh --bootstrap-server localhost:9092 --describe
    
    # Get topic offsets
    bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list localhost:9092 --topic user-actions

JMX Metrics

Kafka exposes many metrics via JMX, such as:

  • Throughput – Messages per second
  • Latency – End-to-end or per-request
  • Consumer Lag – Delay between latest and processed messages
  • Disk Usage – Per broker/topic

These can be scraped into Prometheus/Grafana or other monitoring tools.

🩺 Troubleshooting Common Issues

Issue 1: "Topic already exists"

bash.js
    # Check existing topics
    bin/kafka-topics.sh --list --bootstrap-server localhost:9092
    
    # Delete and recreate if needed
    bin/kafka-topics.sh --delete --topic my-topic --bootstrap-server localhost:9092

Issue 2: "Consumer group not found"

bash.js
    # List all consumer groups
    bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --list
    
    # Describe a specific group
    bin/kafka-consumer-groups.sh --bootstrap-server localhost:9092 --group my-group --describe

Issue 3: "Connection refused"

  • Check if Kafka is running: ps aux | grep kafka
  • Check port availability: netstat -tlnp | grep 9092
  • Check firewall settings
  • Verify bootstrap.servers configuration matches where Kafka is listening

✅ Best Practices

For Development

  1. Use Docker – Easy to start/stop and reset state
  2. Single Broker – Enough for local development
  3. Replication Factor = 1 – No need for replication in dev
  4. Short Retention – Save disk space locally

For Production

  1. Multiple Brokers – For fault tolerance and scalability
  2. Replication Factor ≈ 3 – For durability
  3. Proper Monitoring – JMX + dashboards + alerts
  4. Security – SSL/TLS, authentication, and authorization
  5. Resource Planning – CPU, memory, disk, and network capacity

🚀 Next Steps

Now that you have Kafka running locally and can:

  • Create and manage topics
  • Produce and consume messages
  • Work with consumer groups
  • Use basic monitoring commands

You’re ready to:

  1. Integrate Kafka into applications using official client libraries
  2. Build real microservices that communicate via Kafka topics
  3. Experiment with scaling – more brokers, more partitions, more consumers
  4. Add observability – dashboards, metrics, and alerts for your Kafka cluster

This hands-on experience is the foundation for using Kafka confidently in real-world systems.

Hands-on Examples

Docker Compose Setup

# docker-compose.yml
      version: '3.8'
    
      services:
        zookeeper:
          image: confluentinc/cp-zookeeper:latest
          hostname: zookeeper
          container_name: zookeeper
          ports:
            - "2181:2181"
          environment:
            ZOOKEEPER_CLIENT_PORT: 2181
            ZOOKEEPER_TICK_TIME: 2000
          volumes:
            - zookeeper-data:/var/lib/zookeeper/data
            - zookeeper-logs:/var/lib/zookeeper/log
    
        kafka:
          image: confluentinc/cp-kafka:latest
          hostname: kafka
          container_name: kafka
          depends_on:
            - zookeeper
          ports:
            - "9092:9092"
            - "9101:9101"
          environment:
            KAFKA_BROKER_ID: 1
            KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
            KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
            KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
            KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
            KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
            KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
            KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
            KAFKA_JMX_PORT: 9101
            KAFKA_JMX_HOSTNAME: localhost
          volumes:
            - kafka-data:/var/lib/kafka/data
    
        kafka-ui:
          image: provectuslabs/kafka-ui:latest
          container_name: kafka-ui
          depends_on:
            - kafka
          ports:
            - "8080:8080"
          environment:
            KAFKA_CLUSTERS_0_NAME: local
            KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:29092
            KAFKA_CLUSTERS_0_ZOOKEEPER: zookeeper:2181
    
      volumes:
        zookeeper-data:
        zookeeper-logs:
        kafka-data:
    
      # Start the environment
      docker-compose up -d
    
      # Check status
      docker-compose ps
    
      # View logs
      docker-compose logs kafka
    
      # Stop the environment
      docker-compose down

This Docker Compose setup provides a complete Kafka development environment with management UI accessible at http://localhost:8080.

Previous
Module 5: Consumer Groups in Kafka
Next
Module 7: Kafka with Python

Related Tutorials