Skip to main content

Command Palette

Search for a command to run...

Kafka for People Who Understand It in Theory

No buzzwords, no hand-waving, just Kafka explained through one small project

Updated
β€’22 min readβ€’View as Markdown
Kafka for People Who Understand It in Theory
D
I'm a mobile/web developer πŸ‘¨β€πŸ’» who loves to build projects and share valuable tips for programmers Follow me for Flutter, React/Next.js, and other awesome tech-related stuff πŸ˜‰

For many of us, Kafka lives in our head as a vague shape. I knew the sentence people say about it like it's a distributed event streaming platform and blah...blah..blah. I could repeat it in a conversation and nobody would stop me. But if you had asked me what a partition actually does, or why two consumers might see the same message twice, I would have started a sentence and quietly hoped you'd interrupt me.

So I built something small to force the issue. It's called MetricPulse. It's a tiny monitoring pipeline: fake servers report their CPU and memory, Kafka holds those readings, and a React dashboard updates live in the browser without refreshing anything.

This post is the version of the explanation I wish I'd found when I started. No prior Kafka knowledge assumed. If you know what a function is and you've seen a terminal before, you're qualified.

Here is what we're building:

producer  ->  Kafka  ->  WebSocket bridge  ->  React dashboard
                  \
                   ->  alerter (separate consumer)

Let's start with why any of this exists.

The problem nobody explains before showing you Kafka

Imagine you run 50 servers. Every few seconds each one reports how it's doing: CPU at 42%, memory at 78%, that kind of thing. This stream of health numbers has a name: telemetry. It's just time-series data, meaning "a number attached to a moment in time."

Now, where do those numbers go?

The obvious answer is: build an HTTP API and have every server POST to it. That works on day one. It gets uncomfortable by month three.

The first problem is that one API becomes a bottleneck. All 50 machines hammer a single endpoint.

The second problem is that if the API goes down, the metrics are gone. Not delayed. Gone.

The third problem is today one service reads those metrics. Tomorrow you want another and then another...Every new consumer means changing that same endpoint, and now four teams are stepping on each other in the same file.

The fix is to stop having producers talk to consumers directly and put something in the middle. Producers write to it and move on. Consumers read from it whenever they're ready, at their own pace, without knowing or caring who else is reading.

That middle piece is Apache Kafka.

The mental model that finally made it click

Kafka documentation throws five nouns at you in the first paragraph. Here's the version that stuck for me. And these are the most important topics you need to understand and then you are all set.

A topic: Think of it as a mailbox with a name. In our example its gonna be called infra-metrics. Everything about server health goes in this one mailbox.

A producer is anyone dropping letters into the mailbox. Our servers.

A consumer is anyone picking letters up. Ex: Our alerter, our dashboard bridge in this example.

A partition is a lane inside that mailbox. If the mailbox has three lanes, three workers can sort mail at the same time instead of forming a queue behind one another. Partitions are how Kafka goes fast.

An offset is a line number inside a lane. Letter 0, letter 1, letter 2, and so on. It's also a bookmark: a consumer can say "I've read up to line 47 in lane 2," walk away, come back tomorrow, and resume at 48.

The important thing about this post office: picking up a letter does not remove it. Kafka keeps messages around for a configured retention period regardless of who has read them. That's the whole trick. It's why ten different consumers can each read the full stream without interfering with each other, and why a consumer that crashes can catch up instead of losing data.

Everything else in this post is an implementation detail on top of that idea.

Part 1: Getting Kafka running without installing Kafka

Let's get our hands dirty and understand by an example. So, I did not install Kafka on my machine. I ran it with Docker Compose.

So Kafka needs two doors into the same building. One door advertised as localhost:9092 for apps on my machine, and one advertised as kafka:29092 for apps inside the Docker network, where kafka is the service name Docker resolves for you.

In Kafka terms, a door is called a listener. Here's the Compose file:

services:
  kafka:
    image: apache/kafka:3.8.1
    container_name: metricpulse-kafka
    ports:
      # Host apps (producer/consumer on your Mac) use localhost:9092
      - "9092:9092"
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,PLAINTEXT_DOCKER://0.0.0.0:29092,CONTROLLER://0.0.0.0:9093
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092,PLAINTEXT_DOCKER://kafka:29092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_DOCKER:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT_DOCKER
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@localhost:9093
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "/opt/kafka/bin/kafka-broker-api-versions.sh --bootstrap-server localhost:9092 >/dev/null 2>&1",
        ]
      interval: 5s
      timeout: 10s
      retries: 10
      start_period: 15s

That's a wall of environment variables, so let me walk the ones that matter.

KAFKA_LISTENERS is where Kafka actually binds and listens. 0.0.0.0 means "accept connections on any network interface." Three doors here: 9092, 29092, and 9093.

KAFKA_ADVERTISED_LISTENERS is the one that confused me, and it's the important one. This is the address Kafka tells clients to use. A client connects once to say hello, Kafka replies "great, now talk to me at this address," and the client reconnects there. If you advertise the wrong address, your client connects successfully and then immediately fails on the second hop, which produces a bewildering error message. Host apps get told localhost:9092. Container apps get told kafka:29092.

KAFKA_LISTENER_SECURITY_PROTOCOL_MAP just tells Kafka that all three of these doors are unencrypted plaintext. Fine for local learning. Not fine for production.

KAFKA_PROCESS_ROLES: broker,controller plus the CONTROLLER listener means this single container is playing both roles. Older Kafka setups needed a separate ZooKeeper container for coordination. Modern Kafka handles it internally with something called KRaft, which is why you'll see older tutorials with an extra service that you no longer need.

All the REPLICATION_FACTOR: 1 settings are saying "there is one broker, don't try to make copies on other brokers, there aren't any." In production these would be 3.

The healthcheck runs a Kafka CLI command in a loop until it succeeds. This matters later, because it lets other containers wait for Kafka to actually be ready rather than merely started.

Start it:

docker compose up -d

Now let's create a topic infra-metrics.

docker exec -it metricpulse-kafka /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 \
  --create \
  --topic infra-metrics \
  --partitions 3 \
  --replication-factor 1

Three partitions means three lanes, so up to three consumers in a group can work in parallel. Replication factor 1 means one copy, because we have one broker.

To see whether it is created run below command:

docker exec -it metricpulse-kafka /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 \
  --list

you should see infra-metrics.

Part 2: The producer, or writing metrics into Kafka

A producer is just a program that writes messages to a topic. Mine pretends to be three servers and sends a CPU and memory reading for each, every two seconds. I used Node.js with KafkaJS.

Start with the connection: createsrc/producer.js. Make sure to install kafkajs package.

import { Kafka, logLevel } from "kafkajs"

// 1) Who are we talking to?
// "brokers" = list of Kafka servers. Locally we have one on port 9092.
const kafka = new Kafka({
    clientId: "metricpulse-producer",
    brokers: ["localhost:9092"],
    logLevel: logLevel.ERROR,
})

// 2) Create a producer object (does not connect yet)
const producer = kafka.producer()

clientId is a label that shows up in Kafka's logs and metrics. It doesn't affect behavior, but when something breaks at 2am you'll be glad your clients have names.

brokers is a starting point, not the full picture. You give Kafka one address, it tells you about the rest of the cluster. This is called the bootstrap server, and it's why the CLI flag is --bootstrap-server rather than --kafka-address.

logLevel.ERROR is me turning down the noise so I could see my own console logs.

Now the fake data:

// Fake servers (in real life these would be real hosts
const HOSTS = ["api-server-1", "api-server-2", "db-server-1"]

// Helper: random number between min and max
function randomBetween(min, max) {
    return Number((Math.random() * (max - min) + min).toFixed(1))
}

// Build one metric reading
function createMetric(host, metricName, value, unit) {
    return {
        host,
        metric: metricName,
        value,
        unit,
        timestamp: new Date().toISOString()
    }
}

That object shape is worth pausing on, because it's the actual contract of the whole system. Kafka itself stores bytes. It has no opinion about your data.

Read it as a sentence: who (host), what (metric), the number (value), what the number means (unit), and when (timestamp). Every time-series point in every monitoring system you'll ever touch is some variation of this.

// Create a small batch of metrics for all hosts
function createBatch() {
    const batch = []

    for (const host of HOSTS) {
        // sometimes spike CPU so a consumer can alert
        const cpu = Math.random() < 0.75 ? randomBetween(90, 99) : randomBetween(10, 85)

        batch.push(createMetric(host, "cpu_usage", cpu, "percent"))
        batch.push(
            createMetric(host, "memory_usage", randomBetween(30, 90), "percent")
        )
    }
    return batch
}

Now the sending:

async function start() {
  // 3) Connect session to Kafka
    await producer.connect()
    console.log("Producer connected to kafka")

  // 4) Every 2 seconds, send a batch
    setInterval(async () => {
        const metrics = createBatch()
        // WHY KEY MATTERS:
            // Same key β†’ same partition β†’ order preserved for that host.
            // Different keys β†’ can go to different partitions β†’ more parallelism.
        const messages = metrics.map((metric) => ({
            key: metric.host,
            value: JSON.stringify(metric)
        }))

        try {
            const result = await producer.send({
                topic: "infra-metrics",
                messages
            })

            console.log(`Sent ${messages.length} metrics | partitions used: `,
                result.map((r) => r.partition)
            )
        } catch (err) {
            console.log("Failed to send metrics: ", err.message)
        }
    }, 2000)
}

// If something blows up, print it and exit
start().catch((err) => {
    console.log("Producer crashed: ", err)
    process.exit(1)
})

JSON.stringify is not optional. Kafka message values are bytes. You cannot hand it a JavaScript object and hope. Serialize going in, parse coming out. Bigger systems replace JSON with formats like Avro or Protobuf that enforce a schema, but the principle is identical.

key: metric.host is the line that took me longest to appreciate. The key is a routing hint. Kafka roughly computes hash(key) % numberOfPartitions to pick a lane. Same key means same hash means same lane, every time.

Why does that matter? Because Kafka only guarantees ordering within a partition, not across the whole topic. By keying on hostname, every reading from api-server-1 lands in the same lane and arrives in the order it was sent. Without a key, Kafka spreads messages around and api-server-1's 10am reading could get processed after its 10:01 reading. For metrics that's annoying. For bank transactions it's a lawsuit.

Now to run it, Make sure Kafka is still up:

docker ps
node src/producer.js

You should see something like:

Producer connected to kafka
Sent 6 metrics | partitions used: [ 1, 0 ]

Part 3: The consumer, plus the three words that unlock Kafka

Now let's create a consumer which will consume the data of the producer. create src/consumer.js

A consumer reads messages and does something with them. Mine prints normal readings and tells about high CPU.

import { Kafka, logLevel } from "kafkajs"

const kafka = new Kafka({
    clientId: "metricpulse-consumer",
    brokers: ["localhost:9092"],
    logLevel: logLevel.ERROR
})

const CPU_ALERT_THRESHOLD = 90

const consumer = kafka.consumer({
    groupId: "metricpulse-alerters"
})

Everything here is familiar except groupId, which is the single most important concept in this whole post. I'll come back to it in a second.

The handler:

function processMetric(metric, meta) {
    const where = `partition=${meta.partition} offset=${meta.offset}`

    const isCpu = metric.metric === "cpu_usage"

    if (isCpu && metric.value >= CPU_ALERT_THRESHOLD) {
        console.log(`ALERT high CPU | host=${metric.host} value=${metric.value}% | ${where}`)
        return
    }

    console.log(`ok | ${metric.host} ${metric.metric}=${metric.value}${metric.unit === "percent" ? "%" : ""} ${where}`)
}

Printing partition and offset in every log line turned out to be one of the better choices I made. When something looks wrong, those two numbers tell you exactly which lane and which line number you're looking at, which turns "the data is weird" into a question you can actually investigate.

Now the subscription and the run loop:

async function start() {
    // connect consumer to kafka
    await consumer.connect()
    console.log("Consumer connected to Kafka")

    // subscribe to the topic
    await consumer.subscribe({
        topic: "infra-metrics",
        fromBeginning: false
    })

     // Kafka loop: poll kafka -> hand you one message -> you process it
    await consumer.run({
        eachMessage: async ({ topic, partition, message }) => {
            const raw = message?.value.toString()

            if (!raw) {
                console.log("Skipped empty message")
                return
            }

            try {
                // Producer stored a string. Consumer turns it back into an object
                const metric = JSON.parse(raw)

                processMetric(metric, {
                    topic,
                    partition,
                    offset: message.offset,
                    key: message.key?.toString()
                })
            } catch (err) {
                console.log("Bad message, skipping: ", err.message, "raw=", raw)
            }
        }
    })
}

message.value.toString() is JSON.stringify in reverse. Bytes come out, you turn them into a string, then parse.

The try/catch around JSON.parse is not defensive paranoia, it's necessary. Kafka will happily accept a malformed message from some other producer you don't control, and if one bad message can crash your consumer, you've built a system where anyone can take down your alerting by typing badly. Log it, skip it, keep going.

Consumer groups, offsets, and the fromBeginning flag

Here's the part that reorganized my understanding.

A consumer group is a team badge. groupId: "metricpulse-alerters" tells Kafka: this process is a member of the alerters team. Kafka then tracks bookmarks under that team name, not per process.

Two rules follow from that, and together they are most of what Kafka does for you:

Same group ID means share the work. Start three consumers with the same group ID on a three partition topic, and Kafka hands each one a different partition. Each message gets processed once by the team. Kill one, and Kafka reassigns its partitions to the survivors automatically. That's your horizontal scaling and your failover, both free.

Different group ID means an independent full copy of the stream. My alerter is in metricpulse-alerters. My dashboard bridge is in metricpulse-dashboard. Both receive every single message. Neither knows the other exists. This is why you can add an analytics consumer, an archiver, and a machine learning pipeline later without touching a single line of producer code. That decoupling is the entire selling point.

Offset = Think of one partition as a notebook with numbered lines. It only means something inside one partition. Kafka assigns it when the message is stored.

partition 0 of infra-metrics
─────────────────────────────
offset 0:  {"host":"api-server-1","metric":"cpu-usage",...}
offset 1:  {"host":"api-server-1","metric":"memory_usage",...}
offset 2:  {"host":"api-server-2","metric":"cpu-usage",...}
offset 3:  ...
offset 4:  ...   ← newest messages keep getting higher numbers

And fromBeginning, which I misunderstood completely at first. I assumed false meant "never replay old messages, ever." It doesn't. This flag only applies when the group has no bookmark yet, meaning the very first time that group ID ever connects.

  • false on a brand new group: start at the live end, ignore history.

  • true on a brand new group: replay from the earliest message Kafka still has.

  • Either value on an existing group: irrelevant. Kafka resumes from the committed offset.

Once that clicked, a lot of behavior I'd found spooky became boring, which is the goal.

By the way, if you want to prove the replay claim to yourself: stop the consumer, let the producer run for a minute, then start the consumer again. It floods with everything it missed. That moment is when Kafka stopped being abstract for me. The data was just sitting there, waiting.

Part 4: WebSockets, or how the browser gets told instead of asking

Kafka now holds a live stream. Getting it onto a screen is a separate problem.

The default approach is polling: the browser asks the server "anything new?" every two seconds. It works, and it's wasteful. You're mostly sending requests that come back empty, and whatever you learn is up to two seconds stale.

A WebSocket flips the direction. The browser opens a connection once and holds it open. Then the server can push data down whenever it feels like it, with no request needed. HTTP is knocking on a door repeatedly. A WebSocket is leaving the phone line open.

So I built a bridge (ws-server.js) that does three things: accept browser connections on port 8080, consume Kafka with its own group, and broadcast each metric to every connected client.

Create ws-server.js. Make sure to npm install ws

import { WebSocketServer } from "ws"
import { Kafka, logLevel } from "kafkajs"

// listens for WS connections
const wss = new WebSocketServer({ port: 8080 })

// connection event => fires when a client connects
wss.on("connection", (socket) => {
    console.log("Client connected. Total clients: ", wss.clients.size)

    // push to one client
    socket.send(
        JSON.stringify({
            type: "welcome",
            message: "Connected to MetricPulse live feed"
        })
    )

    socket.on("close", () => {
        console.log("Client disconnected. Total clients: ", wss.clients.size)
    })
})

The welcome message exists so the browser gets immediate proof the connection is alive, rather than staring at a blank screen wondering whether it's broken or just waiting.

Notice it carries a type field. Every message I send has one, which lets the client route messages by type instead of guessing from their shape.

// push to all clients - how a live dashboard feed works
function broadcast(data) {
    const payload = JSON.stringify(data)

    for (const client of wss.clients) {
        if (client.readyState === 1) {
            client.send(payload)
        }
    }
}

readyState === 1 means OPEN. A socket can be in the middle of closing while still sitting in that set, and sending to it throws. I stringify once outside the loop rather than per client.

Then the Kafka side, which has a retry loop:

const consumer = kafka.consumer({
    groupId: "metricpulse-dashboard"
})

async function startKafkaConsumer() {
    const maxAttempts = 30

    for (let attempt = 1; attempt <= maxAttempts; attempt++) {
        try {
            await consumer.connect()
            await consumer.subscribe({ topic: "infra-metrics", fromBeginning: false })

            await consumer.run({
                eachMessage: async ({ message }) => {
                    const raw = message.value?.toString()
                    if (!raw) return

                    let metric
                    try {
                        metric = JSON.parse(raw)
                    } catch (err) {
                        console.log("Bad kafka message, skipping: ", err.message)
                        return
                    }

                    broadcast({
                        type: "metric",
                        host: metric.host,
                        metric: metric.metric,
                        value: metric.value,
                        unit: metric.unit,
                        timestamp: metric.timestamp
                    })
                }
            })

            return
        } catch (err) {
            console.log(`Kafka bridge attempt ${attempt}/${maxAttempts} failed: ${err.message}`)

            try {
                await consumer.disconnect()
            } catch {
                // ignore disconnect errors while retrying
            }

            if (attempt === maxAttempts) throw err
            await new Promise((resolve) => setTimeout(resolve, 2000))
        }
    }
}

Three things I care about here.

The group ID is metricpulse-dashboard, deliberately different from the alerter's. That's the "different group means independent copy" rule doing real work. Both consumers see every metric. If I'd reused the alerter's group ID, they would have split partitions between them and my dashboard would show roughly half the data while my alerts fired on the other half, which is a bug that would have taken me an embarrassingly long evening to find.

The retry loop exists because Kafka takes a few seconds to become ready, and once this bridge is running in Docker alongside it.

Also worth calling out: the brokers list reads from an environment variable.

brokers: [process.env.KAFKA_BROKERS || "localhost:9092"]

Same code, two environments. On my Mac it falls back to localhost:9092. Inside Docker I pass kafka:29092. This one line is the practical version of the two listeners lesson from Part 1, and it's also the exact pattern that makes a service deployable anywhere later.

The path is now complete:

producer -> Kafka -> bridge -> WebSocket clients

Run Step 1:

node src/ws-server.js

You should see:

WebSocket server listening on ws://localhost:8080
Broadcast: api-server-1 cpu-usage 54.2
...

Part 5: The React dashboard

The UI opens a WebSocket, folds incoming events into per host state, and renders a card for each host. I'm using Vite here with React configurations.

First the card, which is deliberately dumb:

function MetricCard({ host, cpu, memory, updatedAt }) {
  const cpuHot = cpu !== null && cpu >= 90;

  return (
    <section className="card">
      <h2>{host}</h2>

      <div className="metrics-row">
        <div>
          <p className="label">CPU</p>
          <p className={`value ${cpuHot ? "hot" : ""}`}>
            {cpu === null ? "β€”" : `${cpu}%`}
          </p>
        </div>

        <div>
          <p className="label">Memory</p>
          <p className="value">{memory === null ? "β€”" : `${memory}%`}</p>
        </div>
      </div>

      <p className="meta">
        {updatedAt
          ? `Updated ${new Date(updatedAt).toLocaleTimeString()}`
          : "Waiting for first metric…"}
      </p>
    </section>
  );
}

export default MetricCard;

This component knows nothing about Kafka or WebSockets. It receives four props and renders them. Three servers on screen, one component, three sets of props.

The cpu === null checks matter more than they look. Metrics arrive one field at a time, so there's a real moment where a host has a CPU number but no memory number yet.

Now the state management, which is where the interesting decision lives:

function App() {
  const [status, setStatus] = useState("connecting");
  const [welcome, setWelcome] = useState("");
  // host -> { cpu, memory, updatedAt }
  const [hosts, setHosts] = useState({});

  useEffect(() => {
    const ws = new WebSocket("ws://localhost:8080");

    ws.onopen = () => setStatus("open");
    ws.onclose = () => setStatus("closed");
    ws.onerror = () => setStatus("error");
    
   // on getting message setting up the data
    ws.onmessage = (event) => {
      let data;
      try {
        data = JSON.parse(event.data);
      } catch {
        console.log("Non-JSON message:", event.data);
        return;
      }

      if (data.type === "welcome") {
        setWelcome(data.message);
        return;
      }

      if (data.type !== "metric") return;

      setHosts((prev) => {
        const current = prev[data.host] || {
          cpu: null,
          memory: null,
          updatedAt: null,
        };

        const next = { ...current, updatedAt: data.timestamp };

        if (data.metric === "cpu_usage") {
          next.cpu = data.value;
        }
        if (data.metric === "memory_usage") {
          next.memory = data.value;
        }

        return {
          ...prev,
          [data.host]: next,
        };
      });
    };

    return () => {
      ws.close();
    };
  }, []);
  // ...
}

Why store hosts as an object instead of an array? Because messages arrive as individual fields, not complete snapshots. A CPU reading for api-server-1 must not erase that host's memory value. Keying by hostname makes "find this host and update one field" a direct lookup instead of a search

Part 6: Putting the bridge in a container

Kafka was already containerized. Packaging my own Node service was the last piece. Create Dockerfile

FROM node:22-alpine
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci --omit=dev
COPY src ./src
EXPOSE 8080
CMD ["node", "src/ws-server.js"]

Line by line.

node:22-alpine is the base layer, Node 22 on Alpine Linux. Alpine is chosen because it's tiny, which means faster builds and pulls.

WORKDIR /app creates and moves into that directory. Every path after this is relative to it.

The next two lines are the one Docker trick worth internalizing early. I copy package.json and package-lock.json first, install dependencies, and only then copy my source code.

npm ci rather than npm install installs exactly what the lockfile specifies, no version drift. --omit=dev skips dev dependencies, because the container doesn't need my test runner.

EXPOSE 8080 is documentation. It declares intent but doesn't actually publish the port. The ports mapping in Compose does that.

CMD is what runs when the container starts.

And the .dockerignore, which is easy to skip and shouldn't be:

node_modules
dashboard
npm-debug.log
.git

Then the bridge joins Compose: (docker-compose.yml)

  bridge:
    build: .
    image: metricpulse-bridge:local
    container_name: metricpulse-bridge
    ports:
      - "8080:8080"
    environment:
      # Use the Docker-network listener, not localhost
      KAFKA_BROKERS: kafka:29092
    depends_on:
      kafka:
        condition: service_healthy
    restart: unless-stopped

build: . means build from the Dockerfile here rather than pulling a published image.

KAFKA_BROKERS: kafka:29092 is the payoff for that environment variable in Part 4. The bridge is inside the Docker network now, so it uses the Docker door.

depends_on with condition: service_healthy is the healthcheck from Part 1 finally earning its keep. Plain depends_on only waits for the container to start, which for Kafka is several seconds before it's actually usable. service_healthy waits for the healthcheck to pass.

restart: unless-stopped means Docker brings it back if it crashes, but respects me deliberately stopping it.

One mental model that helped: an image is the package, a container is a running instance of that package. You build once and run many.

Running the whole thing

# 1) Kafka + bridge
docker compose up -d --build

# 2) Producer (on your machine)
node src/producer.js

# 3) Optional alerter (on your machine)
node src/consumer.js

# 4) Dashboard
cd dashboard && npm run dev

Open the Vite URL. Cards should populate within a couple of seconds, bridge logs should scroll past, and the alerter terminal should start complaining about CPU.

Now stop the producer. The dashboard freezes on the last values it received. Start it again. Updates resume. The data really is flowing through a broker, and the pieces really are independent.

The actual lesson

Reading documentation gives you definitions. You can hold definitions in your head for a year without them connecting to anything.

Building forces the weird parts into the open. Why did only two of my three partitions get traffic? Where exactly does WebSocket cleanup belong? Why does the group ID matter so much more than it looks? None of those questions occurred to me while reading. All of them occurred within an hour of writing code.

This example is deliberately small. But small enough to hold entirely in my head, big enough that the architecture is genuinely the same shape as what real monitoring systems do.

If you're stuck on Kafka, don't read another overview. Build the smallest thing that has a producer and two consumers, and let it break on you.