Right now, Orders does two separate things:

• save the order in Postgres as PENDING

• publish OrderCreated to Kafka

If saving to Postgres works but publishing to Kafka fails, the order gets stuck. It exists in the database, but no other service knows about it. That means the workflow stops halfway through. This is a classic reliability problem in distributed systems. In this part, we will solve it using the Outbox pattern.

What the Outbox pattern does

The Outbox pattern changes the workflow so that saving data and saving the event happen together.

Before (current flow)

Right now, the flow looks like this:

• write the order to the database

• publish the event to Kafka

That sounds fine at first, but there is a dangerous gap between those two steps.

If the database write succeeds but Kafka publish fails, the order is saved, but the event is lost. That leaves the system in an inconsistent state.

After (with the Outbox pattern)

With the Outbox pattern, the flow changes to this:

• write the order to the database

• write the event into a database table called outbox_events

• do both of those in the same database transaction

• a background worker later reads from outbox_events and publishes to Kafka

This is the key idea:

instead of publishing directly to Kafka inside the request flow, we first save the event safely in the database

We are not changing Inventory in this part. Inventory already works fine once it receives the event. The problem is “how to guarantee the event gets published”.

Step 1: Add migrations for the outbox table

The first thing we need for the Outbox pattern is a new database table to store events before they are published to Kafka.

Create this migration file:

services/orders/migrations/000002_outbox.up.sql

CREATE TABLE IF NOT EXISTS outbox_events (
  id BIGSERIAL PRIMARY KEY,
  event_id TEXT NOT NULL UNIQUE,
  topic TEXT NOT NULL,
  key TEXT NOT NULL,
  payload BYTEA NOT NULL,
  created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  published_at TIMESTAMPTZ NULL
);

CREATE INDEX IF NOT EXISTS idx_outbox_unpublished
  ON outbox_events(id)
  WHERE published_at IS NULL;

Now create the down migration:

services/orders/migrations/000002_outbox.down.sql

DROP TABLE IF EXISTS outbox_events;

What this table is for

This table stores events that still need to be published to Kafka.

Instead of trying to publish directly inside the request flow, we first write the event into outbox_events. Later, a background worker will read from this table and publish those events to Kafka.

So this table is basically a safe holding area for messages waiting to be published.

After this step, the Orders service has a place to store events safely in the database before they are sent to Kafka. That is the foundation of the Outbox pattern.

We have not changed the request flow yet, but now we have the table that makes the rest of the pattern possible.

Step 2: Add transaction support in Orders store

Right now, the Orders store probably inserts rows directly using the database pool. That works fine when we only insert one thing at a time. But for the Outbox pattern, we now need to do two inserts together:

• insert the order row

• insert the outbox event row

And we need both of those to happen inside the same database transaction.

That way:

• either both inserts succeed

• or neither one is saved

This is what makes the Outbox pattern reliable.

Open:

services/orders/internal/store/orders.go

Add these methods:

import "github.com/jackc/pgx/v5"

// Begin starts a DB transaction.
func (s *OrdersStore) Begin(ctx context.Context) (pgx.Tx, error) {
	return s.db.Begin(ctx)
}

// CreatePendingTx inserts an order inside a transaction as PENDING.
func (s *OrdersStore) CreatePendingTx(ctx context.Context, tx pgx.Tx, userID, note, sku string, quantity int) (int64, error) {
	var id int64
	err := tx.QueryRow(ctx,
		`INSERT INTO orders (user_id, note, status, sku, quantity)
		 VALUES (\(1, \)2, \(3, \)4, $5)
		 RETURNING id`,
		userID, note, StatusPending, sku, quantity,
	).Scan(&id)
	return id, err
}

Keep your existing Create and GetByID methods. We are not replacing them. We are only adding transaction-aware versions.

Why we need transactions here

The Outbox pattern depends on one important guarantee:

the order row and the outbox event row must be saved together

Without a transaction, this could happen:

1. order insert succeeds

2. outbox insert fails

Now the order exists, but the event does not. That is exactly the kind of broken state we are trying to avoid. With a transaction, the database treats both inserts as one unit of work.

So the result is:

• both are committed together

• or both are rolled back together

Step 3: Create an outbox package in Orders

Now we will create a small package to manage outbox events inside the Orders service.

Open a new folder:

mkdir -p services/orders/internal/outbox

Then create this file:

services/orders/internal/outbox/outbox.go

package outbox

import (
	"context"
	"time"

	"github.com/jackc/pgx/v5"
)

//This struct represents one row in the outbox_events table.
type Event struct {
	ID      int64 // database row ID
	EventID string // unique ID of the event
	Topic   string // Kafka topic to publish to
	Key     string // Kafka message key
	Payload []byte // raw event bytes, usually JSON in our case
}

//This function inserts a new row into outbox_events.
// The important part is that it takes a `pgx.Tx`, which means it runs inside an existing database transaction.
func InsertTx(ctx context.Context, tx pgx.Tx, evt Event) error {
	_, err := tx.Exec(ctx,
		`INSERT INTO outbox_events (event_id, topic, key, payload)
		 VALUES (\(1, \)2, \(3, \)4)`,
		evt.EventID, evt.Topic, evt.Key, evt.Payload,
	)
	return err
}

// This function fetches a batch of unpublished outbox rows.
func FetchUnpublishedForUpdate(ctx context.Context, tx pgx.Tx, limit int) ([]Event, error) {
	rows, err := tx.Query(ctx,
		`SELECT id, event_id, topic, key, payload
		 FROM outbox_events
		 WHERE published_at IS NULL
		 ORDER BY id
		 LIMIT $1
		 FOR UPDATE SKIP LOCKED`, // This locks the selected rows so another worker cannot take the same rows at the same time. If some rows are already locked by another worker, PostgreSQL just skips them and returns other available rows.
		limit,
	)
	if err != nil {
		return nil, err
	}
	defer rows.Close()

	var out []Event
	for rows.Next() {
		var e Event
		if err := rows.Scan(&e.ID, &e.EventID, &e.Topic, &e.Key, &e.Payload); err != nil {
			return nil, err
		}
		out = append(out, e)
	}
	return out, rows.Err()
}

//This function updates the outbox row after Kafka publishing succeeds.
func MarkPublishedTx(ctx context.Context, tx pgx.Tx, id int64) error {
	_, err := tx.Exec(ctx,
		`UPDATE outbox_events SET published_at = \(2 WHERE id = \)1`,
		id, time.Now().UTC(),
	)
	return err
}

This package is the database helper for the Outbox pattern. It gives us three main operations:

• insert a new outbox event inside a transaction

• fetch unpublished outbox events

• mark an event as published after Kafka publish succeeds

So instead of writing raw outbox SQL in many places, we keep it in one focused package.

Step 4: Add outbox publisher loop

Now that Orders can store unpublished events in the outbox_events table, we need something that actually reads those rows and publishes them to Kafka.

That job is handled by a background publisher loop.

Create this file:

services/orders/internal/outbox/publisher.go

package outbox

import (
	"context"
	"log"
	"time"

	"github.com/jackc/pgx/v5/pgxpool"
	"github.com/segmentio/kafka-go"
)

type Publisher struct {
	DB     *pgxpool.Pool // to read and update rows in the outbox table
	Writer *kafka.Writer //to publish messages to Kafka
}

// Every 300 milliseconds, the publisher wakes up and checks whether there are any unpublished events waiting in the outbox.
// Because outbox publishing is a background job. It should keep running while the service is running, and keep retrying unpublished events until they are successfully sent.
func (p *Publisher) Run(ctx context.Context) {
	ticker := time.NewTicker(300 * time.Millisecond)
	defer ticker.Stop()

	for {
		select {
		case <-ctx.Done():
			return
		case <-ticker.C:
			p.publishBatch(ctx)
		}
	}
}

// This function does one batch of outbox publishing work.
//You can think of it as: “Try to publish up to 25 unpublished events right now.”
func (p *Publisher) publishBatch(ctx context.Context) {
    // The publisher starts a database transaction before fetching rows.
	tx, err := p.DB.Begin(ctx)
	if err != nil {
		log.Printf("outbox begin tx failed: %v", err)
		return
	}
	defer func() { _ = tx.Rollback(ctx) }()
    
    // This reads up to 25 unpublished outbox rows.
	events, err := FetchUnpublishedForUpdate(ctx, tx, 25)
	if err != nil {
		log.Printf("outbox fetch failed: %v", err)
		return
	}
	if len(events) == 0 {
		_ = tx.Commit(ctx)
		return
	}

	for _, e := range events {
        // For every fetched outbox row. This sends the event to Kafka
		err := p.Writer.WriteMessages(ctx, kafka.Message{
			Topic: e.Topic,
			Key:   []byte(e.Key),
			Value: e.Payload,
		})
		if err != nil {
			// rollback => events stay unpublished, retry later
			log.Printf("outbox kafka publish failed: %v", err)
			return
		}
        // If Kafka publish succeeds, we then do. This updates published_at in the database.
		if err := MarkPublishedTx(ctx, tx, e.ID); err != nil {
			log.Printf("outbox mark published failed: %v", err)
			return
		}
	}
    
    // This final commit makes all those changes permanent.
	if err := tx.Commit(ctx); err != nil {
		log.Printf("outbox commit failed: %v", err)
	}
}

This file adds the background worker that makes the Outbox pattern actually work.

Its job is simple:

• look for unpublished events in outbox_events

• publish them to Kafka

• mark them as published

So instead of publishing directly during the HTTP request, Orders now has a dedicated background process for publishing safely.

Step 5: Update create-order handler to write to Outbox instead of Kafka directly

Where: services/orders/internal/http/http.go

Right now, the create-order handler publishes the event to Kafka directly using:

kafkabus.PublishOrderCreated(...)

In this part, we replace that direct publish with the Outbox flow.

The new sequence becomes:

• start a database transaction

• insert the order row

• insert the outbox event row

• commit the transaction

• return the response

That means the HTTP request no longer depends on Kafka being available immediately.

Now update the “create order” logic to:

// We begin a database transaction
tx, err := s.store.Begin(ctx)
if err != nil {
	http.Error(w, "db error", http.StatusInternalServerError)
	return
}
defer func() { _ = tx.Rollback(ctx) }()

// This creates the order row inside the transaction. At this point, we have a real order ID that we can use in the event.
id, err := s.store.CreatePendingTx(ctx, tx, req.UserID, req.Note, req.Sku, req.Quantity)
if err != nil {
	http.Error(w, "db error", http.StatusInternalServerError)
	return
}

// This creates the event payload we eventually want to publish to Kafka.
evt := kafkabus.OrderCreated{
	EventID:  uuid.New().String(),
	Type:     "OrderCreated",
	Time:     time.Now().UTC(),
	OrderID:  id,
	UserID:   req.UserID,
	Sku:      req.Sku,
	Quantity: int32(req.Quantity),
}

// This turns the event struct into bytes so it can be stored in the outbox_events table.
payload, _ := json.Marshal(evt)

// This stores the event inside the outbox_events table in the same transaction as the order row
if err := outbox.InsertTx(ctx, tx, outbox.Event{
	EventID: evt.EventID,
	Topic:   "orders.v1",
	Key:     strconv.FormatInt(id, 10),
	Payload: payload,
}); err != nil {
	http.Error(w, "db error", http.StatusInternalServerError)
	return
}

// Only after both inserts succeed do we commit the transaction.
if err := tx.Commit(ctx); err != nil {
	http.Error(w, "db error", http.StatusInternalServerError)
	return
}

writeJSON(w, http.StatusAccepted, createOrderResponse{ID: id})

In summary, we remove the direct Kafka publish from the create-order handler. Instead, we store the order row and the event row together inside one transaction, then return 202 Accepted. The actual Kafka publish is now handled later by the outbox worker.

Step 6: Start the outbox publisher from Orders main.go

Now that Orders can store events in the outbox_events table, we need to start the background publisher when the service boots.

Open: services/orders/cmd/orders/main.go

After creating the Kafka bus:

bus := kafkabus.New(...)

start the Outbox publisher:

pub := &outbox.Publisher{
	DB:     pool,
	Writer: bus.OrdersWriter,
}
go pub.Run(ctx)

This creates the background worker that continuously checks the outbox_events table for unpublished events.

Once it finds them, it:

• publishes them to Kafka

• marks them as published in the database

How to test Outbox

I encountered few problems which you might also. So there are few fixed which we need to do:

1. We need to remove Topic from the outbox message (the writer already targets orders.v1) in /orders/internal/outbox/publisher.go

err := p.Writer.WriteMessages(ctx, kafka.Message{
    // Topic: -- removed
	Key:   []byte(e.Key),
	Value: e.Payload,
})

1. Add red panda dependency in inventory service

inventory: 
  ...
    depends_on:
      redpanda:
        condition: service_healthy

That's it. Now lets go ahead and test it

This is the most important part of this section, because it proves that the Outbox pattern is actually solving the reliability problem we introduced earlier.

We want to verify two things:

• the normal flow still works

• events are not lost even when Kafka is temporarily down

Test 1: Normal flow still works

First, make sure the normal event-driven flow still behaves as expected.

Create an order and confirm that it eventually becomes:

• CONFIRMED

• or CANCELLED

This tells us that adding the Outbox pattern did not break the existing async workflow.

Test 2: Kafka-down test

This is the test that really proves the Outbox pattern is working.

Step 1: Stop Redpanda

docker compose stop redpanda

Now Kafka is unavailable.

Step 2: Create an order

curl -X POST http://localhost:8080/v1/orders \                                             
  -H 'Content-Type: application/json' \
  -d '{"userId":"u1","sku":"burger","quantity":1}'

The request should still succeed. That means:

• the order is created in Postgres

• the outbox event is also saved in Postgres

• but the event cannot be published yet because Kafka is down

So at this point, the order should exist in the database as:

• PENDING

And Inventory will not process it yet, because it never received the event.

This is the key difference from the old design.

Step 3: Start Redpanda again

docker compose start redpanda

Once Kafka comes back:

• The outbox publisher wakes up, it reads the unpublished event from outbox_events, it publishes that event to Kafka. Inventory consumes it. Inventory processes the stock reservation. Orders later receives the result and updates the order status

So the order should eventually move from:

PENDING -> CONFIRMED/ CANCELLED

Conclusion

At this point, QuickBite is starting to feel much more like a real production-style backend.

We solved one of the most common reliability problems in event-driven systems: what happens when Kafka is unavailable at the wrong time. With the Outbox pattern in place, Orders no longer risks losing the event just because the broker is temporarily down. The order and the event are stored safely together, and publishing can happen later. That is a big step forward.

But reliability is not finished yet. The next challenges are duplicates, retries, and bad messages. In the next part, we will introduce idempotency so repeated deliveries do not corrupt state, and we will also start making Inventory durable so it can survive restarts properly.

In this part, we improve that design by introducing Kafka. We will use Redpanda locally since it is Kafka-compatible and simple to run. We will understand what Kafka and Redpanda is.

Instead of making Orders wait for Inventory, we will switch to an event-driven flow. So, Orders will publish an event, Inventory will consume it, and then Inventory will publish the result.

We keep the gRPC server running in Inventory because it is still useful for learning and future steps, but in Part 5 the main workflow is Kafka-based.S

This is the point where the project starts to feel like a real distributed system.

What we are building in this part

In this part, we will change the order flow from synchronous to asynchronous.

Before (Part 4)

The flow looked like this:

• Orders creates the order as PENDING

• Orders calls Inventory directly using gRPC

• Orders immediately updates the order to CONFIRMED or CANCELLED

That means Orders has to wait for Inventory before it can finish the request.

After (Part 5)

Now the flow will look like this:

• Orders creates the order as PENDING

• Orders publishes an event to Kafka: OrderCreated

• Inventory consumes that event and makes a stock decision

• Inventory publishes a result event: InventoryReserved or InventoryRejected

• Orders consumes the result event and updates the order status

So instead of waiting directly for Inventory, Orders hands off the work through Kafka.

What Kafka is

Kafka is a system for sending and storing messages, often called events.

Instead of one service calling another service directly and waiting for a response, a service can publish an event to Kafka. Other services can then read that event and react to it.

You can think of Kafka as a shared event pipeline between services.

Some simple terms to understand

Topic

A topic is a named stream of events.

Example:

orders.v1

If the Orders service publishes an OrderCreated event, it might send it to the orders.v1 topic.

You can think of a topic like a category or channel where related messages are stored.

Producer

A producer is a service that publishes messages to a topic.

For example:

• Orders can be a producer when it publishes OrderCreated

Consumer

A consumer is a service that reads messages from a topic.

For example:

• Inventory can be a consumer when it reads OrderCreated

Consumer group

A consumer group allows multiple consumers to share the work of reading messages.

This is useful when you want to scale processing across multiple instances of the same service.

You can think of a consumer group as a team of workers splitting the same queue.

Partition

A topic is split into partitions.

Partitions are what allow Kafka to scale. Instead of one big single stream, a topic can be divided into multiple smaller streams that can be processed in parallel.

So:

• one topic

• multiple partitions

• more throughput and more scaling

Offset

An offset is the position of a message inside a partition.

You can think of it like a message number.

So if a consumer reads up to offset 42, it means it has processed messages up to that point in that partition.

Why Kafka is useful

One of Kafka’s biggest strengths is that consumers can crash, restart, and continue from where they left off. That means the system is more resilient. A service does not have to stay alive all the time to avoid losing work.

This is one of the reasons Kafka is so useful in distributed systems. In our case, Kafka lets us stop making Orders wait directly on Inventory.

Instead:

• Orders publishes an event

• Inventory reads it and processes it

• Inventory publishes the result

• Orders reads that result later

That makes the system more asynchronous and less tightly coupled.

Why Redpanda?

For this project, we will use Redpanda locally.

Redpanda is Kafka-compatible, which means our code can talk to it the same way it would talk to Kafka. The big advantage is that Redpanda is much easier to run in local development. We can start it as a single container, without needing a more complex setup.

It also comes with a command-line tool called rpk, which is very useful for:

• creating topics

• checking cluster state

• debugging message flow

That makes it a great choice for learning and local experimentation.

So for this project, Redpanda gives us the Kafka model and APIs we want, but with a much simpler developer experience.

Step 1: Add Redpanda to Docker Compose

In your root docker-compose.yml, add this service:

  redpanda:
    image: redpandadata/redpanda:latest
    healthcheck:
      test: ["CMD-SHELL", "curl -fsS http://127.0.0.1:9644/v1/status/ready >/dev/null"]
      interval: 5s
      timeout: 5s
      retries: 12
      start_period: 30s
    command:
      - redpanda
      - start
      - --overprovisioned
      - --smp
      - "1"
      - --memory
      - 1G
      - --reserve-memory
      - 0M
      - --node-id
      - "0"
      - --check=false
      - --kafka-addr
      - internal://0.0.0.0:9092,external://0.0.0.0:19092
      - --advertise-kafka-addr
      - internal://redpanda:9092,external://localhost:19092
    ports:
      - "19092:19092"
      - "9644:9644"

This adds Redpanda as a local Kafka-compatible broker for our project.

What this service does

This container runs Redpanda, which will act as our event system.

It will store topics such as:

• orders.v1

• inventory.v1

and later our services will publish and consume events through it.

You might ask Why there are so many command options?

Don't worry, most of these options are there to make Redpanda easier to run on a local machine.

For example:

• --overprovisioned

• --smp "1"

• --memory 1G

• --reserve-memory 0M

• --check=false

These settings reduce the resource requirements and make Redpanda friendlier for local development.

You do not need to memorize all of them. The main thing to understand is that they help us run a lightweight single-node Redpanda setup on our laptop.

Why we configure both internal and external addresses

This is the most important part of the setup. Redpanda needs to be reachable from two different places:

1. From inside Docker Compose

Other containers, like Orders and Inventory, need to connect to Redpanda using the Docker service name:

redpanda:9092

That is the internal address.

2. From your laptop

If you want to connect from your machine using tools like rpk, you cannot use the Docker service name redpanda.

From your laptop, you need to connect using:

localhost:19092

That is the external address.

So here we are saying:

• containers should use redpanda:9092

• your laptop should use localhost:19092

It was all configuration part, so you don't need to memorize it, just understand what it is doing and why. That's it.

Step 2: Create topics

First, start the containers:

docker compose up -d --build

Once everything is running, create the Kafka topics:

docker compose exec redpanda rpk topic create orders.v1 -p 3
docker compose exec redpanda rpk topic create inventory.v1 -p 3

These commands create two topics:

• orders.v1

  • This topic will carry events produced by the Orders service.

  • For example:

    • OrderCreated

    Inventory will later consume events from this topic.

• inventory.v1

  • This topic will carry events produced by the Inventory service.

  • For example:

    • InventoryReserved

    • InventoryRejected

    Orders will later consume these result events.

To check if the topics are created successfully run below command:

What are partitions?

A Kafka topic is split into partitions. You can think of a partition as one separate ordered lane of messages inside a topic.

So when we create a topic with:

rpk topic create orders.v1 -p 3

we are saying:

create the orders.v1 topic with 3 partitions

That gives us a few benefits.

Parallel processing: Multiple consumers can process different partitions at the same time, which helps the system scale.

Better throughput: Instead of all messages going through one single lane, Kafka can spread work across multiple partitions.

Ordering by key: If we publish messages with the same key, such as orderId, Kafka will consistently send those messages to the same partition.

That is useful because Kafka guarantees ordering within a partition. So if all events for order 123 use the same key, Kafka keeps those events in order for that order.

Step 3: Add Kafka library in Go services

To talk to Redpanda from Go, we need a Kafka client library.

For this project, we will use kafka-go, which is a popular Go library for working with Kafka-compatible systems.

We need to add it to both services, because:

• Orders will publish events

• Inventory will consume events and publish results

Add it to Orders:

cd services/orders
go get github.com/segmentio/kafka-go
go mod tidy
cd ../..

Add it to Inventory:

cd services/inventory
go get github.com/segmentio/kafka-go
go mod tidy
cd ../..

This adds the kafka-go library as a dependency in the service’s go.mod file. That means the service can now use Kafka producers and consumers in Go code.

You might ask why to add it for both the service. At first, it might seem like only one service needs Kafka, but both do.

Orders will use Kafka to:

• publish OrderCreated events

• later consume inventory result events

Inventory will use Kafka to:

• consume OrderCreated

• publish InventoryReserved or InventoryRejected

So both services need Kafka support.

Step 4: Define event formats (simple JSON for now)

In a production system, you could also use protobuf for Kafka events. But while learning, JSON is easier to work with because you can print the payload directly and understand what is being sent.

So for now, we will keep the event format simple and use JSON.

We will use two main kinds of events:

• OrderCreated

• InventoryReserved / InventoryRejected

OrderCreated

This event will be published by the Orders service when a new order is created.

It contains:

• orderId - which order this event belongs to

• userId - which user placed the order

• sku - which product is being ordered, such as burger

• quantity - how many items are requested

• eventId - a unique ID for this event

• time - when the event was created

This event tells the rest of the system:

“A new order was created, and inventory should now decide whether stock can be reserved.”

InventoryReserved / InventoryRejected

These events will be published by the Inventory service after it processes an order.

They contain:

• orderId → which order this result belongs to

• reserved → true if stock was reserved, false if it was rejected

• reason → an optional explanation, usually used when reservation fails

• eventId → a unique ID for this event

• time → when the result event was created

So these events tell Orders:

• whether stock reservation succeeded

• and, if it failed, why

The eventId is important because later we will use it for idempotency and de-duplication. In event-driven systems, the same message can sometimes be delivered more than once. A unique event ID helps the service recognize whether it has already processed that event before.

Step 5: Orders publishes OrderCreated instead of calling gRPC

This is the first major design change in this part. Until now, Orders was calling Inventory directly using gRPC. Now we are changing that flow. Instead of waiting for Inventory immediately, Orders will publish an event to Kafka.

That means Orders will no longer ask:

“Can you reserve stock right now?”

Instead, it will say:

“An order was created. Someone should process it.”

That “someone” will be the Inventory service.

5.1 Create a Kafka “bus” in Orders

To keep Kafka setup in one place, we will create a small helper package.

Create: services/orders/internal/kafkabus/bus.go

package kafkabus

import (
	"strings"

	"github.com/segmentio/kafka-go"
)

// This struct holds the Kafka clients that Orders needs.
type Bus struct {
    // This is the Kafka producer for Orders.
	OrdersWriter    *kafka.Writer
    // This is the Kafka consumer for Orders.
	InventoryReader *kafka.Reader
}

// This creates the Kafka writer and reader.
func New(brokers []string) *Bus {
	return &Bus{
        /*
            1. connect to the given Kafka brokers
            2. publish messages to orders.v1
            3. use kafka.Hash{} to choose partitions based on message key
        */
		OrdersWriter: &kafka.Writer{
			Addr:                   kafka.TCP(brokers...),
			Topic:                  "orders.v1",
			Balancer:               &kafka.Hash{},
			AllowAutoTopicCreation: true,
		},
        
        // This creates a Kafka consumer for inventory.v1.
		InventoryReader: kafka.NewReader(kafka.ReaderConfig{
			Brokers:  brokers,
			Topic:    "inventory.v1",
            // This is the most important part:
            // This means the Orders service reads inventory results as part of the consumer group named orders-service.
			GroupID:  "orders-service",
			MinBytes: 1,
			MaxBytes: 10e6,
		}),
	}
}

// This closes both the reader and the writer when the Orders service shuts down.
func (b *Bus) Close() error {
	_ = b.InventoryReader.Close()
	return b.OrdersWriter.Close()
}

// This helper takes a comma-separated string of broker addresses and turns it into a Go slice.
func BrokersFromEnv(s string) []string {
	parts := strings.Split(s, ",")
	out := make([]string, 0, len(parts))
	for _, p := range parts {
		p = strings.TrimSpace(p)
		if p != "" {
			out = append(out, p)
		}
	}
	return out
}

What this file is doing

This file creates a small Kafka wrapper for the Orders service.

It gives Orders two things:

• a writer for publishing order events

• a reader for consuming inventory result events

So instead of putting Kafka setup directly inside main.go or inside HTTP handlers, we keep it in one dedicated place.

That makes the code cleaner and easier to extend later.

5.2 Define events and publish function

Now we need to define the actual event structures that Orders will use when talking through Kafka.

Create: services/orders/internal/kafkabus/events.go

package kafkabus

import (
	"context"
	"encoding/json"
	"strconv"
	"time"

	"github.com/segmentio/kafka-go"
)

// This is the event that the Orders service will publish when a new order is created.
type OrderCreated struct {
	EventID  string    `json:"eventId"`
	Type     string    `json:"type"`
	Time     time.Time `json:"time"`
	OrderID  int64     `json:"orderId"`
	UserID   string    `json:"userId"`
	Sku      string    `json:"sku"`
	Quantity int32     `json:"quantity"`
}

// This is the event that Inventory will publish after processing the order.
type InventoryResult struct {
	EventID   string    `json:"eventId"`
	Type      string    `json:"type"`
	Time      time.Time `json:"time"`
	OrderID   int64     `json:"orderId"`
	Reserved  bool      `json:"reserved"`
    // The omitempty tag on Reason means that if the reason is empty, it will be left out of the JSON.
	Reason    string    `json:"reason,omitempty"`
}

// This helper takes an OrderCreated struct and publishes it to Kafka.
func PublishOrderCreated(ctx context.Context, w *kafka.Writer, evt OrderCreated) error {
    // Since we are using JSON for events right now, this turns the Go struct into a JSON byte payload.
	val, err := json.Marshal(evt)
	if err != nil {
		return err
	}

    // This writes the event into Kafka.
	return w.WriteMessages(ctx, kafka.Message{
		Key:   []byte(strconv.FormatInt(evt.OrderID, 10)),
		Value: val,
	})
}

This file gives us two things:

• the Go structs that represent our Kafka events

• a helper function PublishOrderCreated to publish an OrderCreated event

So instead of building Kafka messages manually all over the codebase, we define the event format once and reuse it.

5.3 Update Orders HTTP handler: create order + publish event

In this part of the blog series, we are keeping the flow simple.

For now, when a new order is created, we will:

• create the order in Postgres as PENDING

• publish an OrderCreated event to Kafka

• return 202 Accepted

Later, when we introduce the Outbox pattern, we will make this flow safer even if Kafka is temporarily down. But for now, this version is enough to teach the event-driven flow clearly.

Open:services/orders/internal/http/http.go

Since the HTTP handler now needs to publish Kafka events, the Orders HTTP server needs access to the Kafka bus.

Update the Server struct like this:

type Server struct {
	store *store.OrdersStore
	inv   *inv.Client
	bus   *kafkabus.Bus
}

func NewServer(store *store.OrdersStore, inv *inv.Client, bus *kafkabus.Bus) *Server {
	return &Server{store: store, inv: inv, bus: bus}
}

Earlier, the HTTP server only needed access to the database store. Now it also needs access to Kafka, because after creating an order it must publish an OrderCreated event.

So the server now depends on two things:

• the store, for database operations

• the bus, for Kafka operations

Now inside the handleCreateOrder function in http.go After creating the order as PENDING, build the event like this:

// Create order as PENDING
// ...

evt := kafkabus.OrderCreated{
	EventID:  uuid.New().String(),
	Type:     "OrderCreated",
	Time:     time.Now().UTC(),
	OrderID:  id,
	UserID:   req.UserID,
	Sku:      req.Sku,
	Quantity: int32(req.Quantity),
}

This creates the event payload that will be sent to Kafka.

It tells the rest of the system:

“A new order was created, and Inventory should now process it.”

Now publish it using the Kafka bus:

if err := kafkabus.PublishOrderCreated(ctx, s.bus.OrdersWriter, evt); err != nil {
	http.Error(w, "kafka publish failed", http.StatusServiceUnavailable)
	return
}

writeJSON(w, http.StatusAccepted, createOrderResponse{ID: id})

If publishing fails, we return:

• 503 Service Unavailable

That means the order was created in the database, but the event could not be published yet. Again, this is a simplification for now. Later, the Outbox pattern will solve this more safely.

Also, We use 202 Accepted instead of 201 Created because the full order processing is no longer finished immediately.

One important detail: publishing an event does not automatically update the order status. The status changes only if Inventory consumes the event and sends a result back, and if Orders is also consuming those result events.

That means we must start two background consumer loops in the next steps.

IMPORTANT: Make sure you comment out the previous code that we wrote in the Part 4 because no we are no longer updating status directly:

Step 6: Inventory consumes OrderCreated and publishes result

Now Inventory stops being just a gRPC server and starts acting as a Kafka consumer too.

That means Inventory will now:

• read OrderCreated events from Kafka

• try to reserve stock

• publish a result event back to Kafka

So Inventory becomes the service that reacts to order events and decides whether stock can actually be reserved.

Let's now also create a Kafka bus helper for Inventory service.

6.1 Create a Kafka bus helper for Inventory

To keep the Kafka setup clean, we will create a small bus helper for the Inventory service.

Create: services/inventory/internal/kafkabus/bus.go

package kafkabus

import (
	"strings"

	"github.com/segmentio/kafka-go"
)

type Bus struct {
    // This reads from: order.v1
	OrdersReader     *kafka.Reader
    // This writes to: inventory.v1
	InventoryWriter  *kafka.Writer
}

func New(brokers []string) *Bus {
	return &Bus{
		OrdersReader: kafka.NewReader(kafka.ReaderConfig{
			Brokers:  brokers,
			Topic:    "orders.v1",
			GroupID:  "inventory-service",
			MinBytes: 1,
			MaxBytes: 10e6,
		}),
		InventoryWriter: &kafka.Writer{
			Addr:                   kafka.TCP(brokers...),
			Topic:                  "inventory.v1",
			Balancer:               &kafka.Hash{},
			AllowAutoTopicCreation: true,
		},
	}
}

func (b *Bus) Close() error {
	_ = b.OrdersReader.Close()
	return b.InventoryWriter.Close()
}

func BrokersFromEnv(s string) []string {
	parts := strings.Split(s, ",")
	out := make([]string, 0, len(parts))
	for _, p := range parts {
		p = strings.TrimSpace(p)
		if p != "" {
			out = append(out, p)
		}
	}
	return out
}

What this bus means

This Kafka bus gives Inventory two things:

• a reader for consuming order events

• a writer for publishing inventory result events

The OrdersReader reads from: orders.v1 . That is where the Orders service publishes OrderCreated events. The InventoryWriter writes to: inventory.v1. That is where Inventory will publish the result after checking stock.

6.2 Start the Inventory Kafka consumer loop

Inventory will run a background goroutine that continuously reads from orders.v1.

For each OrderCreated event, it will:

1. decode the event

2. try to reserve stock

3. publish an inventory result event to inventory.v1

Create: services/inventory/internal/kafkabus/consume.go

package kafkabus

import (
	"context"
	"encoding/json"
	"log"

	"github.com/segmentio/kafka-go"
)

func ConsumeOrders(ctx context.Context, r *kafka.Reader, handle func(OrderCreated) error) {
	for {
		msg, err := r.FetchMessage(ctx)
		if err != nil {
			log.Printf("inventory consumer stopped: %v", err)
			return
		}

		var evt OrderCreated
		if err := json.Unmarshal(msg.Value, &evt); err != nil {
			log.Printf("bad order event json: %v", err)
			_ = r.CommitMessages(ctx, msg) // poison => commit and move on
			continue
		}

		if err := handle(evt); err != nil {
			log.Printf("handle order event failed: %v", err)
			// no commit => retry later
			continue
		}

		_ = r.CommitMessages(ctx, msg)
	}
}

What this is doing? This is the core Kafka consumer loop for Inventory. It keeps reading messages from orders.v1. If the JSON is bad, It logs the error, commits the message, and moves on. And if the handling succeed, It commits the message, which means Inventory is done with it.

6.3 Define the event structs and publish helper

Now we define the event structures Inventory will use.

Create services/inventory/internal/kafkabus/events.go

package kafkabus

import (
	"context"
	"encoding/json"
	"strconv"
	"time"

	"github.com/segmentio/kafka-go"
)

type OrderCreated struct {
	EventID   string    `json:"eventId"`
	Type      string    `json:"type"`
	Time      time.Time `json:"time"`
	OrderID   int64     `json:"orderId"`
	UserID    string    `json:"userId"`
	Sku       string    `json:"sku"`
	Quantity  int32     `json:"quantity"`
}

type InventoryResult struct {
	EventID  string    `json:"eventId"`
	Type     string    `json:"type"` // InventoryReserved | InventoryRejected
	Time     time.Time `json:"time"`
	OrderID  int64     `json:"orderId"`
	Reserved bool      `json:"reserved"`
	Reason   string    `json:"reason,omitempty"`
}

func PublishInventoryResult(ctx context.Context, w *kafka.Writer, evt InventoryResult) error {
	val, err := json.Marshal(evt)
	if err != nil {
		return err
	}
	return w.WriteMessages(ctx, kafka.Message{
		Key:   []byte(strconv.FormatInt(evt.OrderID, 10)),
		Value: val,
	})
}

These structs define the JSON payloads Inventory reads and writes.

OrderCreated

This is the event Inventory receives from Orders.

InventoryResult

This is the event Inventory publishes after checking stock.

Using orderId as the Kafka key is important because it helps keep all events for the same order in the same partition, which preserves ordering for that order.

6.4 Update Inventory main.go

Now wire Kafka into the Inventory service.

First, add a helper method outside main():

// outside of main
func (s *inventoryServer) tryReserve(sku string, qty int32) error {
	if sku == "" {
		return errors.New("sku is required")
	}
	if qty <= 0 {
		return errors.New("quantity must be > 0")
	}

	s.mu.Lock()
	defer s.mu.Unlock()

	available := s.stock[sku]
	if available < qty {
		return errors.New("insufficient stock")
	}

	s.stock[sku] = available - qty
	return nil
}

This reuses the same stock-reservation logic, but now for Kafka events instead of only gRPC requests.

Then inside main():

brokers := kafkabus.BrokersFromEnv(getenv("KAFKA_BROKERS", "localhost:19092"))
bus := kafkabus.New(brokers)
defer bus.Close()

srv := newInventoryServer()
inventoryv1.RegisterInventoryServiceServer(grpcServer, srv)

ctx := context.Background()

go kafkabus.ConsumeOrders(ctx, bus.OrdersReader, func(evt kafkabus.OrderCreated) error {
	err := srv.tryReserve(evt.Sku, evt.Quantity)

	out := kafkabus.InventoryResult{
		EventID: uuid.New().String(),
		Time:    time.Now().UTC(),
		OrderID: evt.OrderID,
	}

	if err != nil {
		out.Type = "InventoryRejected"
		out.Reason = err.Error()
		out.Reserved = false
	} else {
		out.Type = "InventoryReserved"
		out.Reserved = true
	}

	return kafkabus.PublishInventoryResult(ctx, bus.InventoryWriter, out)
})

This is where Inventory becomes part of the async workflow. First, it connects to Kafka It reads broker addresses from KAFKA_BROKERS, creates the Kafka bus, and keeps it open for the lifetime of the service.

Then, it starts a background goroutine. That goroutine continuously consumes OrderCreated events from orders.v1.

For each event, Inventory:

• checks stock using tryReserve

• creates an InventoryResult

• publishes that result to inventory.v1

If stock is available:

• type = InventoryReserved

• reserved = true

If stock is not available:

• type = InventoryRejected

• reserved = false

• reason = error message

Step 7: Orders consumes Inventory results and updates status

Now we complete the async loop.

Inventory is already publishing result events into inventory.v1, but that alone does not change anything in the Orders database. Orders must also read those result events and update the stored order status.

So Orders will now start a background goroutine that continuously reads messages from inventory.v1

These messages are the result events published by Inventory after it processes an order.

For each InventoryResult, Orders will update the order status in the database:

• if reserved is true → set status to CONFIRMED

• if reserved is false → set status to CANCELLED

At this point, the flow is split into two stages.

Stage 1

Orders accepts the request, stores the order as PENDING, and publishes an event.

Stage 2

Later, after Inventory processes the event, Orders receives the result and updates the final order status.

So the final state change now happens in the background, not during the original HTTP request.

This is the async update pattern in action.

Create services/orders/internal/kafkabus/consume.go to consume inventory.v1 continuously.

package kafkabus

import (
	"context"
	"encoding/json"
	"log"

	"github.com/segmentio/kafka-go"
)

// ConsumeInventoryResults reads messages from inventory.v1 and passes them to handle().
// If handle returns error, we do not commit, so Kafka will retry.
func ConsumeInventoryResults(ctx context.Context, r *kafka.Reader, handle func(InventoryResult) error) {
	for {
		msg, err := r.FetchMessage(ctx)
		if err != nil {
			log.Printf("orders: inventory consumer stopped: %v", err)
			return
		}

		var evt InventoryResult
		if err := json.Unmarshal(msg.Value, &evt); err != nil {
			log.Printf("orders: bad inventory event json: %v", err)
			_ = r.CommitMessages(ctx, msg) // poison => commit and move on
			continue
		}

		if err := handle(evt); err != nil {
			log.Printf("orders: handle inventory event failed: %v", err)
			continue // no commit => retry
		}

		_ = r.CommitMessages(ctx, msg)
	}
}

This loop keeps reading messages from inventory.v1. If the JSON is invalid It logs the error, commits the message, and moves on. That is because a poison message will not become valid just by retrying. If handling fails It does not commit the message. That means Kafka can deliver it again later.

If handling succeeds, It commits the message, which means Orders has processed it successfully. This gives us an at-least-once processing model.

Start the background consumer in main.go

Now inside services/orders/cmd/orders/main.go 's main() function, after creating the HTTP server, start the consumer goroutine:

ctx := context.Background()

go kafkabus.ConsumeInventoryResults(ctx, bus.InventoryReader, func(evt kafkabus.InventoryResult) error {
  switch evt.Type {
  case "InventoryReserved":
    return orderStore.UpdateStatus(ctx, evt.OrderID, store.StatusConfirmed)
  case "InventoryRejected":
    return orderStore.UpdateStatus(ctx, evt.OrderID, store.StatusCancelled)
  default:
    return nil
  }
})

This starts a background worker inside Orders. Whenever Inventory publishes a result event, Orders reads it and applies the final state change. If the result is InventoryReserved Orders updates the order to CONFIRMED. If the result is InventoryRejected Orders updates the order to CANCELLED.

So this is the final step that closes the async workflow.

Step 8: Update docker-compose

Now both services need to know how to reach Kafka.

Add KAFKA_BROKERS to the orders and inventory services in docker-compose.yml:

  orders:
    depends_on:
        ...
     redpanda:
       condition: service_healthy
    environment:
      ...
      KAFKA_BROKERS: "redpanda:9092"

  inventory:
    environment:
      ...
      KAFKA_BROKERS: "redpanda:9092"

Both services now use Kafka.

How to test

Now let’s test the new Kafka-based flow end to end.

Start fresh with:

docker compose down -v
docker compose up --build

This removes old containers and volumes, then rebuilds and starts the full system again.

Create an order:

Send a request to the Orders API:

curl -s -X POST localhost:8080/v1/orders \
  -H "Content-Type: application/json" \
  -d '{"userId":"u1","sku":"burger","quantity":2,"note":"kafka test"}'

If the request is accepted, you should get back an order ID.

Because the flow is now asynchronous, this does not mean the order is fully processed yet. It only means Orders accepted the request, stored the order, and published an event.

Now fetch the order:

curl -s localhost:8080/v1/orders/1

At first, you should see the order in: PENDING. Then, after Inventory consumes the event and publishes the result, the order should become: CONFIRMED or CANCELLED.

To see the flow happening across both services, run:

docker compose logs -f orders inventory

What we learned in Part 5

In this part, we introduced Kafka and changed the order flow from synchronous processing to an event-driven design.

We learned:

• how Kafka topics and consumer groups work

• how to publish events with a key

• how to consume messages and commit offsets

• why asynchronous processing leads to a PENDING state before the final result

• how services become more independent and resilient when they communicate through events

This part is important because it changes how the system behaves. Orders no longer has to wait directly for Inventory. Instead, the two services communicate through Kafka, which makes the system more decoupled and better suited for scaling and failure handling.

It also introduces an important idea that shows up in many real systems: work is often accepted first, then completed later. That is why a PENDING state becomes a normal part of the design.

Conclusion

We now have our first event-driven order flow.

Orders no longer waits directly for Inventory. Instead, it accepts the request quickly and lets inventory processing happen asynchronously through Kafka. That makes the system feel much closer to how real distributed backends are designed.

At the same time, this version is not fully safe yet.

There is still an important failure case: if Kafka is down at the moment we create an order, the event can be lost. That can leave an order stuck in PENDING with no way to finish the workflow.

So in the next part, we will focus on reliability. We will move toward a Saga-style workflow and then introduce the Outbox pattern, which will make sure important events are not lost.

In this part, we connect them for the first time.

The goal is simple: when a new order is created, the Orders service will call the Inventory service to reserve stock. If stock is available, the order can move forward. If not, the order will be cancelled.

This is an exciting step because QuickBite now starts behaving like a real multi-service backend, where one service depends on another to complete a request.

What we are building in this part

We will keep the same public API for the app:

• POST /v1/orders

• GET /v1/orders/{id}

So from the client’s point of view, nothing changes.

But now POST /v1/orders will do more work behind the scenes:

1. create an order in Postgres with status PENDING

2. call the Inventory service over gRPC: ReserveStock(sku, quantity)

3. if Inventory reserves the stock, update the order to CONFIRMED

4. if Inventory rejects the request, update the order to CANCELLED

This means the order status is no longer just stored data. It now reflects real business logic and the result of communication between two services.

Why we connect with gRPC first (before Kafka)

Later in the project, we will move this flow to Kafka and make it asynchronous. But before jumping there, it helps to first build a simple synchronous version that works.

That way, we can understand the basic interaction clearly:

• Orders calls Inventory

• Inventory responds

• Orders updates its state based on the result

gRPC works well for this because it gives us a strict shared contract, typed request and response objects, and built-in support for deadlines and timeouts.

This part is really about learning how two services communicate directly, and how to handle success and failure in that setup, before moving to a more advanced event-driven design.

Step 1: Add "UpdateStatus" to the Orders store

The Orders service already knows how to create an order. Now it also needs a way to update the order status after the Inventory service responds.

Open:

services/orders/internal/store/orders.go

Add this method:

func (s *OrdersStore) UpdateStatus(ctx context.Context, id int64, status string) error {
	_, err := s.db.Exec(ctx, `UPDATE orders SET status = \(2 WHERE id = \)1`, id, status)
	return err
}

What this method does

This method updates the status column of an existing order.

It takes:

• ctx - the request context

• id - the order ID

• status - the new status value, such as CONFIRMED or CANCELLED

Internally, it runs this SQL:

UPDATE orders SET status = \(2 WHERE id = \)1

So if the order ID is 1 and the new status is CONFIRMED, it updates that row in the database.

Step 2: Create an Inventory gRPC client inside Orders

Now the Orders service needs a way to talk to the Inventory service. To keep the rest of the code clean, we will create a small client wrapper.

First, create the folder:

mkdir -p services/orders/internal/inventory

Now create the file:

services/orders/internal/inventory/client.go

package inventory

import (
	"context"
	"time"

	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials/insecure"

	inventoryv1 "example.com/quickbite/proto/gen/go/inventory/v1"
)

// this struct keeps two things: conn, api
type Client struct {
	conn *grpc.ClientConn // the underlying gRPC connection
	api  inventoryv1.InventoryServiceClient // the generated gRPC client from the proto package
}

// this function creates a new Inventory client.
func New(addr string) (*Client, error) {
	ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
	defer cancel()

    // it connects to the Inventory service using DialContext
    // we wrap it in timeout so that if Inventory is down, Order will not hand forever whole trying to connect.
	conn, err := grpc.DialContext(
		ctx,
		addr,
    
    // for local development, we use insecure transport cuz the services are communicating inside Docker Compose on an internal network. Layer we could switch to TLS for production.
	grpc.WithTransportCredentials(insecure.NewCredentials()),
		grpc.WithBlock(), // by default gRPC may return immediately and connect in the background so we use WithBlock so the call waits until the connection is actually ready.
	)
	if err != nil {
		return nil, err
	}

	return &Client{
		conn: conn,
		api:  inventoryv1.NewInventoryServiceClient(conn),
	}, nil
}

// this closes the gRPC connection when the orders service shuts down.
func (c *Client) Close() error {
	return c.conn.Close()
}

// This method will be used by Order service.
func (c *Client) Reserve(ctx context.Context, sku string, qty int32) error {
    //creates the short request timeout
	ctx, cancel := context.WithTimeout(ctx, 2*time.Second)
	defer cancel()
    
    // Then it calls the inventory gRPC method
	_, err := c.api.ReserveStock(ctx, &inventoryv1.ReserveStockRequest{
		Sku:      sku,
		Quantity: qty,
	})
	return err
}

This file creates a small wrapper around the generated gRPC client.

Instead of letting the HTTP handler deal with low-level gRPC setup, we move that logic into a dedicated package. That way, the rest of the Orders service can use a simple method like:

client.Reserve(ctx, "burger", 2)

What to understand here

1) We use DialContext with a timeout

If the Inventory service is down, we do not want the Orders service to wait forever. The timeout makes failure happen quickly and clearly.

2) We use insecure transport for local development

Since the services are talking to each other inside Docker Compose, we keep it simple for now. Later, we can add TLS when we move to more production-like environments.

3) We treat success as “no error”

Notice that Reserve(...) does not return true or false.

Instead:

• if stock is reserved successfully - return nil

• if Inventory rejects the request - return an error

This keeps the Orders-side logic simple. The handler only needs to check:

• no error - proceed

• error - handle failure

Step 3: Update Orders main.go to create the Inventory client

Now that we have a small gRPC client wrapper for Inventory, the Orders service needs to create that client when it starts.

Open:

services/orders/cmd/orders/main.go

Add this import:

inventory "example.com/quickbite/orders/internal/inventory"

Then create the client using an address from the environment:

invAddr := getenv("INVENTORY_GRPC_ADDR", "localhost:9090")

// ...db logic

// This creates gRPC client connection to the Inventory service.
invClient, err := inv.New(invAddr)
if err != nil {
	log.Fatalf("inventory client connect failed: %v", err)
}
defer invClient.Close()

This code creates the Inventory gRPC client when the Orders service starts. getenv("INVENTORY_GRPC_ADDR", "localhost:9090") reads the Inventory service address from an environment variable. If the variable is not set, it falls back to localhost:9090.

That makes local development easier, because the Orders service knows where to find Inventory by default. Later, in Docker Compose, this value will usually be something like:

inventory:9090

because services talk to each other using service names inside the Docker network.

Why do this in main()?

We create the Inventory client in main() because startup is the place where we usually create long-lived dependencies such as db connections, external clients, configuration, HTTP servers.

Then we pass those dependencies into the parts of the app that need them.

That keeps the code organized and makes the application easier to test and maintain.

Now the Orders service has an Inventory client, but the HTTP handlers still do not know about it yet. So the next step is to update the HTTP server constructor and pass this client into it. That way, when someone creates an order, the handler can call Inventory through gRPC.

Step 4: Update Orders HTTP server to use Inventory

Now we connect the Orders HTTP layer to the Inventory gRPC client.

Open:

services/orders/internal/http/http.go

4.1 Update the Server struct and constructor

Add imports:

inv "example.com/quickbite/orders/internal/inventory"

Then update the Server struct so it holds both:

• the Orders store

• the Inventory client

type Server struct {
	store *store.OrdersStore
	inv   *inv.Client
}

func NewServer(store *store.OrdersStore, invClient *inv.Client) *Server {
	return &Server{store: store, inv: invClient}
}

This means the HTTP server can now talk to both:

• Postgres, through the store

• Inventory, through the gRPC client

4.2 Update handleCreateOrder to call Inventory

Now we update the create-order handler so it does real business work.

The new flow is:

1. create the order as PENDING

2. call Inventory to reserve stock

3. update the order status based on the result

Replace your create handler with this version:

func (s *Server) handleCreateOrder(w http.ResponseWriter, r *http.Request) {
	var req createOrderRequest
    // first we decode the JSON and check the fields.
	if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
		http.Error(w, "invalid json", http.StatusBadRequest)
		return
	}

	req.UserID = strings.TrimSpace(req.UserID)
	req.Sku = strings.TrimSpace(req.Sku)

	if req.UserID == "" {
		http.Error(w, "userId is required", http.StatusBadRequest)
		return
	}
	if req.Sku == "" {
		http.Error(w, "sku is required", http.StatusBadRequest)
		return
	}
	if req.Quantity <= 0 {
		http.Error(w, "quantity must be > 0", http.StatusBadRequest)
		return
	}

	ctx, cancel := context.WithTimeout(r.Context(), 5*time.Second)
	defer cancel()

	// 1) Create order as PENDING
    // At this point, the order is stored in Postgres with a PENDING status.
	id, err := s.store.Create(ctx, req.UserID, req.Note, req.Sku, req.Quantity)
	if err != nil {
		http.Error(w, "db error", http.StatusInternalServerError)
		return
	}

	// 2) Ask Inventory to reserve stock
	err = s.inv.Reserve(ctx, req.Sku, int32(req.Quantity))
	if err != nil {
		// If reserve fails, mark order cancelled
		_ = s.store.UpdateStatus(ctx, id, store.StatusCancelled)

		// We handle "insufficient stock" differently from "inventory down"
		st, ok := status.FromError(err)
		if ok && st.Code() == codes.FailedPrecondition {
			http.Error(w, "insufficient stock", http.StatusConflict) // 409
			return
		}

		http.Error(w, "inventory unavailable", http.StatusServiceUnavailable) // 503
		return
	}

	// 3) Success -> CONFIRMED
	_ = s.store.UpdateStatus(ctx, id, store.StatusConfirmed)

	writeJSON(w, http.StatusCreated, createOrderResponse{ID: id})
}

You will need these extra imports in http.go:

"google.golang.org/grpc/codes"
"google.golang.org/grpc/status"

Why we create the order first

You might wonder why we insert the order before reserving stock. The reason is that we want to create a real record as early as possible. That gives us:

• a real order ID

• a traceable record in the database

• a history of what happened, even if Inventory fails

So if stock reservation fails, we still know, the order was attempted and it was later cancelled.

This becomes even more useful later when we move to Kafka and event-driven processing, because keeping a record early fits naturally with async workflows.

Step 5: Update Orders main.go server creation

Back in:

services/orders/cmd/orders/main.go

you previously created the HTTP server like this:

srv := httpapi.NewServer(orderStore)

Now that the Orders HTTP server also needs access to the Inventory gRPC client, update it to:

srv := httpapi.NewServer(orderStore, invClient)

Earlier, the HTTP server only needed the Orders store because it was only talking to Postgres. Now the create-order handler also needs to call the Inventory service through gRPC.

That means the HTTP server needs access to two dependencies: orderStore and invClient.

So this line is simply passing both of those dependencies into the server when it is created.

Step 6: Update Docker Compose (connect containers using service name)

Now we need to make sure the Orders service can reach the Inventory service when both are running inside Docker Compose.

In the root docker-compose.yml, make sure the Inventory service exists, like this:

inventory:
    build:
      context: ./services/inventory
    ports:
      - "9090:9090"

Then update the orders service environment to include:

orders:
    build:
        ...
    environment:
        ...
        INVENTORY_GRPC_ADDR: "inventory:9090"

Why inventory:9090 works

Inside Docker Compose, each service name becomes a hostname. So inventory resolves to the inventory container automatically. This is one of the most useful Compose features.

That means the service named inventory: can be reached by other containers using inventory. So when Orders connects to inventory:9090 it is connecting to the inventory container on port 9090,

Step 7: Run the full system

Now it is time to run the full system and see both services working together.

From the repo root, start fresh with:

docker compose down -v

ISSUES You might run into while running the container:

1. Missing Go module dependencies

• The Orders service started importing:

  • example.com/quickbite/proto/...

  • google.golang.org/grpc

  but those dependencies were not listed in services/orders/go.mod.

• That meant the Orders module knew about the import paths in code, but its module file did not yet declare them properly.

To fix this: I added the missing dependencies:

• example.com/quickbite/proto

• google.golang.org/grpc

In services/orders/go.mod file

I also added

replace example.com/quickbite/proto => ../../proto

This follows the same local-development pattern used by Inventory. It tells Go to use the local proto folder instead of looking for that module somewhere remote.

After that, I ran: go mod tidy so go.mod and go.sum were updated correctly.

1. Docker build could not see the shared Proto module

• The Orders Docker build was using:

  context: ./services/orders
  

• That meant Docker only saw files inside the services/orders directory.

• But the Orders service also depends on the shared proto module, which lives outside that folder. So during the build, Docker could not access the proto directory. That is why the build failed.

To fix that:

I updated the Orders Dockerfile to match the Inventory Dockerfile structure.

The main changes were:

• use repo-root paths

• set WORKDIR to /src/services/orders

• copy the shared Proto module into /src/proto

• build the binary with: go build -d /orders

Update the /orders/Dockerfile with this:

# Build stage: compiles the Go library.
# Start from the image that has the Go compiler. Name this stage as "builder".
FROM golang:1.26.1 AS builder
# Set the working directory in the container to /app.
WORKDIR /src/services/orders
# Copy go.mod and go.sum in that dir
COPY services/orders/go.mod services/orders/go.sum ./
COPY proto/go.mod proto/go.sum /src/proto/
# Downloads the dependencies.
RUN go mod download 
# Copy the remaining files
COPY proto /src/proto
COPY services/orders .
# Compile your go app into a single executable file.
RUN CGO_ENABLED=0 GOOS=linux go build -o /orders ./cmd/orders

# Run stage: runs image with the compiled binary.
# gcr.io/distroless/static:nonroot -> minimal Linux image without any additional packages.
FROM gcr.io/distroless/static:nonroot 
# Set the working directory in the container to /.
WORKDIR / 
# Copt the built file from stage 1 (builder) and copy it into this final image. So now the final image contains basically /orders (your executable)
COPY --from=builder /orders /orders
# run the program as non-root user and not as admin/root for better security. If someone gains access to the container, they won't have root access.
USER nonroot:nonroot 
# And the a label saying 8080, this container will listen on port 8080. You still need -p 8080:8080 to map the container port to the host port.
EXPOSE 8080 
# When the container starts, runs "/orders"
ENTRYPOINT ["/orders"]

Also in docker-compose.yml change the Order build section to use the repo root as the build context:

This is important because now the Docker build can see both: services/orders and proto. Without that, the Dockerfile would still not be able to access the shared Proto module.

Now finally run:

docker compose up --build

Make sure everything is running:

docker compose ps

You should see your main containers running, such as: orders, inventory, db, redpanda.

Step 8: Test it end-to-end

Now let’s test the full flow and make sure Orders and Inventory are actually working together.

8.1 Successful order

Create an order:

curl -s -X POST localhost:8080/v1/orders \
  -H "Content-Type: application/json" \
  -d '{"userId":"u1","sku":"burger","quantity":2,"note":"grpc test"}'

If everything works, you should get a response like:

{"id":1}

Now fetch that order:

curl -s localhost:8080/v1/orders/1

You should see:

• status: CONFIRMED

8.2 Insufficient stock (should cancel)

Now try creating an order with a quantity that is too large:

curl -i -X POST localhost:8080/v1/orders \
  -H "Content-Type: application/json" \
  -d '{"userId":"u2","sku":"burger","quantity":99,"note":"too many"}'

This time, Inventory should reject the reservation because there is not enough stock. You should get: 409 Conflict and message insufficient stock.

8.3 Inventory down (should return 503)

Now let’s test what happens when one service is unavailable.

In another terminal, stop Inventory.

docker compose stop inventory

Then try creating another order:

curl -i -X POST localhost:8080/v1/orders \
  -H "Content-Type: application/json" \
  -d '{"userId":"u3","sku":"pizza","quantity":1,"note":"inventory down"}'

This time, Orders cannot reach Inventory at all. You should see 503 Service Unavailable.

That means the Orders service handled the failure in a controlled way instead of hanging forever or crashing.

Why this test matters

This is an important moment because it introduces a core idea in distributed systems:

services can fail, and your application must handle that cleanly

In a single-service app, failures are already important. In a multi-service system, they become even more important because one service may be healthy while another is down or unreachable.

That is why this step matters so much. It teaches us how to think about:

• success

• business rejection, such as insufficient stock

• infrastructure failure, such as an unavailable service

What we learned in Part 4

In this part, we connected two services and handled real business outcomes.

We learned how to:

• create a gRPC client in Go

• call another service with timeouts

• handle gRPC errors and map them to HTTP responses

• use Docker Compose networking for service discovery

• update order state based on service-to-service results

This is the first truly distributed step in the project. Orders is no longer working as a standalone service. It now depends on Inventory to complete part of its workflow.

Conclusion

We now have a working flow where creating an order actually checks inventory and updates the order status. This is a big milestone because it introduces service-to-service communication and real failure cases.

But this approach still has one big limitation: it is synchronous. If Inventory is slow or down, Orders is directly affected. In real systems, this often becomes a bottleneck.

That is exactly why the next part matters.

In Part 5, we will introduce Kafka and move this flow to an event-driven approach. That allows Orders to accept requests quickly, and Inventory to process them asynchronously, with better reliability and scaling options.

In this part, we will add the second service: Inventory. We will also introduce gRPC and protobuf (proto files). This is the first time we connect services in a more structured way.

Even if you haven’t used gRPC before, don’t worry. I’ll explain the purpose and the moving parts in a simple way. This is going to be a really interesting one. Let's jump right into it

Why add Inventory as a separate service?

In the real world, inventory is not just a function. It has its own data and its own rules. For example:

• Inventory needs to know how many items are left.

• Inventory must stop two orders from reserving the same last item at the same time.

• Inventory might be slow or unavailable, and Orders should not crash because of it.

By separating Inventory into its own service, we learn service boundaries early. This is important because later we will add Kafka and make the workflow async. But before we go async, it helps to first understand a simple service-to-service call.

What gRPC is and why we use it?

Most apps start with REST because it is simple and easy to understand. REST usually sends JSON, which is human readable and works really well for client apps like mobile and web apps. In our project, we are still using REST for the client side.

But inside the backend, services also need to talk to each other. For example, the Orders service needs to ask the Inventory service to reserve stock. For this kind of service-to-service communication, we want something more strict and reliable than manually writing JSON requests and responses.

That is where gRPC helps.

1. With gRPC, we first define the API in a .proto file. This file describes what function a service provides, what input it expects, and what output it returns. Both services then generate code from that file.

1. Because both sides use code generated from the same .proto file, they both follow the exact same request and response structure. This reduces mistakes and keeps communication clear.

3. gRPC is fast and efficient, but the biggest reason we use it here is not speed. The biggest benefit is that both services follow one shared definition, so they always agree on how to talk to each other.

You can think of a .proto file as a shared agreement between services.

What will we build in this part?

In this part, We will add:

1. A new service: inventory

2. A shared proto module that contains the contract

3. A gRPC server in Inventory

4. (For now) Inventory will keep stock in memory so we can focus on gRPC

5. We will run Orders + Inventory with Docker Compose

In later parts, Inventory will get its own Postgres and become durable. But here we start simple.

Step 1: Create the proto module

I already explained the overall concept of protobuf file above but if you still want to know more about it, I would recommend watching this video to understand why we need .proto file and what its advantages are over JSON and XML: What is Protocol Buffer

Now, once you are done watching it, from the repo root run:

mkdir -p proto/inventory/v1
cd proto
go mod init quickbite/proto
cd ..

This creates a separate Go module for proto-generated code. It helps because both services can depend on it without copying files around.

Step 2: Define the Inventory gRPC contract

This file defines how the Inventory service and any other service will communicate with each other using gRPC.

This .proto file is acting like a shared agreement between services. It tells both sides:

• what service exists

• what function it provides

• what data should be sent

• what data should be returned

Instead of manually guessing request and response formats, both services follow the same definition.

Create:

proto/inventory/v1/inventory.proto

syntax = "proto3"; // This tells Protobuf which version of the language you are using.

// This is the namespace of this proto. It helps organize things and avoid naming conflicts. Here its referring like inventory.v1 belongs to inventory version 1.
package inventory.v1;

// This is specifically for Go code generation. When you generate Go code from this .proto file, this tells the generator where the Go package should live
option go_package = "quickbite/proto/gen/go/inventory/v1;inventoryv1";

// This defines a gRPC service. Inside it, you define one RPC method: ReserveStock meaning now the InventoryService has one function called ReserveStock.
service InventoryService {
  // This is the actual function definition. It says input type = ReserveStockRequest and output type = ReserveStockResponse.
  // In plain language: Client sends a ReserveStockRequest, and server sends back a ReserveStockResponse.
  rpc ReserveStock(ReserveStockRequest) returns (ReserveStockResponse);
}

// A message is like a data structure. It has two fields: sku and quantity
message ReserveStockRequest {
  string sku = 1; // These numbers are field numbers.
  int32 quantity = 2; // Protobuf uses these numbers internally when encoding/decoding data.
}

// This is the response structure.
message ReserveStockResponse {
  bool reserved = 1;
}

This file is basically saying:

• We have an Inventory service

• It provides a function called ReserveStock

• That function takes a request with sku and quantity

• It returns a response telling us whether the reservation worked

Step 3: Generate Go code from the proto file (using Buf)

Buf is a tool for working with .proto files. We use Buf because it keeps protobuf tooling consistent across machines. It also avoids “it works on my laptop” problems.

Instead of everyone using different local setups, Buf makes the code generation process predictable and consistent.

Create: proto/buf.yaml

version: v1

• This tells Buf, this folder is a Buf module. You can think of it as a small config file that marks this folder as the place where our protobuf definitions live.

Create: proto/buf.gen.yaml

version: v1 
plugins:
    - plugin: buf.build/protocolbuffers/go 
    - out: gen/go 
    - opt: paths=source_relative

    - plugin: buf.build/grpc/go 
    - out: gen/go 
    - opt: paths=source_relative

• This file tells Buf how to generate code.

• It uses two plugins:

  • buf.build/protocolbuffers/go
    This generates normal Go code for protobuf messages.

  • buf.build/grpc/go
    This generates gRPC-specific Go code, such as the client and server interfaces.

• The generated files will be written inside:

  proto/gen/go
  

  The option paths=source_relative means Buf will keep the generated folder structure similar to the original .proto file structure.

Now run below command to generate the code:

docker run --rm -v "$(pwd)":/workspace -w /workspace/proto bufbuild/buf:1.34.0 generate

This command runs Buf inside Docker, so you do not need to install Buf locally on your machine. That makes the setup easier and keeps the generation process the same for everyone.

Step 4: Create a Go workspace (go.work)

So go.work is a file that helps Go work with multiple local modules together during development.

You might ask why we need it?

• At this point in the project, we have separate modules like: services/orders, proto

Each one has its own go.mod file, so by default Go treats them as separate projects. But our Orders service wants to import generated code from the Proto module locally.

Without a go.work file, Go may think:

“Where do I download this module from?”

and it may expect that module to exist on GitHub or somewhere else online.

That is not what we want during local development. We want Go to understand that both modules already exist inside the same repo.

So let's create one. From the repo root run:

go work init ./services/orders ./proto

It creates a go.work file at the repo root. What it basically tells Go is:

“These local folders are Go modules. While I am developing, treat them as part of one workspace.”

Step 5: Make Orders depend on the proto module

Now we need to tell the Orders module that it depends on the Proto module.

From repo root:

cd services/orders
go mod edit -require=quickbite/proto@v0.0.0
go mod tidy
cd ../..

This adds the Proto module as a dependency of the Orders module.

In simple words, this tells Go:

“The Orders service needs code from the Proto module.”

That is important because later the Orders service will import the generated gRPC and protobuf code from the Proto module.

Why @v0.0.0?

We are still working locally, and this module is not published anywhere yet. So v0.0.0 is just used as a placeholder version.

Sometimes when you run this step, Go may show an error about the module path being invalid. This happens because names like:

quickbite/proto
quickbite/orders
quickbite/inventory

are not valid full Go module paths. Go module paths usually need a proper domain-style prefix, such as:

example.com/quickbite/proto
example.com/quickbite/orders
example.com/quickbite/inventory

To fix this issue, we need to update module paths to valid example.com/quickbite/... paths and regenerate protos.

Inside: inventory.proto, all go.mod, main.go, http.go wherever you see:

quickbite/proto, quickbite/orders, quickbite/inventory update it with example.com/quickbite/proto, example.com/quickbite/orders and example.com/quickbite/inventory

Also update any old Go imports to use the new example.com/... paths.

After changing the module paths, regenerate the protobuf Go code so the generated files use the updated import paths.

cd proto

docker run --rm -v "$(pwd)/..":/workspace -w /workspace/proto bufbuild/buf:1.34.0 generate

This regenerates the Go code from your .proto file with the new package paths.

In short, In this step, we made Orders depend on the Proto module so it can use the generated gRPC code. If Go complains about invalid module paths, we fix that by switching to proper example.com/quickbite/... module names and regenerating the protobuf code.

Step 6: Create the Inventory service module

Now we will create the Inventory service, just like we created the Orders service.

First, create the folder for the Inventory service:

mkdir -p services/inventory/cmd/inventory
cd services/inventory

This gives us a standard Go service structure. The cmd/inventory folder will later contain the main.go file that starts the Inventory service.

Now initialize the Inventory module:

go mod init example.com/quickbite/inventory

This creates a go.mod file for the Inventory service.

In simple words, this tells Go:

“This folder is its own Go module.”

Next, add the gRPC library:

go get google.golang.org/grpc

This adds the Go gRPC runtime, which we need because the Inventory service will expose a gRPC server.

Now add protobuf support:

go get google.golang.org/protobuf

This is needed because the Go code generated from the .proto file depends on the protobuf library.

Now tell the Inventory module that it depends on the shared Proto module:

go mod edit -require=example.com/quickbite/proto@v0.0.0

Why we need it? Because the generated gRPC and protobuf code lives inside the Proto module. So the Inventory service needs to declare:

“I depend on the Proto module.”

That way, Inventory can import and use the generated code later.

Again, v0.0.0 is just a placeholder version because we are working locally.

Finally run go mod tidy and add Inventory to the workspace

go work use ./services/inventory

This updates go.work so Go treats Inventory as part of the same local workspace as: Order, Proto

To conclude, In this step, we created the Inventory service as a separate Go module, added the dependencies it needs for gRPC and protobuf, connected it to the shared Proto module, and added it to the local Go workspace.

Step 7: Write the Inventory gRPC server

Now we will create the Inventory service itself. This service will:

• listen on port 9090

• expose the gRPC method ReserveStock

• keep stock in memory for now

• safely update stock when requests come in

Create: services/inventory/cmd/inventory/main.go

package main

import (
	"context"
	"log"
	"net"
	"sync"

	"google.golang.org/grpc"
	"google.golang.org/grpc/codes"
	"google.golang.org/grpc/status"

	inventoryv1 "quickbite/proto/gen/go/inventory/v1" // generated Go code in proto
)

// This is our Actual gRPC server
type inventoryServer struct {
	inventoryv1.UnimplementedInventoryServiceServer

	mu    sync.Mutex // lock for handling multiple requests at the same time
	stock map[string]int32 // inventory data. Its just an in memory Go map for now.
}

// creates the server with starting stock
func newInventoryServer() *inventoryServer {
	return &inventoryServer{
		stock: map[string]int32{
			"burger": 5,
			"pizza":  10,
		},
	}
}

// The gRPC method
// This is the server implementation of the method defined in the proto:
// So whenever a client calls ReserveStock, this Go function runs.
func (s *inventoryServer) ReserveStock(ctx context.Context, req *inventoryv1.ReserveStockRequest) (*inventoryv1.ReserveStockResponse, error) {
    
    // First this checks the request is valid.
	if req.GetSku() == "" {
		return nil, status.Error(codes.InvalidArgument, "sku is required")
	}
	if req.GetQuantity() <= 0 {
		return nil, status.Error(codes.InvalidArgument, "quantity must be > 0")
	}

    // Then it locks the mutex, checks how much stock is available, and compares it to the requested quantity.
	s.mu.Lock()
	defer s.mu.Unlock()

    // check availability
	available := s.stock[req.Sku]
	if available < req.Quantity {
		return nil, status.Error(codes.FailedPrecondition, "insufficient stock")
	}

    // deduct stock
	s.stock[req.Sku] = available - req.Quantity
	return &inventoryv1.ReserveStockResponse{Reserved: true}, nil
}

// This main function starts the server
// It opens port 9090, creates a new gRPC server, registers our inventoryServer, and starts listening for requests.
// So once this program is running, the Inventory service is ready to accept gRPC calls on port 9090.
func main() {
	lis, err := net.Listen("tcp", ":9090")
	if err != nil {
		log.Fatalf("listen failed: %v", err)
	}

	grpcServer := grpc.NewServer()
	inventoryv1.RegisterInventoryServiceServer(grpcServer, newInventoryServer())

	log.Println("inventory gRPC listening on :9090")
	log.Fatal(grpcServer.Serve(lis))
}

This file creates a simple gRPC server for the Inventory service. It uses the Go code generated from our .proto file, which is why we import:

inventoryv1 "example.com/quickbite/proto/gen/go/inventory/v1"

That generated package contains the request and response types, along with the gRPC server interface we need to implement.

Step 8: Add Inventory Dockerfile

Now we need a Dockerfile for the Inventory service so we can package it into a container image.

Create: services/inventory/Dockerfile

FROM golang:1.26.1 AS builder // This starts from an image that already has Go installed.

// sets the working directory inside the container.
WORKDIR /src/services/inventory
// copies only the Inventory module dependency files first.
COPY services/inventory/go.mod services/inventory/go.sum ./
// copies the Proto module’s go.mod and go.sum into:
COPY proto/go.mod proto/go.sum /src/proto/
// downloads Go dependencies. At this point, Docker only has the module files, not all source code yet.
RUN go mod download
// Now copy the full Proto module source into the container.
COPY proto /src/proto
// copy the full Inventory service source code into the current working directory.
COPY services/inventory .
// This compiles the app.
RUN CGO_ENABLED=0 GOOS=linux go build -o /inventory ./cmd/inventory

// Now you start a completely new image.
FROM gcr.io/distroless/static:nonroot
// Set working directory to root.
WORKDIR /
// copies the compiled binary from the builder stage into the final runtime image.
COPY --from=builder /inventory /inventory
// Run the container as a non-root user.
USER nonroot:nonroot
EXPOSE 9090
// tells Docker what to run when the container starts.
ENTRYPOINT ["/inventory"]

This Dockerfile builds the Inventory service into a container image.

It uses a two-stage build:

• builder stage - compiles the Go application

• runtime stage - runs only the compiled binary in a very small image

This keeps the final image smaller, cleaner, and more secure.

Step 9: Update the docker-compose.yml

Now we add the Inventory service to docker-compose.yml so Docker Compose can build and run it for us.

    inventory:
      build:
        context: .
        dockerfile: services/inventory/Dockerfile
      ports:
        - "9090:9090"

This tells Docker Compose:

• where to find the Dockerfile for Inventory

• how to build the Inventory image

• which port should be exposed

Step 10: Testing

Now let’s test whether the Inventory service is working correctly.

First, start the project with:

docker-compose up --build

If everything starts correctly, you should see a log message like this:

inventory gRPC listening on :9090

That tells us the Inventory service is running and listening for gRPC requests on port 9090.

And because it is gRPC, you can’t easily test it with curl.

Why can we not easily use curl with gRPC

Since this service uses gRPC, we cannot test it as easily as a normal REST API with curl.

REST usually works with: HTTP, JSON, and normal endpoints like /orders.

gRPC is different. It uses:

• protobuf messages

• a stricter contract

• a different communication format

So for testing gRPC services, we usually use a tool like grpcurl.

On Mac, install it with: brew install grpcurl

Then run the command below:

grpcurl \
  -plaintext \
  -import-path proto \
  -proto inventory/v1/inventory.proto \
  -d '{"sku":"burger","quantity":2}' \
  localhost:9090 \
  inventory.v1.InventoryService/ReserveStock

Output: You should see reserved

What we learned

In this part, we added a second service and introduced the idea of strict service contracts.

We learned:

• why .proto files are useful for service-to-service APIs

• how code generation works, from .proto files to Go code

• how to run a gRPC server in Go

• how to keep multiple services aligned by sharing a Proto module

• how a go.work workspace makes local development easier

At first, this can feel like a lot of moving parts. But once everything is connected, the benefit becomes clear. Instead of each service guessing request and response shapes, both services follow one shared contract. That makes communication safer, cleaner, and easier to maintain.

Conclusion

At this point, we now have two services running locally:

• Orders, which uses REST

• Inventory, which uses gRPC

We also have a shared contract that defines the Inventory API in a clean and strict way.

This is an important step because it introduces a real pattern used in many backend systems: public APIs are often kept simple and client-friendly with REST, while internal service-to-service communication uses stronger contracts with gRPC.

In the next part, we will connect Orders and Inventory together. Once they start talking to each other, we will quickly run into the kinds of problems that make distributed systems interesting: timeouts, failures, retries, and handling requests that are still in progress. That will naturally lead us to the next step in the project: Kafka and event-driven workflows.

See you in the next one...

This post follows the same structure as the final project we will build. That means we do it in a clean way from the beginning: proper folders, Postgres migrations, and Docker Compose.

Even if you are new to backend systems, this part will make sense because we will move slowly and explain why each file exists.

WARNING: This is going to be a LONG read so make sure you have enough time to read.

https://media.giphy.com/media/v1.Y2lkPWVjZjA1ZTQ3YjM5OWE5NmpwanpiYWNocnB5aDNsZzFlcTBlNTJwYTI3czQybWNxNSZlcD12MV9naWZzX3NlYXJjaCZjdD1n/XCfEnqL6CSekcYJY4m/giphy.gif

What we are building in this part

We will build the Orders service with these endpoints:

• GET /healthz -> basic health check

• POST /v1/orders -> create a new order

• GET /v1/orders/{id} -> fetch an order

We will store orders in a Postgres database, and we will run everything locally using Docker Compose.

Project structure

Inside the repo, this is the structure we use:

quickbite/ 
  services/ 
    orders/ 
      cmd/ 
        orders/ 
          main.go 
      internal/ 
        db/ 
          db.go 
        http/ 
          http.go 
        store/ 
          orders.go 
      migrations/ 
        000001_create_orders_table.up.sql                 
        000001_create_orders_table.down.sql 
     Dockerfile 
     go.mod 
  docker-compose.yml

Why this structure matters

This layout is common in real Go services:

• cmd/orders/main.go is the entry point. It wires things together and starts the server.

• internal/ contains code that should be used only by this service.

• internal/http contains routing + request/response handling.

• internal/store contains database queries.

• internal/db contains DB connection logic.

• migrations/ contains versioned SQL changes (your DB schema history).

This separation keeps the code easy to grow. When the project becomes bigger (Kafka, outbox, DLQ), you won’t end up with one giant main.go.

Step 1: Create the Orders module

From repo root:

mkdir -p services/orders
cd services/orders
go mod init quickbite/orders

This creates go.mod. It tells Go the module name and tracks dependencies.

Make sure you have Go installed on your system.

Step 2: Add database migrations

For someone who doesn't know what Migration is:

• A migration is a saved database update.

• Instead of putting database setup code directly in your app, you create small files that describe each change, like creating a table or updating data.

• The changes run one by one in the correct order, and the system keeps track of which ones have already been applied.

Create the migrations folder:

mkdir -p migrations

Now create:

services/orders/migrations/000001_create_orders_table.up.sql

CREATE TABLE IF NOT EXISTS orders ( 
    id BIGSERIAL PRIMARY KEY, -- gives each order an automatic unique number.
    user_id TEXT NOT NULL, 
    note TEXT NOT NULL DEFAULT '', 
    status TEXT NOT NULL DEFAULT 'PENDING', -- new orders start as "PENDING" unless changed.
    sku TEXT NOT NULL DEFAULT '', 
    quantity INT NOT NULL DEFAULT 1, 
    created_at TIMESTAMPTZ NOT NULL DEFAULT now() 
);

CREATE INDEX IF NOT EXISTS idx_orders_user_id ON orders(user_id); 
CREATE INDEX IF NOT EXISTS idx_orders_status ON orders(status); 
CREATE INDEX IF NOT EXISTS idx_orders_sku ON orders(sku);

This SQL creates an orders table to store order records, but only if it does not already exist.

The three CREATE INDEX lines make searches faster when looking up orders by:

• user_id, status, sku

Second, create down.sql file:

services/orders/migrations/000001_create_orders_table.down.sql

DROP TABLE IF EXISTS orders;

Why are migrations better than “create table" in code?

So, when you deploy software, code changes, and database changes must move together safely, right? And migrations make the database changes visible, repeatable, and versioned in Git. And so, if you change your schema later, you add a new migration file instead of quietly changing code.

Step 3: Write the DB connection helper

Now that we have our migration files in place, let's connect our application to the PostgreSQL database. And in order to do that, we need to create a helper function called MustConnectWithRetry. Which will try to connect it with proper error handling.

Let's create: services/orders/internal/db/db.go

package db

import (
	"context"
	"log"
	"time"

	"github.com/jackc/pgx/v5/pgxpool"
)

// Keeps trying to connect to Postgres until it works, or until it gives up and crashes.
func MustConnectWithRetry(dsn string) *pgxpool.Pool {
	var lastErr error

	// it will try upto 30 times with the sleep of 0.5 second.
	for i := 0; i < 30; i++ {
		ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
		db, err := pgxpool.New(ctx, dsn)
		cancel()

		if err == nil {
			ctx2, cancel2 := context.WithTimeout(context.Background(), 2*time.Second)
			defer cancel2()

			// if ping succeeds return the pool and you are connected.
			pingErr := db.Ping(ctx2)
			if pingErr == nil {
				return db
			}
			lastErr = pingErr
			db.Close()
		} else {
			lastErr = err
		}
		time.Sleep(500 * time.Millisecond)
	}
	log.Fatalf("db connect failed after retries: %v", lastErr)
	return nil
}

You might ask, why are we writing the logic of retrying?

When the service starts, PostgreSQL may not be ready yet, especially when running with Docker Compose. Instead of failing immediately, this helper function retries the connection several times before giving up.

It creates a Postgres connection pool, checks that the database is reachable with a ping, and returns the pool only when the connection is confirmed to be working. Each attempt is wrapped in a timeout so the application does not hang forever, and a short delay between retries gives the database time to finish starting.

Step 4: Write the database store

Now that the application can connect to PostgreSQL, the next step is to define how orders are stored and retrieved. This file is responsible for the data-access layer for orders. In other words, it contains the code that talks directly to the database.

Create: services/orders/internal/store/orders.go

package store

import (
	"context"
	"errors"
	"time"

	"github.com/jackc/pgx/v5"
	"github.com/jackc/pgx/v5/pgxpool"
)

// These constants are for the status of the order
const (
	StatusPending   = "PENDING"
	StatusConfirmed = "CONFIRMED"
	StatusCancelled = "CANCELLED"
)

// This is how our Order model gonna look like
type Order struct {
	ID        int64
	UserID    string
	Note      string
	Status    string
	Sku       string
	Quantity  int
	CreatedAt time.Time
}

type OrdersStore struct {
	db *pgxpool.Pool
}

func NewOrdersStore(db *pgxpool.Pool) *OrdersStore {
	return &OrdersStore{db: db}
}

// This method inserts a new row into the orders table and returns the generated order ID.
func (s *OrdersStore) Create(ctx context.Context, userID, note, sku string, quantity int) (int64, error) {
	var id int64
	err := s.db.QueryRow(ctx,
		`INSERT INTO orders (user_id, note, status, sku, quantity)
		 VALUES (\(1, \)2, \(3, \)4, $5)
		 RETURNING id`,
		userID, note, StatusPending, sku, quantity,
	).Scan(&id)
	return id, err
}

// Getting the order with ID from DB when user request it.
func (s *OrdersStore) GetByID(ctx context.Context, id int64) (Order, error) {
	var o Order
	err := s.db.QueryRow(ctx,
		`SELECT id, user_id, note, status, sku, quantity, created_at
		 FROM orders WHERE id = $1`,
		id,
	).Scan(&o.ID, &o.UserID, &o.Note, &o.Status, &o.Sku, &o.Quantity, &o.CreatedAt)

	if err != nil {
		if errors.Is(err, pgx.ErrNoRows) {
			return Order{}, pgx.ErrNoRows
		}
		return Order{}, err
	}
	return o, nil
}

We start by defining an Order struct, which represents how an order looks in the application, along with constants for the possible order statuses. Using constants keeps the status values consistent across the codebase and avoids hardcoded strings scattered in multiple places.

The OrdersStore type wraps the Postgres connection pool and exposes methods for creating and retrieving orders. The Create method inserts a new row into the orders table and returns the generated ID, while GetByID fetches an order and maps the selected columns into an Order struct.

Step 5: Write the HTTP layer routes & validation

Now that the database layer is ready, the next step is to expose that functionality through HTTP endpoints. This server layer connects incoming API requests to the order store.

In simple terms, this is the part of the application that receives requests like “create an order” or “fetch order 1” and turns them into database operations.

Create: services/orders/internal/http/http.go

package http

import (
	"context"
	"encoding/json"
	"errors"
	"net/http"
	"strconv"
	"strings"
	"time"

	"github.com/go-chi/chi/v5"
	"github.com/jackc/pgx/v5"

	"quitebite/orders/internal/store"
)

// We need to create a server and this server has the Orderstore. This will give HTTP handlers an access to the store to call its methods 
type Server struct {
	store *store.OrdersStore
}

func NewServer(store *store.OrdersStore) *Server {
	return &Server{store: store}
}

// Lets create Routes now and returns the HTTP router for this service.
func (s *Server) Routes() http.Handler {
	r := chi.NewRouter()

	// Health check endpoint.This used to verify the service is running.
	r.Get("/healthz", func(w http.ResponseWriter, r *http.Request) {
		w.WriteHeader(http.StatusOK)
		w.Write([]byte("ok"))
	})

	// Create a new order.
	r.Post("/v1/orders", s.handleCreateOrder)

	// Get an existing order by id.
	r.Get("/v1/orders/{id}", s.handleGetOrder)

	return r
}

// Here we are creating a `createOrderRequest`This tells the client what data it needs when sending a create order request
type createOrderRequest struct { 
    UserID string `json:"userId"`
    Sku string `json:"sku"`
    Quantity int `json:"quantity"`
    Note string `json:"note"`
}

// Similarly `createOrderResponse`. This tells client what response to expect in return
type createOrderResponse struct {
	ID int64 `json:"id"`
}

// handleCreateOrder handles POST /v1/orders
func (s *Server) handleCreateOrder(w http.ResponseWriter, r *http.Request) {
	var req createOrderRequest

	// Decode request JSON into req.
	if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
		http.Error(w, "invalid json", http.StatusBadRequest)
		return
	}

	// Remove leading/trailing spaces from important string fields.
	req.UserID = strings.TrimSpace(req.UserID)
	req.Sku = strings.TrimSpace(req.Sku)

	// Validate required fields.
	if req.UserID == "" {
		http.Error(w, "userId is required", http.StatusBadRequest)
		return
	}
	if req.Sku == "" {
		http.Error(w, "sku is required", http.StatusBadRequest)
		return
	}
	if req.Quantity <= 0 {
		http.Error(w, "quantity must be > 0", http.StatusBadRequest)
		return
	}

	// Create a request-scoped context with timeout
	// so DB work does not hang forever.
	ctx, cancel := context.WithTimeout(r.Context(), 3*time.Second)
	defer cancel()

	// Insert a new pending order into the database.
	id, err := s.store.CreatePending(ctx, req.UserID, req.Note, req.Sku, req.Quantity)
	if err != nil {
		http.Error(w, "db error", http.StatusInternalServerError)
		return
	}

	// Return the created order id.
	writeJSON(w, http.StatusAccepted, createOrderResponse{ID: id})
}

// getOrderResponse is the JSON shape returned to the client 
// when fetching an order. 
type getOrderResponse struct { 
    ID int64 `json:"id"`
    UserID string `json:"userId"` 
    Note string `json:"note"` 
    Status string `json:"status"` 
    Sku string `json:"sku"` 
    Quantity int `json:"quantity"`
    CreatedAt string `json:"createdAt"`
}

// handleGetOrder handles GET /v1/orders/{id}
func (s *Server) handleGetOrder(w http.ResponseWriter, r *http.Request) {
	// Read the "id" path parameter from the URL.
	idStr := chi.URLParam(r, "id")

	// Convert id from string to int64.
	id, err := strconv.ParseInt(idStr, 10, 64)
	if err != nil || id <= 0 {
		http.Error(w, "invalid id", http.StatusBadRequest)
		return
	}

	// Create a timeout for the DB lookup.
	ctx, cancel := context.WithTimeout(r.Context(), 3*time.Second)
	defer cancel()

	// Fetch the order from the database.
	o, err := s.store.GetByID(ctx, id)
	if err != nil {
		// If no row exists, return 404.
		if errors.Is(err, pgx.ErrNoRows) {
			http.Error(w, "not found", http.StatusNotFound)
			return
		}

		// Any other DB issue becomes 500.
		http.Error(w, "db error", http.StatusInternalServerError)
		return
	}

	// Convert the store model into the API response shape.
	resp := getOrderResponse{
		ID:        o.ID,
		UserID:    o.UserID,
		Note:      o.Note,
		Status:    o.Status,
		Sku:       o.Sku,
		Quantity:  o.Quantity,
		CreatedAt: o.CreatedAt.UTC().Format(time.RFC3339), // standard JSON-friendly timestamp
	}

	// Return the order as JSON.
	writeJSON(w, http.StatusOK, resp)
}

// writeJSON is a helper function to send JSON responses.
func writeJSON(w http.ResponseWriter, status int, v any) {
	w.Header().Set("Content-Type", "application/json")
	w.WriteHeader(status)

	// Encode the value as JSON and write it to the response body.
	_ = json.NewEncoder(w).Encode(v)
}

The Routes method creates a Chi router and registers three endpoints: a health check, a route for creating orders, and a route for fetching an order by ID. This is the entry point that turns the application into a web service.

For the create-order endpoint, we define a request struct that matches the JSON sent by the client. The handler decodes the JSON body, trims unnecessary whitespace, validates the required fields, and then uses a request-scoped timeout before calling the store layer. If the insert succeeds, it returns the new order ID as JSON.

For the fetch-order endpoint, the handler reads the id from the URL path, validates it, queries the store, and maps the database model into a response struct. If the order does not exist, it returns 404 Not Found; otherwise, it returns the order as JSON with a properly formatted timestamp.

Step 6: Write the entrypoint

At this point, we have all the building blocks: database connection logic, the order store, and the HTTP server. The main function is where everything is assembled and the application actually starts.

In simple terms, this is the entry point of the service. Let's create: services/orders/cmd/orders/main.go

package main

import (
	"log"
	"net/http"
	"os"
	"strings"

	"quitebite/orders/internal/db"

	httpapi "quitebite/orders/internal/http"
	"quitebite/orders/internal/store"
)

func main() {
    // getting the port and database url from the `env`
	port := getenv("PORT", "8080")
	dsn := getenv("DATABASE_URL", "postgres://postgres:postgres@localhost:5432/orders?sslmode=disable")
    
    // checking the database connection
	pool := db.MustConnectWithRetry(dsn)
	defer pool.Close()

    // creating an order store
	orderStore := store.NewOrdersStore(pool)
    // creating http server
	srv := httpapi.NewServer(orderStore)

	addr := ":" + port
	log.Printf("orders service listening on %s", addr)
    // starting the server and listening to the port
	log.Fatal(http.ListenAndServe(addr, srv.Routes()))
}

// helper function to get the env values
func getenv(key, def string) string {
	v := strings.TrimSpace(os.Getenv(key))
	if v == "" {
		return def
	}
	return v
}

It begins by reading configuration from environment variables. The service uses PORT to decide which port to listen on and DATABASE_URL to connect to PostgreSQL. Default values are provided so the application can run locally without additional setup.

Next, the code connects to the database using MustConnectWithRetry, which keeps startup reliable in environments where Postgres may take a moment to become available. Once the connection pool is ready, it is passed into NewOrdersStore, which creates the store layer responsible for database operations.

The store is then injected into NewServer, which creates the HTTP API server. Finally, the application logs the address it is listening on and calls http.ListenAndServe to begin accepting requests.

This is how everything works together:

Step 7: Add dependencies and build locally

Now that we have everything in place. Let's add all the required dependencies.

First, we will add chi router package and pgxpool which is the connection pool package from PostgreSQL. So instead of opening one fresh DB connection for every query, your app asks the pool for a connection, uses it, and returns it for reuse.

From services/orders: run

go get github.com/go-chi/chi/v5 
go get github.com/jackc/pgx/v5/pgxpool 
go mod tidy

This will install the dependencies

Step 8: Dockerfile

Now that the application is working locally, the next step is to package it into a container image. This Dockerfile uses a multi-stage build, which means we build the Go binary in one stage and run it in a much smaller image in the final stage.

This approach keeps the final image lightweight, secure, and production-friendly.

# Build stage: compiles the Go library.
# Start from the image that has the Go compiler. Name this stage as "builder".
FROM golang:1.23 AS builder 
# Set the working directory in the container to /app.
WORKDIR /app 
# Copy go.mod and go.sum in that dir
COPY go.mod go.sum ./ 
# Downloads the dependencies.
RUN go mod download 
# Copy the remaining files
COPY . . 
# Compile your go app into a single executable file.
RUN CGO_ENABLED=0 GOOS=linux go build -o orders ./cmd/orders

# Run stage: runs image with the compiled binary.
# gcr.io/distroless/static:nonroot -> minimal Linux image without any additional packages.
FROM gcr.io/distroless/static:nonroot 
# Set the working directory in the container to /.
WORKDIR / 
# Copt the built file from stage 1 (builder) and copy it into this final image. So now the final image contains basically /orders (your executable)
COPY --from=builder /app/orders /orders 
# run the program as non-root user and not as admin/root for better security. If someone gains access to the container, they won't have root access.
USER nonroot:nonroot 
# And the a label saying 8080, this container will listen on port 8080. You still need -p 8080:8080 to map the container port to the host port.
EXPOSE 8080 
# When the container starts, runs "/orders"
ENTRYPOINT ["/orders"]

You might ask why we are using multi-stage build. The reason behind it is that it separates building from running. If you used only a single image, your final container would also contain: Go toolchain, module cache, source code, etc. That would make the image larger and increase the attack surface.

With a multi-stage build, the final image contains only what is needed to run the service.

Step 9: Docker Compose

At this point, we have a database container, a migration step, and the Go API service. Docker Compose lets us define all three in one file and run them together as a small local system.

This is useful because the application does not depend on just one container. It depends on Postgres being available, the schema being created, and only then the API starting up.

services:
    # service name: db
    db:
      image: postgres:16
      # these env variables are read by the postgres image on first startup to initialize the database.
      environment:
        POSTGRES_USER: postgres
        POSTGRES_PASSWORD: postgres
        POSTGRES_DB: orders
      ports: 
        - "5432:5432" # left side is your machine's port, right side is container port.
      # this is where postgres will store its data.
      volumes:
        - dbdata:/var/lib/postgresql/data
      # healthcheck is Compose asking: Are you ready?
      healthcheck:
        # pg_isready checks if the DB is ready to accept connections.
        test: ["CMD-SHELL", "pg_isready -U postgres -d orders"]
        interval: 2s # check every 2 seconds.
        timeout: 3s # each check can take up to 3 seconds.
        retries: 30 # try 30 times before giving up.
    
    # this is a one time job container that runs the migrations.
    migrate:
      image: migrate/migrate:v4.17.1 # popular migration CLI
      # mounts your local migrations folder to the container.
      volumes:
        - ./services/orders/migrations:/migrations
      command: ["-path", "/migrations", "-database", "postgres://postgres:postgres@db:5432/orders?sslmode=disable", "up"]
      depends_on:
        db:
          condition: service_healthy # this makes sure the db is ready before running the migrations.
  
    orders:
      # compose will run docker build using Dockerfile in ./services/orders directory. It creates an image for your Go service.
      build:
        context: ./services/orders
      environment:
        PORT: "8080"
        DATABASE_URL: "postgres://postgres:postgres@db:5432/orders?sslmode=disable"
      ports:
        - "8080:8080"
      #  compose will start db container before orders container starts.
      depends_on:
        migrate:
          # this makes sure the migrations are completed successfully before starting the orders container.
          condition: service_completed_successfully

# define a volume called dbdata. without this, data would be lost when the container is stopped.
volumes: 
  dbdata:

3 Services here:

1) The db service

• Starts a PostgreSQL database for the orders system. It also sets the default database name, username, and password, and exposes port 5432 so the database can be reached from outside the container if needed.

• A volume named dbdata is attached so the data is saved even if the container is removed and started again. The health check is especially important here because it keeps checking whether PostgreSQL is actually ready to accept connections, rather than just assuming the container is usable as soon as it starts.

2) The migrate service

• It is responsible for applying the database migrations. In other words, it prepares the database schema before the application begins using it.

• It mounts the local migrations folder into the container and runs the migration tool with the database connection string. This means the database tables and structure are created from the migration files automatically, instead of relying on manual setup. It also depends on the database being healthy first, which prevents the migration tool from running too early.

3) The orders service

• It is the actual Go application. It is built from the code inside ./services/orders, gets its configuration through environment variables, and exposes port 8080 so the API can be reached from the browser, Postman, or another service. It depends on the migrate service completing successfully, which ensures the app only starts after the database is fully prepared.

Step 10: Run and test

Now it's time to put everything into test and check to make sure it works as expected. After setting up the database, migrations, and API service, the next job is to start the system and verify that each part is responding correctly.

From the root of your app run below commands:

docker compose down -v 
docker compose up --build

The first command stops any existing containers and removes their volumes. This helps you start from a clean state, which is useful during testing because it avoids leftover data from earlier runs.

The second command rebuilds the images and starts the services defined in docker-compose.yml. Rebuilding ensures your latest code and configuration changes are included.

Once the containers are running, test the health endpoint:

curl -s localhost:8080/healthz

This sends a request to the service’s health check endpoint. This endpoint is a quick way to verify that the application is up and responding before you test any business functionality. If this request fails, there is no point in testing the rest of the API yet.

Next, let's create an order and check that endpoint:

curl -s -X POST localhost:8080/v1/orders -H "Content-Type: application/json" -d '{"userId":"u1","sku":"burger","quantity":2,"note":"part 2 test"}'

This makes a POST request to the /v1/orders endpoint with a JSON payload containing:

• userId: the user placing the order

• sku: the item being ordered

• quantity: how many units to create

• note: an optional note attached to the order

If the create request returns something like {"id":1}, You can fetch that order with:

curl -s localhost:8080/v1/orders/1

What we learned

In this part, we built the Orders service. We did not keep everything in one file, and we did not rely on a magic setup. Instead, we created a clean structure (cmd/ and internal/), wrote the database queries in one place, and kept the HTTP logic separate from the database logic. This makes the code easier to read today and easier to grow later.

We also connected the service to PostgreSQL and introduced migrations, which is a big milestone. Migrations solve a common real-world problem: your database schema changes over time, and you need a safe and repeatable way to apply those changes across different machines and environments. By running migrations automatically with Docker Compose, we made our local setup much closer to what a production pipeline looks like.

At this point, we have a working base: we can create orders, store them in Postgres, and fetch them using an API. This is the foundation we will build on in the next parts. In the next post, we’ll introduce the next service (Inventory) and start connecting services together, which is where the system begins to feel like a real distributed application.

See you in the next one...

So I built this app, a food ordering system. And don't think it's JUST a food ordering system. It has a lot of concepts, tools, etc., working internally, which we will learn soon. There is gonna be a mobile app, and the backend is in Go. Over time, I added tools like Docker, PostgreSQL, gRPC, Kafka (Redpanda), and Kubernetes (kind). I also added important reliability patterns like Outbox, Idempotency, and Dead Letter Queues (DLQ). These are the same ideas that show up in real production systems.

In this series, I’ll share what I built and what I learned, step by step, in the same order I learned it. Each post will explain the “why” first, then the “how”.

https://media.giphy.com/media/v1.Y2lkPTc5MGI3NjExdHlqNjV2Ym50ZnUzb3NhcW1rdHlydHRiMXV4bnN2cnJzZTBmemVvbyZlcD12MV9naWZzX3NlYXJjaCZjdD1n/5zf2M4HgjjWszLd4a5/giphy.gif

What we are building

QuickBite is a food ordering app with a very simple flow:

1. A user places an order from the app.

2. The backend creates the order and marks it as PENDING.

3. Inventory checks if the item is available.

4. The backend updates the order to CONFIRMED or CANCELLED.

This looks simple, doesn't it? But it’s a great project because it naturally leads to real backend topics.

The final shape of the system

By the end of this project, QuickBite will have these parts:

• Mobile app: creates orders and shows the order status.

• Orders service (Go): exposes a REST API for the mobile app. It stores orders in PostgreSQL.

• Inventory service (Go): checks and reserves stock. It stores stock and reservations in PostgreSQL.

• Kafka: used for events so services can work asynchronously.

• Kubernetes: used to run everything in a local cluster, like a small production environment.

Even though I built it locally, it follows the same direction a real company system follows. Later, I can move it to AWS, but the learning already happens with local tools.

Why do we need services and events?

At the beginning, it’s tempting to build everything inside one backend. That is fine for learning basics. But once you introduce “inventory”, things get more interesting.

Inventory is not just a function call. Inventory is a part of the business. It has its own rules, its own data, and its own failures. For example, even if the Orders service is working, Inventory might be down. Or Kafka might be down. Or the database might be slow. These failures are normal in real systems, and your code must handle them.

That is why I separated the backend into services and used events:

• Orders create an order and publish OrderCreated.

• Inventory reads OrderCreated, reserves stock, and publishes InventoryReserved or InventoryRejected.

• Orders read the inventory result and update the order status.

Don't worry, we will learn in detail what Services and Events are. But this design makes the system more realistic. It also teaches the most important idea in distributed systems: your system must work even when parts of it fail.

The main workflow (with an example)

Let’s say a user orders burger with quantity 2.

1. The mobile app sends a request:

   • POST /v1/orders

   • payload: { "userId": "u1", "sku": "burger", "quantity": 2 }

2. Orders service writes a row in Postgres:

   • status = PENDING

3. Orders publishes a Kafka event: (We will learn about Kafka in detail so don't stress if you don't know what it is)

   • topic: orders.v1

   • event type: OrderCreated

   • includes: orderId, sku, quantity, eventId

4. Inventory consumes the event and checks stock in its own database:

   • If enough stock exists, it reserves it and stores a reservation record.

5. Inventory publishes back a result event:

   • topic: inventory.v1

   • InventoryReserved or InventoryRejected

6. Orders consume that result and update the order row:

   • CONFIRMED if reserved

   • CANCELLED if rejected

This whole flow is async. That means the user might see PENDING for a short time. This is normal and expected. In the mobile app, I used polling to show the status updates. Later, we can improve this with streaming, but polling is perfect for learning.

What makes this project 'production-style'?

A lot of demos stop at “it works once”. But real systems need more than that. In this app, I also implemented patterns that solve real production problems.

Outbox pattern (so events are not lost)

A common failure is that the service writes to the database successfully, but fails to publish the event. If that happens, the system becomes inconsistent. The order exists in the DB, but Inventory never hears about it.

To fix this, I used the Outbox pattern:

• Save the event into an outbox_events table in the same database transaction as the order write.

• A background worker publishes those outbox events to Kafka.

• If Kafka is down, the outbox keeps retrying later.

This makes the system safer.

Idempotency (so duplicates don’t break things)

Kafka is typically “at least once”. That means a consumer can sometimes receive the same message more than once. This is not a bug. It’s part of the design.

So both services must handle duplicates safely:

• Orders uses a processed_events table to ignore duplicate inventory events.

• Inventory uses a reservations table keyed by order_id so it won’t reserve twice for the same order.

DLQ (dead letter queue) for poison messages

Sometimes you get a message that can never be processed. For example, a message that is not valid JSON. If your consumer keeps retrying forever, it blocks progress.

So I added DLQ topics:

• inventory.dlq.v1

• orders.dlq.v1

Poison messages are sent to DLQ and committed, so the system keeps moving.

Again, You don't need to understand all these in detail. I am just giving you the overview of the whole project. So just read it out and keep it in your mind that we are gonna be implementing these concepts.

What was the real learning part for me?

https://media.giphy.com/media/v1.Y2lkPTc5MGI3NjExbTNsYXNhenZlaXQ4cDMxMWZkbTJlaXdiNTFsMWsxa25xY3k1OGsxdyZlcD12MV9naWZzX3NlYXJjaCZjdD1n/xUXCTpkqwLeh2SrZ4T/giphy.gif

To make sure the system is not fragile, I tested failures on purpose. This is where I learned the most.

Here are examples of the failure tests I ran:

• I stopped Kafka and created an order. The API still accepted the request because of the Outbox. When Kafka came back, events were published, and the order was completed.

• I restarted Inventory and confirmed that stock and reservations were still correct because they were stored in Postgres.

• I produced invalid JSON messages to Kafka and confirmed they went to the DLQ instead of breaking consumers.

These tests are not extra work. They are part of understanding how distributed systems behave.

What this series will cover next

In the next post, I’ll start with the very first step: building the Orders service in Go and connecting it to Postgres. I’ll explain:

• project structure (cmd, internal)

• basic REST endpoints

• Dockerfile basics

• Compose basics

• the first DB schema and migrations

The goal is that you can follow the series and not only rebuild and copy-paste, but also understand why each piece exists.

One thing I learned while building this is tools like Kafka and Kubernetes are not hard because of syntax. They feel hard because they introduce new ways of thinking. The best way to learn them is to build a project where failures happen naturally, and you fix them step by step.

That’s exactly what we’re doing here.

See you in the next blog...

https://media.giphy.com/media/v1.Y2lkPTc5MGI3NjExNWk1YW9yajVrbDluaDJ3ODVubGhrNGQ2amFiZjRrbXU5OTBjb3d0diZlcD12MV9naWZzX3NlYXJjaCZjdD1n/Py1LkJpCEtNiFo0Chr/giphy.gif

But if you see, till now, we were dealing with a single host and a few containers. But in real life, that is not the case. In companies, they have so many products and services, and those services do not run on a single host. They are scattered around the world or in other places. Then you might ask, who manages those services? Is it like someone sitting in front of the display and checking all the time which services are working correctly or not?

If you think that is how they manage, you are mistaken, because managing so many services manually is not possible today. So what's the solution? Who manages it?

To answer that, you have to read this blog. But if you don't want to and just want to know, then here is the short answer:

Docker Swarm / Kubernetes

I bet you want more details than just these two words, right? And to get that, you'll have to read the blog, so, sorry, 😅 I can't give you the short answer. Because, like life, there are no shortcuts to achieving goals. Okay, enough of motivation, let's get to the real thing.

Docker Swarm

We now know how to install Docker, pull images, and work with containers. But as we discussed above, to work with that big scale, we need something that can manage these containers for us. And that's where Docker Swarm comes into the picture.

What is Docker Swarm?

Docker Swarm is a tool that helps you manage multiple Docker machines. Instead of running containers on just one computer, you can connect multiple machines and treat them like a big system. This group of machines is called a cluster.

In a Swarm cluster, each machine is called a node. A node can be a physical server, a virtual machine, a cloud server, or even a small device like a Raspberry Pi. The only thing required is that Docker must be installed, and the machines must be able to talk to each other over the network.

Types of Node

There are two types of nodes: managers and workers.

Managers control the cluster. They decide what tasks should run and which worker should run them.

Workers are the machines that actually run the containers and do the work. Managers tell them what to do, and workers follow those instructions.

Docker Swarm also keeps track of the cluster’s information, like which containers are running and which nodes are available. This information is stored on the manager nodes in a distributed database. The nice thing is that Docker sets this up automatically, so you don’t need to configure it yourself.

If you are concerned about Docker's security, then don't. Because security is built into Swarm. Communication between nodes is encrypted, and each node must prove its identity before joining the cluster. Docker also automatically manages and rotates security certificates, so the system stays secure without extra effort.

Swarm is also used to run and manage applications made of many small services called microservices. In Swarm, the main unit used to run applications is called a service. A service is similar to a container but with extra features. For example, you can easily run multiple copies of it, update it gradually without downtime, or roll back to an older version if something breaks.

There is also one tool called Kubernetes which is a competitor of Docker Swarm. They both orchestrate containerized applications. Well, it's true that Kubernetes is more popular and have more active community and ecosystem. Docker Swarm has it's own benefits too. It's a lot easier to configure and deploy. While it may not be the best solution for bigger companies, but its an excellent technology for small to medium businesses.

I think that's enough of the theory. Let's get to the practical part and see how we can implement this ourselves.

Creating our OWN Swarm Cluster

The example is highly inspired by the book (Docker Deep Dive - Nigel Poulton). The only thing missing there was that he didn't show how to implement it with a single host machine. Because hey, for just the example, we're not gonna have 10 different machines and run and test it, right? So what I did was, I implemented the same thing with the help of AWS. Let's see how we can implement it. The only prerequisite is to have an AWS account. So if you don't have one, I would suggest to make one. I am not gonna show it here otherwise the blog will become lengthy. So make one and then start from here.

Example:

So, in the example, we are going to make a secure swarm cluster with one manager nodes and two worker nodes.

As we saw earlier, that nodes can be virtual machines, physical servers, cloud instances (in our case), or Raspberry Pi systems. The only thing required on those system is to have Docker installed.

Creating 3 EC2 instances ( 1 manager, 2 worker)

Go to https://signin.aws.amazon.com/signin and sign in with the Root user. Once you are in, you will see a dashboard like this.

Now we need 3 machines, right? So let's create 3 EC2 instances. Go or search for EC2 and get in.

Now, click on Launch instance.

Creating Manager Node

Now, we need to name our machine and configure some options.

If you don't know anything about AWS EC2, I think it's the best time to learn. Because in today's time, it good to have some knowledge about it at-least if not a hands on experience. But anyways, you will not be needing to know much about it for this example as it's gonna be really simple.

Here, we are doing nothing but creating a Virtual instances (machines) for us to be able to test Docker Swarm capabilities.

Let's now finish the configuration part. Go ahead and name this instance "mgr1" (this will be our manager). Then select the Ubuntu image

After that, we need to add a key pair. This is required as we are gonna connect this instance to our laptop's terminal and use it. So go ahead and create an RSA (.pem) private key and store it somewhere safe. I named this key docker-swarm-test.

Once this is done, do nothing, just click on Launch Instance. As you can see below, we now have the running manager instance.

Now, we don't need to open these instances to the whole world, right? So what we do is we open a few ports between these instances for them to talk. We can add those ports by going into the Security section. Click on the Security Group link, it will redirect you to the new page where we can add the ports

Click the "Edit Inbound rules".

Add all the ports defined below. I've added the description for each port, what it is used for, so you can take a look at it too. Once done, save the rules, and you are good to go.

Creating Worker Nodes

Now we have one Manager node. We need two worker nodes. The process gonna be the same only except the name of the Instance. Go ahead and create wrk1 and wrk2. I am gonna show wrk1, and then you can create wrk2 on your own. It's pretty simple.

1. Name the instance (wrk1) and select the Ubuntu image.

Select the same key pair that we created.

Now here comes the most IMPORTANT part, so make sure you don't miss it. We need to put all these instances into the SAME VPC and Subnet in order for them to work. How will you make sure? Just open the mgr1 instance (on the right in the screenshot below), go to the Networking tag, and you will see the VPC ID and Subnet ID.

On the worker node (on the left), select the same Subnet ID and VPC ID.

Once done, launch that instance.

IMPORTANT: Make sure you add all the ports like we did for the Manager node.

As you can see now, I have all three instances running on my AWS.

SSH into each instance

Now, let's open these instances in our machine. Open three separate terminals and run below commands

ssh -i ~/Downloads/docker-swarm-test.pem ubuntu@107.21.169.169

Here, make sure you provide the correct path of the key you downloaded (this is the key which gets downloaded when we created the key-pair). And then specify the Public IP Address. Which you will find in the Details tab of the instance

Once done, your terminal will turn into that instance terminal, and you'll see something like this.

Do this for all three instances, and you will have all three instances running inside your host machine in your terminal.

(Top - Manager, Bottom Left - Worker 1, Bottom Right - Worker 2)

Installing Docker on all 3 instances

In order to run Docker Swarm, we first need to install Docker inside these instances. To do that run below commands

sudo apt-get update
sudo apt-get install -y docker.io
sudo systemctl enable --now docker

Run those commands in all three terminals, and you will have Docker installed.

Initializing a new swarm

Now that we have everything in place, we can go ahead and initialize our Docker swarm. One thing to note about is that, Docker nodes that are not part of swarm are said to be in single-engine mode. Once they're added to a swarm they're automatically switched to swarm mode.

Running docker swarm init on a Docker host in single-engine mode will switch that node into swarm mode, create a new swarm, and make the node the first manager of the swarm.

And all the additional nodes can then be joined to the swarm as workers and managers. Joining a Docker host to an exisiting swarm switches them into swarm mode as part of the operation.

Let's run the below command to our manager node to initialize a new swarm.

docker swarm init --advertise-addr <manager-ip-address>

Let's understand the command:

docker swarm init - This tells Docker to initialize a new swarm and make this node the first manager. It also enables swarm mode on the node.

--advertise-addr - As the name suggests, this is the swarm API endpoint that will be advertised to other nodes in the swarm. It will be one of the node's IP address, but can be an external load balancer address.

--listen-addr - This is the IP address that the node will accept swarm traffic on. If not explicitly set, it defaults to the same value as advertise-addr.

To list the nodes in the swarm run sudo docker node ls command.

As you can see, only the mgr1 node is the only node present in the swarm. It's also listed as Leader. Now it's time to add the worker nodes to join this swarm too. To do that run below command in manager node

sudo docker swarm join-token worker

If you want another manager to join the swarm, run:

sudo docker swarm join-token manager

Now go to the worker nodes (wrk1 and wrk2) and join the swarm using the command you got above:

That's it. We've just created 3 node swarm with 1 manager and 2 workers. And you can see that by running sudo docker node ls command into the manager node. Nodes with nothing in the Manager Status are workers.

Also if you notice the asterisk (*) in front of the node ID, it says which node you are logged on to and executing commands from.

Swarm manager high availability

You must be asking, why we added managers and workers, and how they work together, and what's the point of all these?

So in real worlds, it can happen that sometimes one or more nodes can fail, but if you are using Swarm, the survivors will keep the swarm running without interrupting anything.

Right now, we have only one manager, but if you have more than one managers, only one of them is active at any given moment. This active manager is called the "Leader". And the leader is the only one that will ever issue live commands against the swarm.

(Diagram from - Docker Deep Dive book by Nigel Poulton)

Swarm Services

A service in Docker Swarm is a way to run and manage containers across a cluster of machines. It is a feature that only works when Docker is running in swarm mode. A service lets you define how your application should run, and the swarm makes sure that state is always maintained

With services, you can set many of the same options you normally use for containers. For example, you can give the service a name, choose which image to use, map ports, and connect it to networks. But services also add extra features that are useful for modern cloud applications.

You can create services by running docker service create command

docker service create --name web -p 8080:8080 --replica 5 dhruvnakum/web

For example, a command above create a service named web, map port 8080, and run 5 replicas using a specific image. After the command runs, the manager node decides where those containers should run in the cluster. Each node that receives a task downloads the image and starts a container.

Swarm also constantly watches all services. It runs a background process that checks if the actual state matches the desired state. If everything matches, nothing changes. But if something goes wrong, the swarm fixes it automatically.

For example, if one of the machines running a container fails, the number of running containers might drop from 5 to 4. Swarm will notice this and automatically start another container on a different node to bring the total back to 5. This is called self-healing, and it helps keep applications running even when failures happen.

You can see all services in the swarm using the command

docker service ls

If you want to see the containers that belong to a specific service, you can use

docker service ps

Another useful feature is scaling. If your application gets more traffic, you can increase the number of running containers. For example, if traffic doubles, you can scale the service from 5 containers to 10 using the command docker service scale

Example:

docker service scale web=10

web scaled to 10
overall progress: 10 out of 10 tasks
1/10: running
2/10: running
3/10: running
4/10: running
5/10: running
6/10: running
7/10: running
8/10: running
9/10: running
10/10: running
verify: Service converged

When scaling happens, Docker tries to spread the containers evenly across the nodes in the cluster. This helps distribute the workload.

If traffic goes down again, you can scale the service back to a smaller number of containers. For example, you can reduce it from 10 back to 5.

docker service scale web=5

web scaled to 5
overall progress: 5 out of 5 tasks
1/5: running
2/5: running
3/5: running
4/5: running
5/5: running
verify: Service converged

If you no longer need the service, you can remove it using the command docker service rm. When you do this, the service and all of its running containers will be deleted from the swarm.

docker service rm web

Conclusion

Phew!! That was a lot, right? But I hope the practical implementation, with screenshots and everything, helped you get the idea of how swarm works.

Docker Swarm is Docker's native technology for managing clusters of Docker nodes and deploying and managing cloud-native apps.

It's similar to Kubernetes. So if you understood this right now. Understanding Kubernetes will become easier for you.

That's it. I hope you learned something from this. Just want to say this, understanding this only won't make you understand everything until you try it on your own. So make sure you practise this.

See you in the next one, until then....peace

https://giphy.com/gifs/fallontonight-jimmy-fallon-peace-out-tonightshow-2nlbKhgnvAK3sR8ffw?utm_source=iframe&utm_medium=embed&utm_campaign=Embeds&utm_term=https%3A%2F%2Fdhruvnakum.xyz%2F

When people hear Networking, it often sounds complex and scary. But the core ideas are actually simple once you break them down. Let's tackle down and understand the Docker Networking. I will try my best to simplify it, so let's get started.

Docker networking is divided into 3 main parts

1. CNM (Container Network Model) – the design or the plan, you can say

2. libnetwork – the implementation or the code that follows the plan

3. Drivers – the actual network types (bridge, overlay, macvlan, etc.)

A simple way to remember this is:

• CNM is like the blueprint of a house

• libnetwork is like the construction team that follows the blueprint

• Drivers are like the different types of houses you can build (apartment, villa, etc.)

Let’s go step by step and understand these in detail, and make it feel clear.

Part 1: CNM (Container Network Model)

CNM is not software you install. It’s a design document. Docker created CNM to define how container networking should be structured.

CNM says Docker networking is built using 3 building blocks

• Sandbox

• Endpoint

• Network

If you understand these three, you understand the base of Docker networking.

1) Sandbox

A sandbox is an isolated networking environment for a container. That sentence sounds heavy, but here’s what it really means:

Every container gets its own private networking setup, including:

• its own IP address

• its own network interfaces like eth0 inside the container

• its own routing table and rules about where traffic should go

• its own DNS settings

• its own ports

So the sandbox is basically, everything networking related that belongs only to that container.

Think of it like this

• Each container lives in its own room

• That room has its own Wi‑Fi setup, its own address, and its own rules

• Even if two containers are on the same machine, their “rooms” are still separate

On Linux, Docker achieves this separation using something called network namespaces. You don’t have to memorize that right now, just remember the outcome which is, containers behave like separate computers from a networking point of view.

The important thing to remember here is, for example, even when Container A and Container B run on the same Docker host, their networking is still isolated because each has its own sandbox.

2) Endpoint

Your laptop can connect to Wi‑Fi because it has a network card. Containers don’t have physical network cards. So Docker creates a virtual network interface for them. CNM calls this an endpoint.

An endpoint’s job is simple, which is to connect a container’s sandbox to a network.

There are two rules that make endpoints easy to understand

1. One endpoint connects to exactly one network.

   
   1. If a container needs to join two networks, it needs two endpoints.

For Example, If a container must talk to both Network A and Network B, Docker gives it

• Endpoint 1 → Network A

• Endpoint 2 → Network B

That’s how one container can be part of multiple networks at the same time.

3) Network

In Docker, a network is like a virtual switch. It groups endpoints. If two containers connect to the same Docker network, they can usually talk to each other.

If they are not on the same network, they can’t talk to each other by default (unless you set up routing manually).

A very simple analogy is that a Docker network is like a WhatsApp group. People inside the same group can message each other. If you’re not in the group, you can’t message the group members.

Same idea with containers: same network = communication is possible.

Part 2: libnetwork

So far, CNM is just a design. It describes what should exist: sandboxes, endpoints, networks. But Docker needs real code that actually creates and manages these things.

That code is called libnetwork.

So, CNM says what should exist, libnetwork actually creates it and manages it.

Whenever you run commands like:

• docker network create

• docker run

• docker network connect

Docker Engine uses libnetwork behind the scenes. You can think of libnetwork as Docker’s network manager.

Why Docker separated networking from the daemon?

Early on, a lot of Docker networking code lived inside the Docker daemon dockerd. But over time networking became bigger and more complicated:

• multiple network types (bridge, overlay, macvlan…)

• DNS and service discovery features

• load balancing

• multi-host networking

The daemon started becoming too large and messy. That’s why Docker engineers moved networking into a separate library called libnetwork.

This was good because networking could improve without constantly modifying the main daemon, and the design became more modular (cleaner separation), as well as other projects could reuse the networking library

So when someone says networking was ripped out of the daemon and refactored into libnetwork, they mean it was separated into its own component.

Control plane vs Data plane

These two terms show up a lot in networking. Here’s what it means.

Control plane (libnetwork)

The control plane is responsible for deciding how the network should behave and setting up the rules. It does not move packets itself. Instead, it programs the system so packets know where to go. In Docker networking, the control plane is mainly handled by libnetwork.

For Example, when you run docker network create mynet: The control plane performs multiple tasks.

1. Create a Network Object

Docker stores metadata like:

Network Name: mynet
Driver: bridge
Subnet: 172.18.0.0/16
Gateway: 172.18.0.1

This is stored in Docker's internal database.

2. IP Address Management (IPAM)

Docker assigns a subnet to the network.

Example:

Subnet: 172.18.0.0/16
Gateway: 172.18.0.1

Later, when containers join the network:

Container A → 172.18.0.2
Container B → 172.18.0.3

IP allocation is controlled by the IPAM system in the control plane.

3. Select Network Driver

Docker supports different drivers:

• bridge

• overlay

• macvlan

• host

• none

Example:

Driver = bridge

The control plane decides which driver will implement the network.

4. Maintain Network State

Control plane tracks:

• which containers belong to the network

• assigned IPs

• driver configuration

• DNS entries

Example state:

Network: mynet
Containers:
   web -> 172.18.0.2
   db  -> 172.18.0.3

5. Configure System Networking

The control plane tells Linux:

• create bridges

• configure routes

• configure iptables rules

But it doesn't move packets itself. Instead, it programs the data plane.

Data plane (driver)

Data plane means where the real packets move.

Imagine two containers:

Container A
172.18.0.2

Container B
172.18.0.3

They are connected to:

docker0 bridge

now the network structure is like this

Container A
   |
 vethA
   |
 docker0 bridge
   |
 vethB
   |
Container B

Now when A sends a packet to B:

ping 172.18.0.3

The data plane handles the packet movement.

Implementation

Let’s walk through a simple example:

Step 1: Create a bridge network

You run:

docker network create -d bridge mynet

Docker Engine receives the command. It asks libnetwork to create a new network object. libnetwork stores, name=mynet, driver=bridge, settings, IP ranges, etc. And then it calls the bridge driver and says build the real network for this.

The bridge driver then sets up Linux-level things like:

• a Linux bridge device

• NAT / iptables rules

• isolation rules

Step 2: Run a container on that network

You run:

docker run --network mynet nginx

In this case, Docker asks libnetwork to attach this container to “mynet”. So what libnetwork will do is create, a sandbox which is a container’s private network environment and an endpoint.

Then libnetwork connects the endpoint to the network and assigns IP + DNS settings. And finally asks the bridge driver that “connect this endpoint to the bridge”

So a very clean way to remember:

• libnetwork organizes and manages

• the driver does the real network wiring

Single-host Bridge Networks

The simplest Docker network type is the single-host bridge network. The name itself tells you the meaning of it. Single-host works only on one Docker machine. Bridge behaves like a Layer 2 switch.

Every Docker host gets a default bridge network. On Linux it’s called bridge. On Windows it’s called nat.

If you don’t specify a network, containers join this default one

On Linux, this default bridge network is backed by a real Linux bridge called docker0.

Creating your own bridge network

Run the command below to create your own bridge network.

docker network create -d bridge localnet

Then run a container attached to it. For example we have two container c1 and c2.

• run c1 on localnet

docker container run -d --name c1 \
--network localnet \
alpine sleep 1d

• run c2 on localnet

docker container run -d --name c2 \
--network localnet \
ubuntu:latest

Now c2 can reach c1, and very importantly, it can resolve it by name, not just IP.

That leads to an important concept: Service discovery.

Service discovery

Container IP addresses are often not stable. For example, containers restart and may get a new IP, or scaling up/down creates new containers with new IPs.

So you don’t want your app to hardcode IP addresses.

Instead, you want to say: “talk to db”, “talk to api”, “talk to redis”

That is service discovery, automatic mapping from name → IP.

Docker supports this on the same network using an internal DNS system:

Docker runs an internal DNS server that knows container names and IPs, and containers have a local DNS resolver that forwards requests to Docker’s DNS

So if you run ping c1 from inside c2, Docker DNS helps c2 translate “c1” into the correct IP.

This name-based discovery works only within the same Docker network. If two containers aren’t on the same network, Docker DNS won’t resolve them by name for each other.

Port mapping

Containers on a bridge network are mainly meant to talk to each other inside the same host/network. But what if you want a browser, something outside Docker, to reach a container web server? That’s where port mapping comes in.

For example, you have seen this --publish 5000:80 In a command, we write when we run the container. What does that mean?

It simply means that container listens on port 80 and Docker host opens port 5000

your machine : docker container (5000:80)

So the outside world talks to the Docker host, and Docker forwards the traffic to the container. This is extremely common for local development.

Multi-host networking

Bridge networks are great when everything is on one machine. But real systems often run across multiple machines. For example, Host 1 runs some containers. Host 2 runs other containers, and you want containers across hosts to talk like they are on one shared network

By default, containers on different hosts are like they’re in different houses, they’re not on the same local network.

An overlay network solves this by creating a virtual network that spans multiple Docker hosts.

Overlay network

An overlay network is a virtual network built on top of the real network.

For example, you can consider like this:

• real network = normal roads

• overlay network = secret tunnels built on top of those roads

Containers use the tunnels so that, even across different machines, they behave like they’re on the same LAN.

How overlay “tunneling” works

When Container A on Host 1 sends data to Container B on Host 2:

• A sends a normal packet, it thinks B is nearby.

• Docker on Host 1 wraps that packet inside another packet like putting a letter in an envelope

• the outer packet travels over the real network to Host 2

• Docker on Host 2 unwraps it and delivers the original packet to B

Docker has a built-in overlay driver, so you can create an overlay network like:

docker network create -d overlay mynet

Usually this is used with Swarm or other orchestrators

MACVLAN: making containers look like real devices on your LAN

Overlay is about connecting containers across hosts inside a Docker-managed virtual network. Macvlan is a different goal.

Macvlan is used when you want containers to join the real physical network directly.

In other words, instead of hiding containers behind the host like bridge / NAT often does, macvlan makes each container appear like a real machine on your network.

With macvlan, each container gets its own MAC address, its own IP address from your real LAN subnet

So your switch/router sees the container as it sees: your laptop, a server, a VM, and now… the container

macvlan often requires the host network card to be in promiscuous mode. Promiscuous mode is sometimes blocked in corporate networks or cloud providers

So, macvlan is commonly a good fit in controlled environments like data centers.

Overlay vs Macvlan

Overlay:

• Best for container-to-container communication across multiple Docker hosts.

• Containers feel like they are on one shared Docker network.

• Common in clusters, Swarm or microservices

Macvlan:

• Best when containers must be first-class citizens on the physical network

• Containers get real LAN IPs and MACs

• Common when containers must talk to physical servers directly and be reachable from outside without port mapping

Swarm publishing: Ingress mode vs Host mode

In Swarm, you usually deploy services and not individual containers. A service can have replicas running on different nodes. And to allow outside users to access a service, you publish a port.

For example,

• published port (outside) = 5000

• target port (inside container) = 80

Swarm supports two publishing modes:

1) Ingress mode (default)

• Any node can accept traffic, even if it’s not running the container. And the cluster routes it internally to the right node.

• Ingress mode is the default. This means any time you publish a service with -p or --publish it will default to ingress mode.

• To publish a service in host mode you need to use the long format of the --publish flag and add mode=host. Let’s see an example using host mode.

docker service create -d --name c1 \
--publish published=5000,target=80,mode=host \
nginx

2) Host mode

• In this case, only nodes that are actually running the service accept traffic on that port.

In practice, people often talk about routing mesh and load balancing here. If multiple replicas exist, Swarm can spread incoming requests across them.

Conclusion

• So in summary, Docker networking boils down to three things.

  • CNM defines the model - sandbox, endpoint, network.

  • libnetwork is the manager that creates and connects those pieces, and drivers - bridge/overlay/macvlan, do the real packet-moving work.

  • Bridge is for containers on one host, overlay connects containers across hosts like one network, and macvlan puts containers directly onto the physical LAN with their own MAC/IP.

We saw how we can containerize our application and push it to the Docker Hub registry for anyone to use.

In real world, applications are not that simple, right? There are loads of things like, web server, database, cache, some background worker, etc. And things get complicated quickly. If we try to manage this using docker run command and custom scripts then things become messy and hard to maintain.

This is exactly where Docker Compose helps. Docker Compose lets you describe your entire multi-container application in one simple configuration file, and then deploy everything with a single command. Instead of remembering a dozen commands, you maintain one declarative file. Once the app is running, Compose also gives you a small set of commands to manage the whole app’s lifecycle, like start, stop, restart, view status, and tear it down cleanly. Because the configuration is just a text file, you can also store it in Git and treat it like normal code.

Docker Compose works across platforms, and if you use Docker Desktop on Mac, Compose is typically installed automatically. You can quickly verify it by checking the version with docker-compose --version.

Understanding Compose Files (YAML)

Docker Compose uses a YAML file to define a multi-service application. YAML is popular because it’s human-readable and clean for configuration. Technically, YAML is a superset-like format that can represent the same structures as JSON, and Compose also supports JSON, but YAML is the standard and most common format you’ll see in real projects.

By default, Compose looks for a file named docker-compose.yml. If you want to use a different name, for example, a production-specific file, you can pass the filename using the -f flag when you run Compose.

To make this more understandable, imagine a small app built with three services:

• backend

• frontend

• db

Compose lets you define all the services together so they can be started and managed as one unit.

What’s Inside a docker-compose.yml

Just look at the below docker-compose file.

version: "3.8"
services:
  web-fe:
   ...
  redis:
   ...
networks:
  counter-net:

volumes:
  counter-volume:

A Compose file is structured into top-level sections (keys). The file includes mostly these four important top-level keys:

1. version

2. services

3. networks

4. volumes

Other top-level keys also exist (like secrets and configs), but these four are the core ones you’ll see often.

1) version: Compose file format version

The version key is required in this style of Compose file and appears at the top. It defines the version of the Compose file format, you can think of it like the schema/API version for the YAML format. A common mistake is assuming this refers to your Docker Engine version or the Docker Compose binary version it does not. It only describes the format rules for the YAML file itself.

In your example, the Compose file uses version 3.8, which is a widely used format.

2) services: The containers that make up your app

The services section is the heart of Compose. This is where you define each microservice that will run as a container. In the example, there are two services: web-fe and redis.

A really important detail is that Compose uses these service names when it generates container names. So if you define services named web-fe and redis, the containers created will include those names (often combined with the project name, like counter-app_web-fe_1). This naming becomes useful when you’re troubleshooting, checking logs, or inspecting containers later.

3) networks: How containers talk to each other

The networks section tells Docker what networks to create. By default, Compose typically creates bridge networks, which are single-host networks. That means the network connects containers running on the same Docker host. If you need more advanced networking, Compose can reference other drivers, but the default bridge model is perfect for local development and many single-host deployments.

We have the whole dedicated Docker Networking blog upcoming, so stay tuned

In the example, a network named counter-net is defined. Both the service attach to this network, which means they can talk to each other using service names as DNS names.

4) volumes: Persistent data storage

The volumes section defines Docker volumes that should exist for the application. Volumes are used when you want data to survive container restarts and container deletion. This is especially important for databases, but it’s also helpful for development workflows where you want to persist files, cached dependencies, or app state.

In the example, a volume called counter-vol is defined and mounted into the web-fe container. This means some directory inside the container maps to a real persistent storage location managed by Docker.

Breaking Down the Example Services

version: "3.8"
services:
  web-fe:
    build: .
    command: python app.py
    ports:
      - target: 5000
        published: 5000
    networks:
      - counter-net
    volumes:
      - type: volume
        source: counter-vol
        target: /code
  redis:
    image: "redis:alpine"
    networks:
      counter-net:

networks:
  counter-net:

volumes:
  counter-vol:

The web-fe service

The web-fe service is a custom application container. Instead of pulling a prebuilt image from Docker Hub, it uses build: .. That tells Docker: Build an image from the Dockerfile in the current directory. This is a very common development pattern because you keep the application code and Dockerfile together in one repo.

Next, the service defines a command like python app.py. This tells the container what process to run when it starts. In this case, the main process is a Python program called app.py your Flask web app.

It also defines ports, mapping port 5000 inside the container to port 5000 on the host machine. This is what makes the app accessible from your browser. Without a port mapping, the service might run fine but would be isolated inside Docker networking and not reachable directly from your host.

The service attaches to the counter-net network. This is critical because it allows web-fe to communicate with redis over an isolated internal network and it also keeps the service topology tidy instead of throwing everything onto the default network.

Finally, the service mounts a volume. The Compose file mounts counter-vol into /code inside the container. This means files written into /code in the container are actually stored in a Docker managed volume, so they persist beyond the lifecycle of that one container instance.

Putting it all together, Compose deploys one container for web-fe, built from your local Dockerfile, running app.py, exposed on port 5000, connected to the app network, and using persistent storage for the mounted path.

The redis service

The redis service is simpler because it uses an existing image: redis:alpine. This tells Docker to pull the Redis image from Docker Hub if it’s not already present locally.

It also attaches to the same counter-net network, so the web service can talk to it. There’s no need to publish Redis ports to the host unless you specifically want to connect to Redis from outside Docker. Many apps keep Redis internal, accessible only to other containers.

Deploying a Multi-Container App with One Command

Let's clone an example app repo and start it using Compose.

• git clone <https://github.com/red-star25/docker_tutorial_compose_example>

• docker-compose up &

The docker-compose up command is the standard way to bring up an app. When you run it, Compose does several things for you automatically:

1. Builds images that require building (like the web-fe service)

2. Pulls images that need downloading (like redis:alpine)

3. Creates required networks counter-net)

4. Creates required volumes ( counter-volume)

5. Starts containers in the correct configuration

By default, docker-compose up expects the Compose file to be named docker-compose.yml. If your file is named something else, you must pass it with -f. For example, if your file is something-else.yml, you would run docker-compose -f something-else.yml up.

Many people also start apps in “detached mode” using -d, which runs containers in the background and returns your terminal immediately. So instead of using &, you can run docker-compose up -d. Both approaches give you your terminal back, but -d is the standard Compose way.

After the app starts, you can confirm what happened by listing images, containers, and networks. You’ll notice a new image for the web app (created from your Dockerfile) and a container for each service.

docker container ls

docker image ls

At this point, you’ve successfully deployed a multi-container app using Compose.

Managing the App Lifecycle with Compose

Once the app is running, Compose becomes your control panel.

Stopping and removing: docker-compose down

If you want to stop the app and remove the containers and network it created, you use docker-compose down. This is like the tear down the environment command. It shuts down the services and removes the containers (and usually the default network created for the app).

docker-compose down

A key detail from here is that volumes are not deleted by default when you run down. This is intentional. Volumes are meant to store persistent data, so Docker treats them as long-lived resources. That means if your app wrote data to a volume, the data will still be there the next time you bring the app up.

Also, images you built or pulled remain on the system. This is another reason redeployments become faster because Docker doesn’t need to download or rebuild everything again unless something changed.

Bringing it back quickly: docker-compose up -d

If you run docker-compose up -d again after bringing it down, you’ll often see it start faster. The images are already present and the volume still exists, so Compose only needs to recreate containers and the network, then start everything.

Checking status: docker-compose ps

To see what containers belong to the Compose app and their status (running/stopped), you use docker-compose ps. This is more convenient than docker container ls because it focuses on the services in the Compose project.

Seeing processes inside containers: docker-compose top

If you want to see which processes are running inside each service container, you can use docker-compose top. This is useful when debugging “Is my app actually running?” situations.

Stop without deleting resources: docker-compose stop

Sometimes you just want to pause the app without removing containers, networks, and configuration. docker-compose stopstops all containers in the Compose app but keeps them around. You can then confirm their status using docker-compose ps.

Restarting: docker-compose restart

If you’ve stopped the app (or if something is acting strange), docker-compose restart restarts the services. It’s a convenient way to bounce the entire application without destroying and recreating everything.

Key Docker Compose Commands

To wrap everything up, here’s what the common commands mean in simple terms:

docker-compose up starts the whole app (building/pulling images, creating networks/volumes, starting containers).

docker-compose up -d does the same but runs in the background.

docker-compose ps shows the status of the app’s containers.

docker-compose top shows the processes running inside each container.

docker-compose stop stops the containers but keeps them and their resources.

docker-compose restart restarts containers that belong to the app.

docker-compose down stops and removes the app’s containers

docker-compose rm removes stopped containers

Final Thoughts

Docker Compose is powerful because it turns a messy set of manual Docker commands into one clean, version-controlled application definition. It makes it easy to bring up complex multi-service applications on your machine or on a server, and it gives you a consistent way to manage the application over time.

Once you get comfortable reading and writing Compose files especially understanding services, networks, and volumes you’ll find deploying and managing multi-container apps becomes much simpler and far less error-prone.

See you in the next one, until then...

https://giphy.com/gifs/fallontonight-jimmy-fallon-peace-out-tonightshow-2nlbKhgnvAK3sR8ffw?utm_source=iframe&utm_medium=embed&utm_campaign=Embeds&utm_term=https%3A%2F%2Fdhruvnakum.xyz%2F

When you containerize an app, you are basically preparing it so Docker can turn it into an image, and then run that image as a container.

What does “containerizing” actually mean?

Think of containerizing like packing your app into a travel bag:

• Your app code is inside the bag.

• Your app dependencies, like Node.js packages, etc, are inside the bag.

• The instructions on how to start your app are also inside the bag.

Once the bag is packed, anyone can run it the same way, as long as they have Docker.

The usual flow of containerizing an app

Most Docker container workflows follow the same steps:

First, you start with your application code and whatever it depends on (libraries, packages, config files). Next, you write a Dockerfile. This Dockerfile is a set of instructions that tells Docker how to build your app into an image.

After that, you run a Docker build command. Docker reads your Dockerfile line by line and creates an image. Once you have an image, you can store it in an image registry (optional, but very common). Finally, you run a container using that image.

So the flow is:

You have code → you create a Dockerfile → you build an image → you push it to a registry (this is optional) → you run a container.

Example: containerizing a simple single-container app

Let’s say you want to containerize a small Node.js web app.

You copy the code from GitHub and go into the project folder:

• https://github.com/red-star25/docker_tutorial_sample_nodejs_app.git clone this repository

Inside that folder, you’ll find the Dockerfile.

What is a Dockerfile?

A Dockerfile is the starting point for creating a container image. It describes:

• What base system should your app start from

• What software needs to be installed

• What files to copy into the image

• Which command should run when the container starts

One important detail: the folder that contains your app code and files you want to copy into the image is called the build context. Most people keep the Dockerfile at the root of this folder, so it is easy to build.

Also, the name matters: it must be exactly Dockerfile with a capital D, and it must be one word.

Walking through the example Dockerfile

Here is the Dockerfile from the page, and what each line is doing:

The first line is:

• FROM alpine

This means: start with Alpine Linux as the base image. Alpine is a very small Linux image, so it’s popular for small containers. Every Dockerfile must start with a FROM instruction because it sets the base layer.

Then we have:

• LABEL maintainer="dhruv.nakum25@gmail.com"

A label is metadata. It doesn’t install anything or change files in a big way. It’s just extra information added to the image. Adding a maintainer label is a nice practice because people know who to contact about the image.

Next is:

• RUN apk add --update nodejs nodejs-npm

This installs Node.js and npm in the image using Alpine’s package manager apk. This step creates a new image layer because it adds software to the image.

Then:

• COPY . /src

This copies everything in your build context (your current folder) into the image at /src. This also creates a new layer, because you are adding files to the image.

Next:

• WORKDIR /src

This sets the working directory inside the image. From this point onward, commands run as if /src is the current folder. This is usually metadata, not a layer that adds content.

After that:

• RUN npm install

This runs npm install inside /src (because of the WORKDIR). It installs dependencies listed in package.json. This creates another new layer because it adds installed packages into the image.

Then:

• EXPOSE 8080

This tells that the app listens on port 8080 inside the container. This is important for humans and tools, but it’s metadata, not a layer.

Finally:

• ENTRYPOINT ["node", "./app.js"]

This tells Docker what command should run when the container starts. In this case, it runs the Node app. This is also metadata.

Building the Docker image

Once you have the Dockerfile, you build the image using the build command. In this example, the image is called web:latest.

The command is:

• docker image build -t web:latest .

The dot (.) at the end is very important. It tells Docker use my current folder as the build context.

After the build finishes, you can list images to confirm it exists:

• docker image ls

You can also inspect the image to see what settings Docker stored from your Dockerfile:

• docker image inspect web:latest

This is a good way to confirm things like the entry point and the image layers.

Pushing images to a registry (Docker Hub)

Pushing is optional, but it’s very useful. If your image is only on your laptop, no one else can pull it easily. If you push it to a registry, you can download and run it from anywhere.

Docker Hub is the most common public registry and is the default place Docker pushes to.

First you login:

• docker login

Before pushing, you need to tag the image properly. Docker needs three parts:

• Registry (example: docker.io for Docker Hub, even if you don’t write it)

• Repository (often includes your username)

• Tag (like latest, v1, etc.)

If you try to push web:latest directly, Docker will attempt to push it to a repository named web, which you probably don’t own.

So instead, you tag it with your Docker Hub username (Docker ID). Example from the page:

• docker image tag web:latest dhruvnakum/web:latest

This does not remove the old tag. It just adds another tag pointing to the same image.

Now you can push:

• docker image push dhruvnakum/web:latest

Once it is pushed, you can pull it later from anywhere.

Running the container

After the image is built, you run it as a container.

Example command:

• docker container run -d --name mycontainer -p 80:8080 web:latest

Let’s break that down simply:

• -d means run in the background (detached mode)

• --name c1 gives the container a friendly name

• -p 80:8080 maps port 80 on your computer to port 8080 inside the container

• web:latest is the image you want to run

So if your app listens on 8080 inside the container, you can visit http://localhost in your browser because port 80 on your machine is connected to port 8080 in the container.

A closer look: layers vs metadata

Docker reads the Dockerfile from top to bottom, one line at a time. Some lines create real filesystem layers, and some lines only create metadata.

In general:

• Instructions like FROM, RUN, and COPY create layers because they add or change files.

• Instructions like WORKDIR, EXPOSE, ENV, and ENTRYPOINT usually add metadata instead of creating layers.

If you want to see the image build steps and layers, you can use:

• docker image history web:latest

And to verify the final result including entrypoint and layers, use:

• docker image inspect web:latest

Why image size matters

When it comes to Docker images, big is bad.

Smaller images are usually:

• faster to build

• faster to push and pull

• easier to store

• less risky

A common problem is that we install build tools and temporary dependencies, but then we forget to remove them. That makes production images bigger than they need to be.

Multi-stage builds solve this in a clean way. A multi-stage Dockerfile has multiple FROM lines. Each FROM starts a new stage. You can build in one stage, and then copy only the final needed output into the last stage (which stays small and clean).

Build cache best practices

Docker tries to speed up builds using cache.

For each Dockerfile instruction, Docker checks if it already has a cached layer for that exact step. If yes, it reuses it, and the build is faster. If no, Docker builds a new layer.

But here’s the key: once the cache is broken at some step, Docker usually has to rebuild everything after that step too.

That’s why ordering your Dockerfile matters a lot.

Also, Docker checks files you copy. Even if the Dockerfile line didn’t change, if the content of files in the folder changed, the checksum will change and Docker will rebuild that layer.

If you ever want to ignore cache completely, you can build with:

• docker image build --no-cache=true -t web:latest .

Squashing images

Sometimes images end up with many layers. Squashing can combine layers into fewer layers.

This can be helpful when you want fewer layers, but it has a downside: squashed images don’t share layers as efficiently. That can increase storage usage and make pushes/pulls bigger in some cases.

If you want to squash during build, you can add --squash to the build command.

Conclusion

So we learned that containerizing an app is mainly about writing a good Dockerfile and using it to build a clean image. Once you have an image, you can run it the same way on any Docker host, and you can share it by pushing to a registry. That's it.

Next up, we have docker-compose. It's the most fundamental topic for multi-stage builds. And we use it in production. So it's gonna be really fun learning that. But for now, create a simple app, containerize it using Dockerfile, and try it on your own and see if you understood the concept till now. Until then...

We learned about Images. What image is, and how it works internally. In this blog, we are gonna dive deep into Containers and learn what they are, how they differ from VMs, their use cases, their lifecycle, and much more. So let's get right into it.

What is a container?

Containers are one of the main ideas behind Docker. If you understand what a container is and how to run one, most Docker commands will start to make sense.

A container is a running copy of an image. What does that mean!!?

In the previous blog, we learned about images and saw that an image is like a blueprint/class that has everything needed to run an app. And to make use of the class what do we do in programming!!?? Yes, we make objects out of it.

That's what a container is. A container is what you get when you start that image. And you can start many containers from the same image, as you would with a class and an object.

Let's now see the difference between Containers and VMs, because hey, we had VMs earlier, which solved our problems, right? Then why container?

Containers vs Virtual Machines (VMs)

Containers and VMs both run on a host machine (your laptop, a server, or a cloud instance). But they work differently.

How VMs work

VMs use a hypervisor. You might or might not know about a hypervisor, so let me give you the basic idea about it.

Hypervisor

A hypervisor is software that lets you run multiple virtual machines (VMs) on one physical computer. It creates virtual machines, Gives each VM its own CPU, memory and storage. It keeps VMs isolated from each other and shares the physical hardware safely.

So instead of one computer running one OS, you can run: Window, Linux, Another Linux, etc all on the same machine

So each VM has, its own OS and kernel. This is why we say "Hypervisors virtualize hardware."

How do containers work?

Unlike VMs, containers do not create an OS for every app. It shares the host operating system and also shares its kernel.

What it does is split OS resources into isolated containers, meaning each container has its own filesystem, processes, and network space.

This is why we say, “Containers virtualize the operating system.”

One downside of containers is that they are less secure by default than VMs, because isolation is not as strong as that of VMs. There are ways to improve security, but they can add complexity.

Running your first container

Let's now create our first container, and the easiest way to start a container is:

docker container run <image> <app>

Example:

docker container run -it ubuntu /bin/bash

What this does:

• docker container run creates and starts a new container

• -it connects your terminal to the container (interactive mode)

• ubuntu Is the image

• /bin/bash is the app (a shell) that runs inside the container

Your terminal prompt will change because you are now inside the container.

What's happening behind the scenes?

So when you hit enter after running the above command, the Docker client will send this command to the API server running on the Docker Daemon. Docker Daemon will search for the Docker host's local image repository to see if it already has a copy of the requested image. If it finds the image, it will pull it locally, otherwise, it will go to the Docker hub (image repository) and see if it can find it there.

Once the image is pulled, Docker Daemon tells containerd and runc to create and start the container (remember the Docker architecture we discussed in Part -1). Once its done you will be able to get inside the container.

A container runs as long as the main app is running. If the app exists, the container stops. If you stop the container, it stops. If you remove the container, it is gone.

Stopping, starting, and deleting containers

Common lifecycle commands:

• Stop a running container:

docker container stop <container>

• Start a stopped container:

docker container start <container>

• Remove a container forever:

docker container rm <container>

Working with container processes

If you are inside a container running /bin/bash, and you kill that main bash process, the container stops.

To exit the container but keep it running:

• Press: Ctrl-PQ

Now you are back on your host machine, but the container is still running in the background.

Check running containers:

docker container ls

Re-attach using exec

You can open a new shell inside the running container:

docker container exec -it <container_id> bash

This creates a new process inside the container.

So if you type exit here, the container can still stay alive because the original main process is still running.

Container lifecycle

You can give a custom name to your container like this

docker container run --name boo -it ubuntu:latest /bin/bash

After running it, we will be inside the container. Let's create a file here

cd tmp
echo "Hello World" > newfile
ls -l
cat newfile

Detach without stopping it:

• Ctrl-PQ

Stop it:

docker container stop boo

Start it again:

docker container start boo

Go back inside:

docker container exec -it boo bash

Check the file:

cd tmp
ls -l
cat newfile

You will see the file is still there.

So what did we learn from this?

Two important notes

1. That the data lives on the Docker host. If the host fails, the data is lost.

2. Containers are meant to be immutable, so writing data inside them is not a good habit.

That is why Docker has volumes, which store data outside the container. We will look into volumes in the upcoming blog.

Stopping containers gracefully

There is a big difference between stopping and force-killing.

• docker container stop is polite.

  It sends a SIGTERM to the main process (PID 1) and gives it time (usually 10 seconds) to shut down cleanly.

  If it does not stop in time, Docker sends SIGKILL.

• docker container rm -f is not polite.

  It kills the container right away (no clean shutdown).

Restart policies

It’s often a good idea to run containers with a restart policy. Why do we care? Because in the real world, we might have some container that crashes or stops suddenly, and we want them to restart automatically. This is where Restart policies come into play.

Restart policies enable Docker to automatically restart them after certain events or failures have occurred.

Common policies are:

• always - If we attach this policy to the container, Docker will always restart the stopped container if it's closed by Docker or fails.

• unless-stopped

• on-failed (often written as on-failure)

Example: Let's create a container with the always policy attached.

docker container run --name zoo -it --restart always ubuntu:lastest bash

Now If you type exit, and get out of the container, the container stops, but Docker starts it again because the policy is always. Try docker container ls and see.

See how the container was created 19 seconds ago, but has only been up for 11 seconds. This is because the exit command killed it and Docker restarted it. Be aware that Docker has restarted the same container and not created a new one.

Difference between always and unless-stopped:

• always: restarts even after Docker daemon restarts

• unless-stopped: does not restart after daemon restart if it was already in a stopped state

Let's see it in action: Create two container with name always and unless-stopped with respective policies attached

docker container run -d --name always \                                                               
--restart always \
ubuntu:latest sleep 1d

docker container run -d --name unless-stopped \                                                               
--restart always \
ubuntu:latest sleep 1d

Now check with docker container ls

Now lets run, and restart the Docker

docker container stop always unless-stopped

Now run

docker container ls -a

Notice that the “always” container has been restarted but the "unless-stopped" container has not.

The on-failure policy will restart a container if it exits with a non-zero exit code.

Let's clean up everything and remove all the container now.

docker container rm $(docker container ls -aq) -f

Be careful with this command as this command will remove every container running forcefully.

Quick list of useful container commands

• docker container run — start a new container

• docker container ls — list running containers

• docker container ls -a — list all containers (including stopped)

• docker container exec — run a command inside a running container

• docker container stop — stop a container

• docker container start — start a stopped container

• docker container rm — delete a container

• docker container inspect — detailed container info

• Ctrl-PQ — detach without stopping the container

Conclusion

We saw how container is beneficial and works with all the important commands to start, stop, and remove. Also we learned how restart policies plays an important role in Docker ecosystem.

In the next blog, we will implement everything we learned so far by containerizing the application. Untill then...

We learned about some fundamental concepts of Docker and its architecture. We saw why we need Docker in the first place and how it is solving the problems and making things easier than the good old VMs.

In this blog, we are going to understand the core part of Docker, Images. We will see the difference between Images and Containers, and also learn about the Image registries.

We will see the commands that help us pull, create, and run the Docker image and much more. So let's get started

What is a Docker image?

A Docker image is a ready-to-use package that has everything an app needs to run, such as:

• The application code

• The app’s required libraries

• Basic operating system files

If you are a developer, you can think of an image like a class blueprint. You use it to create real running things.

Docker images are mainly used at build time, while containers are what you use at run time when the app is actually running.

Where do Docker images come from?

Most of the time, you do not create images from scratch. You pull them from a place called an image registry, and the most common registry is Docker Hub.

So when we pull an image from the Docker Hub, the Docker daemon downloads it to your computer (your Docker host). After that, Docker can use it to start one or more containers

Images are made of layers

A Docker image is not one single file. It is built from many read-only layers stacked on top of each other.

Inside these layers, we have a small cut-down operating system and all the required files and dependencies for the app. Which means you don't get the full system with all the things.

Docker stacks the layers and shows them as one unified image.

When you pull an image like this:

docker image pull ubuntu:latest

Each line that ends with “Pull complete” is basically one layer being downloaded.

You can also inspect layers using:

docker image inspect ubuntu:latest

The idea here is, every image starts with a base layer, and when we add things (like installing Python or adding your app code), Docker adds new layers on top. For example, we have this Ubuntu image. And then we add Python to it, and then we add source code.

Now, the final image becomes a stack of layers in that order.

(Source: Docker Deep Dive - Nigel Poulton)

Docker also has a system called a storage driver that handles stacking these layers. There are many drivers, but the experience feels the same to you.

Images and containers: how they connect

We will learn about containers in the next blog, but for now, just understand that you can start containers from an image using commands like, docker container run and docker service create. Once a container is created from an image, the image and container are linked, and we cannot delete the image until all containers using it are stopped and deleted. If you try, Docker will give an error.

Also, the purpose of a container is usually to run one app or one service, so the image should only include what that app needs.

For example:

• If the app does not need a shell, the image should not include many shells

• Docker images also do not include a kernel. All containers on a machine share the host machine’s kernel

Image registries and repositories

• An image registry is a central place to store images (for example, Docker Hub).

• Image registries contain one or more image repositories. In turn, image repositories contain one or more images.

Docker Hub has two types of repositories:

• Official repositories

  • Checked and curated by Docker

• Unofficial repositories

  • Can be risky

  • Do not assume they are safe, well-documented, or built correctly

Image names and tags (how you pull the right one)

For official images, pulling is usually:

docker image pull <repository>:<tag>

Examples:

docker image pull alpine:latest
docker image pull redis:latest
docker image pull mongo:4.2.6
docker image pull busybox:latest

If you run:

docker image pull alpine

Docker assumes you mean:

• alpine:latest

Two important notes about latest

If you do not specify a tag, Docker assumes latest. And If the repo does not have a latest tag, the pull will fail.

latest does not mean “newest”. It is just a label. For example, in Alpine, the newest is often tagged edge. So be careful when using latest.

Pulling images from unofficial repos

Same idea, but you include the username or org name:

docker image pull someusername/imagename:version

Pulling from other registries (not Docker Hub)

If the image is in another registry, like there are google registries too, so to get that we include the registry’s DNS name:

docker image pull gcr.io/google-containers/git-sync:v3.1.5

Searching Docker Hub from the command line

You can search Docker Hub using:

docker search alpine

• It searches for repos that match a string in the NAME field

• NAME is the repository name

To show only official repos:

docker search alpine --filter "is-official=true"

Pulling images by digest

Pulling by tag is common, but there is a problem: Tags can change. Someone can push a new version using the same tag. Then you might not know which exact version your running systems are using

For example, you have exampleimage:1.5 with a known bug. You fix it and push it again as exampleimage:1.5Now two different images have used the same tag name over time. It becomes hard to know what is running where

Docker uses a content-based ID called a digest, which is a cryptographic hash.

The idea here is that if the image content changes, the digest changes. So digests are unchangeable, and they uniquely identify the exact image

When you pull an image, Docker often shows the digest:

docker image pull alpine

You can list digests locally:

docker image ls --digests ubuntu

And you can pull the exact same image again using the digest:

docker image pull ubuntu@sha256:d1e2e92...7e495eff4f9

This makes sure you get exactly the image you expect.

Multi-architecture images

Different machines can have different Operating systems (Linux, Windows), different CPU types (x64/amd64, ARM, etc.).

For example, your laptop might be Linux on x64, A Raspberry Pi is Linux on ARM, A Windows server might be Windows on x64

Before multi-architecture support, you had to manually pick the right image, which was confusing.

So, when you run

docker pull golang:latest

Docker automatically detects your OS and CPU, pulls the correct version for your system. You don’t need to specify anything.

How does Docker do this?

Docker uses two important things in the registry:

1. Manifest List: This is nothing but a list of supported platforms for that image tag

   • Example: Linux/amd64, Linux/arm, Windows/amd64

2. Manifests: Each platform has its own Manifest. And that Manifest lists the layers and configuration for that platform

So the Flow is:

docker pull golang:latest
        |
Docker checks manifest list
        |
Finds match for your OS + CPU
        |
Downloads correct layers

Docker also lets you build images for other CPU types using buildx.

Example:

docker buildx build --platform linux/arm/v7 -t myimage:arm-v7 .

You can even build ARM images while working on an x64 machine. How cool is that!!

Deleting Docker images

To delete an image:

docker image rm 02674b9cb179

Delete multiple images:

docker image rm f70734b6a266 a4d3716dbb72

Important rules:

• Deleting an image removes the image and its layers.

• But if a layer is shared by multiple images, it will not be removed until all those images are deleted.

• If an image is used by a running container, Docker will not let you delete it

  • Stop and delete the container first

Shortcut to delete all images (force):

docker image rm $(docker image ls -q) -f

# Here $(docker image ls -q) will list all the images. Its a shortcut.

Conclusion

This is more than enough for you to understand everything about Docker images right now. In the next blog, we will dive into Containers and see what a container is and how it actually works under the hood. Until then...

https://media.giphy.com/media/v1.Y2lkPTc5MGI3NjExbjFtZG8yY3V6azR5ejRoY2p4eXltcWdzcDVrMXVhcjlzdnVrcG5zayZlcD12MV9naWZzX3NlYXJjaCZjdD1n/2nlbKhgnvAK3sR8ffw/giphy.gif

So, I've been reading this book called Docker Deep Dive Zero to Docker by Nigel Poulton. Why? Because I've been working on mobile, web, and cloud applications for quite a long time now. And I really wanted to know how Docker works internally and how it does the magic.

And then I came across this book and thought to give it a try because I love reading, and to be honest, I hardly read technical books to learn about something personally (well, except college and study books, of course lol). So while reading it, I was also making notes for myself.

I will be sharing everything I learned from this book. I will be using the book to explain and rewrite things, making things simpler to understand, because some of the things later on in this book took a while for me to understand. I will also be using Nano Banana to create visuals to help you better understand because I don't want to waste time on graphics too much. And obiously AI does better job than me now in designing these things ;)

So, let's get started without further ado.

The Big Picture

A few years ago, running apps looked very different. Most companies ran one application per server. If the business needed a new app, the IT team often had to buy a new server. This cost a lot of money and wasted resources, because many servers had unused CPU and RAM.

Then virtual machines (VMs) became popular.

VMs Solved One Problem, But Created Another

VMs made it possible to run multiple applications on one server. That was a big improvement. But there was a downside too, which were

• Each VM needs its own full operating system.

• That OS uses CPU and RAM even when the app is small.

• Sometimes each OS also needs its own license.

• VMs can be slow to boot.

• Moving VMs across machines is not always smooth.

So even though VMs helped, they still had a lot of overhead.

Containers: A Lighter Way To Run Apps

To solve the above problem, containers come into the picture. What is a container? For now, just think that a container is similar to a VM in one way. It runs an application in an isolated environment.

But the key difference is that Containers share the host machine’s kernel. They do not need a full OS per container

And because of this, containers are Faster to start, more lightweight, and Easier to move around, which was the problem in VMs, remember?

Google has used container tech for a long time. But for many companies, containers were still too complex to use directly. To address this complexity and make things easier for everyone, Docker Inc. began developing Docker.

What Does “Docker” Mean?

When people say “Docker”, they might mean two things:

1. Docker, Inc. - the company

2. Docker, the technology - the tool that creates and runs containers

Docker, the technology runs on Linux and Windows, and helps you build, run, and manage containers.

Docker Architecture

There are three things that we need to be aware of when we talk about Docker.

1. Docker Runtime

2. Docker Daemon

3. Docker Orchestrator

Let's see each one of them in detail now.

Docker Runtime

Docker uses a “tiered runtime” setup. Low-level and High-level runtime.

When I say Docker uses a tiered runtime architecture, I mean that Docker splits responsibilities between different components instead of having one big program do everything.

Low-level runtime: runc

• This talks to the operating system. Starts and stops containers. Each container typically has a runc instance managing it.

High-level runtime: containerd

• This manages the whole container lifecycle. It pulls images. Sets up networking and calls runc when needed

You do not need to memorize this on day one, but it helps to know Docker has layers.

Docker Daemon

Docker Daemon sits above containerd and performs higher-level tasks, such as exposing the Docker remote API, managing images, volumes, and networks, and more.

The Docker daemon’s main job is to provide an easy-to-use standard interface that abstracts the lower levels.

Docker Orchestrator

Before we understand this layer, we need to understand what orchestration means.

You see, running one container is easy. But in real apps, you often need many containers, like web apps, databases, caches, and background workers.

Orchestration means managing all of those automatically. If you have seen any performance in an orchestra where the orchestrator manages all the musicians, the same concept applies to the Docker too.

• The orchestrator, Start the containers, restart if they crash, scale up when traffic increases, and connect them properly

Docker has its own orchestrator called Docker Swarm, but today most teams use Kubernetes.

Kubernetes in one line (because we are not going into the details right now)

Kubernetes is the most popular platform for deploying and managing containerized apps.

OCI: Why Standards Matter

There is also something called the Open Container Initiative (OCI). In simple terms, it's nothing but a government council that sets standards for the image and runtime formats.

An analogy often used to describe these two standards is rail trails.

This is useful, and it’s fair to say that the two OCI specifications have had a major impact on the architecture and design of the core Docker product.

Installing Docker

Docker is available everywhere to download. There are Windows, Mac, and Linux applications. You can install it in the cloud, on premises, and on your laptop. And there are manual, scripted, and wizard-based installs, too. There literally are loads of ways and places to install Docker.

Docker Desktop

This is the main application we use for everything. Go to Docker Desktop website and download it on your respective operating system. Once installed, you will see the interface something like this:

Docker’s Main Parts

So when you install Docker, you mainly work with two pieces: Docker Client and Docker Daemon

1) Docker Client

This is what you interact with, usually from the terminal.

Open your terminal and run the command below. You will see the current Docker version installed in your system.

Example:

docker version

2) Docker Daemon

This is the background service that does the real work:

• pulls images

• runs containers

• manages networks and volumes

• exposes the Docker API

The client talks to the daemon. This is the high-level explanation. Now, let's get into the details

The Two Core Docker Objects: Images and Containers

Docker Image

A Docker image is like a package that contains:

• a small filesystem like Linux files, or it can be your app, or the dependencies needed to run the app

The important thing here is that it's an image, not a running instance. It is more like a blueprint. You can think of a Docker Image as a Class in the world of programming. And when you create an Object of that Class, it becomes a real thing and gets memory allocated in the system. In this case, that object is Container.

Getting an image onto your Docker host is called pulling. So whenever you want to get the image, you run the command below,

docker image pull ubuntu:latest

And from where this image will come from, you might ask? We will talk about it in the upcoming article, don't worry. For now, just know that it comes somewhere from the internet. And so, after pulling the image, you will see something like this.

To check whether you successfully installed that image, there are two ways: First, to run the command below in your terminal

docker image ls

Each image gets its own unique ID. When referencing images, you can refer to them using either IDs or names

And the second way is to check the Docker Desktop

Docker Container

A container is a running instance created from an image.

So:

• Image = blueprint

• Container = running app

Now that we have an image pulled locally, we can use the Docker container run command to launch a container from it.

docker container run -it ubuntu:latest /bin/bash

Let's understand the command here first:

docker container run tells the Docker daemon to start a new container.

-it flag tells Docker to make the container interactive and to attach the current shell to the container’s terminal.

ubuntu:latest command tells Docker that we want the container to be based on the ubuntu:latest image.

/bin/bash tell Docker which process we want to run inside of the container. For linux we are running a Bash shell.

As you can see, after running the command, we got inside the Ubuntu container. And it's a real Ubuntu image. So you can run the command you run on Ubuntu. But not all of them right now. Why? We will come to it later, don't worry.

You can exit the container without terminating it by Ctr - PQ command

If you want to see all the containers, you can run

docker container ls

Now, if you have exited the container, you can again get back into it using the exec command

docker container exec -it 0acc2d5f2fef bash

We used the -it options to attach our shell to the container’s shell

We can stop and kill the container using the commands below

docker container stop <name/id>

docker container rm <name/id>

IMPORTANT NOTE:

• If you are like this right now...

https://media.giphy.com/media/v1.Y2lkPTc5MGI3NjExcjE1MmVtZ3o4bXF1NzJ2aXMzZ3prYmd1dWU3NnY3ZmZ5ZDcwZ3pjbiZlcD12MV9naWZzX3NlYXJjaCZjdD1n/GPg6PuL5RkeyVCD5wk/giphy.gif

Conclusion

We learned about VMs and saw why it's not efficient to use them anymore. We saw how containers can be so much lighter and faster than VMs.

We also saw what Docker is and how it works using its architecture. We learned about OCI and how it helps maintain the standard practice of images and containers.

We also went through some basic commands of Docker, like:

This big picture view should help you with the upcoming article, where we will dig deeper into images and containers.

See you in the next article, until then....

https://media.giphy.com/media/v1.Y2lkPTc5MGI3NjExbjFtZG8yY3V6azR5ejRoY2p4eXltcWdzcDVrMXVhcjlzdnVrcG5zayZlcD12MV9naWZzX3NlYXJjaCZjdD1n/2nlbKhgnvAK3sR8ffw/giphy.gif

• I've been learning a lot about AWS lately, and to be honest, there's a lot to understand and learn. I decided to take a little break and try to implement what I've learned so far. At the same time, I'm working on a project that's still in progress, and I'll be announcing it soon. I have a backend app in Node.js for this project and wanted to deploy it. So, I thought, why not deploy it on AWS to test my knowledge and see if I've learned anything?

• A little bit about my Node.js app: I'm using JavaScript, and I have MongoDB running. It's on Atlas in the cloud. I think that's all you need to know. If you have a Node.js application running, you can follow similar steps to make your backend accessible to your frontend application or any third-party consumer.

• So, without further ado, let's get started.

Set up an AWS Account

• First, we need an AWS account, so I'm assuming you already have one. If not, it's really easy to create one. Go to https://aws.amazon.com and create an account.

• It might ask for your credit card information, but don't worry, they won't charge you until you exceed the free-tier limit. In this guide, we won't exceed the free tier, so there's no need to worry. Just add your details, and you're all set.

AWS EC2 Instance

• So, we are going to use the AWS EC2 service to deploy our application to AWS. If you're not familiar with EC2, think of it as renting a virtual machine, similar to your laptop or PC. There are many benefits, such as storing data, choosing your operating system, deciding how much computing power you need, selecting the number of CPU cores, RAM, and the type of network you want to use, and more.

• You can customize your virtual machine as you like and rent it from AWS, which is the power of the cloud.

• What we'll do is copy our local app, with all its files and folders, directly to this virtual machine and then run the app there. The advantage is that anyone with the link can access it, not just us on our local machine.

Creating EC2 Instance

• Simply search for EC2 in the search bar and click on it to open the service page.

• Once it's open, you will see a page like this.

  

  • Simply click on a Launch Instance on Dashboard and give it a name.

• It will also ask you to select the operating system image. Choose Ubuntu and scroll down.

• We will use the t2.micro instance because it is part of the free tier. If you need a higher configuration CPU, you can check other instances and select the one that meets your needs.

• To connect to the instance from our local machine or anywhere else, we need access. To do this, we generate a key pair, which we can use to connect to our AWS instance and launch it in our local environment.

• Click on "Generate Key Pair," give it a relevant name, and make sure to select the RSA key pair type. Finally, create the key pair, and it will download to your local machine.

• In the network settings, select:

  • Allow SSH traffic from - My IP - Because we only want to access the instance from our local machine.

  • Allow HTTPS traffic from the internet (so our application can access our Node.js server using an endpoint or URL with HTTPS)

  • Allow HTTP traffic from the internet (so our application can access our Node.js server using an endpoint or URL with HTTP)

• That's it. Now, review the summary and launch the instance.

Connecting via SSH

• Once it’s created, click on Connect to your instance

• This will redirect you to the interface shown below. Here, you can run the instance within AWS Cloud itself and access it, or you can use the SSH Client to access this instance from your local machine, which is what we want.

• Before setting up SSH, remember that we downloaded the Key-Pair. Let's first place it in the correct location.

• I am using a Mac, and I usually store my SSH keys inside the .ssh folder located in the home directory. Let's first navigate to the home directory.

cd ~/

• Now, move the key pair from the download folder to the .ssh folder using the mv command below:

• Once it’s done, we are good to go. So let’s follow these instructions.

• Run chmod A command to change the permission to ensure your key is not publicly viewable.

• Now run

•   ssh -i "<your-key-name>.pem" ubuntu@<your-ip>
  

  to connect to the instance we created.

• Type yes and press Enter

• As you can see, we are now inside the instance we created. Isn't it cool that we can access it from our local machine?

• Now you can perform any tasks that you would normally do on your local machine.

Installing/Updating Packages

• Now that we have our instance connected, before starting anything, we need to ensure we are using the latest packages in Ubuntu and install any available updates. To do this, we run two commands:

sudo apt update

• Which refreshes package lists (checks for updates) and then

sudo apt upgrade

• Which Installs available updates for installed packages

Installing Node.js in an EC2 instance

• Now that we have all the latest packages, we need to install Node.js (or the backend framework/library you need).

• To do this, we need to download and run the NodeSource setup script, which configures the system to install Node.js version 20 from their official repository.

• Use the command below to do that:

curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash -

• Once the NodeSource repository is added, we need to install Node.js and npm (Node.js’s package manager) on our system to use Node.js.

• To do this, run the command below

sudo apt-get install -y nodejs

Pushing Local App to EC2

• Now that we have the Node environment set up, we can upload our local Node.js app to this instance. We can do this using the SSH connection we already set up.

rsync -avz --exclude 'node_modules' --exclude '.git' --exclude '.env' \
  -e "ssh -i ~/.ssh/your-key.pem" \
  . ubuntu@ip-address:~/app

• In simple terms, this command copies the current folder (.) to the remote server’s ~/app directory using SSH. It skips unnecessary files like node_modules, .git, and .env because we don't need those.

NOTE: Be sure to replace ‘your-key’ with your SSH key name in the command above, and replace ‘ip-address’ with your actual IP address.

• To execute this command, go to your code directory and run the command above.

• If you now run the ls command in your instance's home directory, you will see the app folder has been created.

  

• 

  And if you use cd to enter it, you will see all your source code.

Setting up Environment (.env) File.

• If you have a database running locally, like PostgreSQL or MySQL, you can install those on this instance just like we did for Node.js.

• I have MongoDB running on MongoDB Atlas. So, to allow this instance to access the database, we need to add the instance's IP to the whitelist. Simply copy the instance's IP address and paste it into the IP Access List entry.

• Now that everything is set up, we need one more file: the .env file. Remember, we excluded it when we transferred our files to the instance.

• So let’s create the .envfile

cd app/
sudo vim .env # This will create and open .env file

# .env
# paste your environment variables

• You can exit the Vim editor by pressing esc, then : (colon), typing wq, and pressing enter.

• As you can see, we now have the .env file in place.

• Before we run our app, let’s make sure we have downloaded all the required packages by running

npm i

• Now, if we run the app, it should run without any errors.

Configuring Security Rules

• To check if it is running, go to the instance dashboard, select your instance, copy the Public IP address, and paste it into your browser.

• We can't see anything, and it didn't work. Why? Do you remember the security group we set up earlier?

• We haven't specified our IP address and because we are accessing it from our machine we must add our IP address with port numner. So, let's add our IP address and the port number to the inbound rules and see if it works now.

• Click on Add Rules and add Custom TCP and specify your IP by selecting My IP.

• Now, if you run it, it should work fine. I am hitting /test the endpoint, which shows Hello World text.

Running App in Background (systemd)

• But here we have two major problems, and I hope you already know what they are.

• One big issue is that it's running on PORT 7575 and doesn't have a domain name. The second issue is that it's running on our local terminal, so if we close this terminal, we won't be able to access our server using that public IP address.

• First, let's address the second issue. To fix that, we need to find a way to run the application in the background so that no matter what we do in the terminal, the server will stay up the whole time.

• If you want to run your Node app as a background service, we use the systemd command.

• We need to create a service file. This file tells systemd (Linux’s init system) how to start, monitor, and restart your Node.js app as a background service, similar to how SSH or MongoDB are managed.

• Run the command below to create a new service file.

sudo vim /etc/systemd/system/myapp.service

• And paste lines below in it once Vim is opened

[Unit]
Description=Node.js App 
After=network.target multi-user.target

[Service]
User=ubuntu
WorkingDirectory=/home/ubuntu/app
ExecStart=/usr/bin/npm start
Restart=always
Environment=NODE_ENV=production
EnvironmentFile=/home/ubuntu/app/.env
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=myapp

[Install]
WantedBy=multi-user.target

• In the [Unit] section, we include our app description and ensure our app starts after networking is ready (so Node can bind to a port).

• In the [Service] section, we specify how the app will run by including the working directory, environment, and other settings.

Now run

sudo systemctl daemon-reload

• Which Reloads systemd so it notices the new service file.

sudo systemctl enable myapp.service

• Enables the service to auto-start on boot.

sudo systemctl start myapp.service

• Start it right now (without reboot).

sudo systemctl status myapp.service

• Shows current status, logs, and whether it’s running or failed.

• And now, if you kill your terminal/app and still hit the URL, it should work.

Setting up Reverse Proxy (using Caddy)

• The next step is to allow HTTP access directly, without using PORT 7575. Then, we will add a domain name and an SSL certificate.

• To keep your actual web server, localhost:7575, hidden from the world and exposing only the HTTP and HTTPS ports, which are 80 and 443, we use reverse proxies like Nginx or Caddy.

• A reverse proxy is a server that sits in front of your application servers and manages all incoming client requests before sending them to your backend app, such as your Node.js service.

• We use this for security purposes. It allows us to add a firewall before requests reach the actual server. It also enables load balancing, as the reverse proxy can distribute requests to multiple servers.

http://54.86.58.33:7575/

We do

https://api.mydomain.com

• To implement this, there are many services like Nginx and Caddy. For this example, we'll use Caddy because it's easy to set up.

• Caddy is a modern, lightweight reverse proxy and web server, similar to Nginx but with a major advantage:

• Visit this URL and run all the commands.

sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https curl
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list
chmod o+r /usr/share/keyrings/caddy-stable-archive-keyring.gpg
chmod o+r /etc/apt/sources.list.d/caddy-stable.list
sudo apt update
sudo apt install caddy

• Once it is done, let’s remove the custom TCP rule we added in the security group for our IP

• And now, if we run

sudo systemctl start caddy

• And then visit the IP address 54.86.58.33 Without mentioning any port, you should see a Caddy page

• But what we want is to forward the traffic to our application instead of showing this page. So, we will edit the Caddy file by opening it using the command below:

sudo vim /etc/caddy/Caddyfile

• Comment down the root and file_server line and uncomment the reverse_proxy line and add your port.

• And now let’s restart Caddy

sudo systemctl restart caddy

• If you again run this URL, you should be able to see your app.

• That’s it, now you can use this URL anywhere to access the backend APIs.

• You can also set up a custom domain name and attach it to this current IP address (for example, example.com) or something like this using the Route53 service.

Conclusion

• Thank you for taking the time to read my blog. I hope you enjoyed it and learned something new. I'll be sharing more about AWS soon as I continue to learn. Keep learning!!

• Until then…

• Connect with me on LinkedIn, Github, and Twitter.

• In the last part, we successfully put the Go application in a Docker container. Now, we need to deploy it so others can access it.

• First, we need to upload our Docker application to Docker Hub. We'll start with that and then set up a GitHub workflow action to help us deploy our application to AWS EC2.

• You might wonder why we use GitHub Actions. In real-world projects, things change often, and you can't manually deploy the application every time. To automate this, we use a tool like GitHub Actions.

• Let's begin with Docker Hub, and then we'll create the workflow file.

Upload Go Application to DockerHub

• • DockerHub is like GitHub but for Docker images. It's an online place where developers can find and share Docker images.

    * You make your custom image on your computer and push it to DockerHub for others to use.

    * As developers, we just need to pull these images using the command docker pull, and we're ready to go.

    * Now, let's upload our Go application to DockerHub.Step 1: Login to Dockerhub

• First, we need to log into the docker hub in order to upload.

• Run the below command.

docker login

• This takes you to the browser where you can log in or sign up. Once you do that, you'll see a message confirming your login was successful.

  

  • Once you are logged in, go to hub.docker.com

Create Repository

• Just like GitHub needs a repository to store code, Docker Hub needs a repository to store Docker images. Go to the Repositories tab in the Navbar.

• And create a public repository. I already have one, so I won't create it again.

Push Image

• Once you've created the repository, you can push the image we made using the docker push command. But first, we need to tag our local image as latest, as shown below.

docker tag anonymous-go dhruvnakum/anonymous-go:latest

• And now let’s push it

 docker push dhruvnakum/anonymous-go:latest

• Replace dhruvnakum with your username, and if you named the repository differently, use that name instead.

• After you run this, the image will be successfully pushed to the repository we just created.

• 

  Now, let's create a workflow to automate deployment.

Creating the GitHub Actions Workflow

• To automate deployment, we will set up a GitHub Actions workflow.

Create the Workflow File

• In the project, create a new file at .github/workflows/cicd.yml.

Define the CI/CD Pipeline

name: Deploy Go Application

on:
  push:
    branches: 
      - ec2

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Source
        uses: actions/checkout@v4
      - name: Create .env file
        run: echo "PORT=${{ secrets.PORT }}" >> .env
      - name: Login to docker hub
        run: sudo docker login -u ${{ secrets.DOCKER_USERNAME }} -p ${{ secrets.DOCKER_PASSWORD }}
      - name: Build docker image
        run: sudo docker build -t dhruvnakum/anonymous-go .
      - name: Push image to docker hub
        run: sudo docker push dhruvnakum/anonymous-go:latest
  deploy:
    needs: build
    runs-on: self-hosted
    steps:
      - name: Delete old anonymous-go container
        run: sudo docker rm -f anonymous-go  
      - name: Pull new anonymous-go docker image
        run: sudo docker pull dhruvnakum/anonymous-go
      - name: Delete old mongo container
        run: sudo docker rm -f mongo-demo
      - name: Pull new mongo docker image
        run: sudo docker pull mongo:8.0
      - name: Delete old network
        run: sudo docker network rm anonymous-go-nw
      - name: Create networks
        run: sudo docker network create anonymous-go-nw
      - name: Run mongo container
        run: sudo docker run --name mongo-demo --network anonymous-go-nw -d mongo:8.0
      - name: Run docker container
        run: |
          sudo docker run --name anonymous-go \
          --network anonymous-go-nw \
          -p 3000:3000 \
          -e MONGODB_URI="mongodb://mongo-demo:27017" \
          -e JWT_SECRET="secret" \
          -e REDIS_SECRET="secret" \
          -e REDIS_ADDR="redis-11068.c261.us-east-1-4.ec2.redns.redis-cloud.com:11068" \
          -e REDIS_PASSWORD="DM4iBq2pTYNQkweBZmwqGKyDYrj872M8" \
          -e PORT=3000 \
          -d dhruvnakum/anonymous-go:latest

Let me explain what's happening here:

• First, we named the workflow Deploy to Go Application.

• To be safe, I created a new branch called ec2 and pushed all the code to it. We want the workflow to run automatically whenever this branch is updated. Here's how we do it:

on:
  push:
    branches: 
      - ec2

• We have two jobs: Build and Deploy.

• In the Build job, we download our project's code, build a Docker image from it, and push it to Docker Hub.

• In the Deploy job, we pull the Docker image of our project and the Mongo image from Docker Hub, run them inside one container as we did before, and finally, we run that container.

Build Job

• We want this job to run on the latest Ubuntu operating system.

jobs:
  build:
    runs-on: ubuntu-latest

• Now we want to download our project’s code from GitHub inside this OS, so we write this:

name: Checkout source code
uses: actions/checkout@v2

• Since we have environment variables in our code, we need to make them available here. To do this, we must add these secrets in GitHub Action Secrets. Go to the project's settings, and under secrets and variables, add these secrets.

• Now, to get this from there, we create .env inside this OS and push everything:

- name: Create .env file
  run: echo "PORT=${{ secrets.PORT }}" >> .env

• Now that we have all the secrets. We Login to the docker:

- name: Login to docker hub
  run: sudo docker login -u ${{ secrets.DOCKER_USERNAME }} -p ${{ secrets.DOCKER_PASSWORD }}

• Then we build the docker image of our project:

- name: Build docker image
  run: sudo docker build -t dhruvnakum/anonymous-go .

• Finally, we push the image to the docker hub:

- name: Push image to docker hub
  run: sudo docker push dhruvnakum/anonymous-go:latest

Deploy Job

• In this job, everything we write will run on the AWS EC2 instance provided by GitHub.

• We want this job to start after the build job finishes successfully.

deploy:
    needs: build

• In this job, we first want to delete all the old containers and pull the latest ones. Once that is done, we run the container using docker run command.

deploy:
    needs: build
    runs-on: self-hosted
    steps:
      - name: Delete old anonymous-go container
        run: sudo docker rm -f anonymous-go  
      - name: Pull new anonymous-go docker image
        run: sudo docker pull dhruvnakum/anonymous-go
      - name: Delete old mongo container
        run: sudo docker rm -f mongo-demo
      - name: Pull new mongo docker image
        run: sudo docker pull mongo:8.0
      - name: Delete old network
        run: sudo docker network rm anonymous-go-nw
      - name: Create networks
        run: sudo docker network create anonymous-go-nw
      - name: Run mongo container
        run: sudo docker run --name mongo-demo --network anonymous-go-nw -d mongo:8.0
      - name: Run docker container
        run: |
          sudo docker run --name anonymous-go \
          --network anonymous-go-nw \
          -p 3000:3000 \
          -e MONGODB_URI="mongodb://mongo-demo:27017" \
          -e JWT_SECRET="secret" \
          -e REDIS_SECRET="secret" \
          -e REDIS_ADDR="redis-11068.c261.us-east-1-4.ec2.redns.redis-cloud.com:11068" \
          -e REDIS_PASSWORD="DM4iBq2pTYNQkweBZmwqGKyDYrj872M8" \
          -e PORT=3000 \
          -d dhruvnakum/anonymous-go:latest

Before we move ahead I forgot to change the URL in the main from localhost to 0.0.0.0 . So make sure you have this inside main.go at the end.

log.Fatal(r.Run("0.0.0.0:" + os.Getenv("PORT")))

Setting Up AWS EC2 Instance

• Before deploying the project, we need to set up an EC2 instance.

• To do this, log in to the AWS Management Console and go to the EC2 dashboard.

• Use this link: https://aws.amazon.com/ec2/

• It might ask for your card details, but don't worry; they won't charge you unless you exceed the limit, which you won't be charged for for this purpose.

• Search for EC2 and click on it.

  

• Now click on the “Launch Instance.”

• Now, give the instance name and select the Ubuntu image

• Also, select Key pair. If you don’t have one. You can create it by clicking on the Create new key pair button.

• Once it is done, click on the Launch Instance button on the right

• Since our app is running on port 3000, go to the Security Group settings and allow inbound traffic on port 3000 to make sure the application is accessible.

• To do that, click on the Instance ID

• Then click on the Security tab and there click on Security Group

• Click on Edit inbound rules and then add custom TCP with Port range 3000 as shown below and save the rules.

• Now, we are ready to run the instance.

Setting up Docker inside EC2 Instance,

• Our instance is ready. First, we need to make sure Docker is installed. To do this, let's connect to the instance. Click the Connect button after selecting the instance.

• Once you do that you will see a window like this:

• To confirm whether docker is there or not, run docker command, you will most likely see this:

  

• Now, to install everything, run the below commands

sudo apt-get update && sudo apt-get install docker.io -y && sudo systemct| start docker && sudo chmod 666 /var/run/docker.sock && sudo systemcti enable docker && docker --version

• Once it is done, run docker version to see if it’s installed correctly.

Register a Self-Hosted GitHub Runner

• A self-hosted runner is simply a computer or server that you own (like an AWS EC2 instance) that runs GitHub Actions instead of using GitHub’s default machines.

• To set up, go to the project settings, and inside Actions, you will see this runner tab.

• Click on `New self-hosted runner:

• Select Linux

• And run all the commands you see there inside your EC2.

• Once you run the last step ./run.sh command, you will see a runner with the status idle :

• But there is one problem, that runner does not run on detached mode in our instance, If you kill it by pressing CTRL+C, you wont be able to run it.

• To resolve this issue, just run the below command

sudo ./svc.sh install

sudo ./svc.sh start

• Now, your running is running in the background, and you can perform other operations inside your instance.

• Finally, we are done with it. Now, we can test our application.

Testing Your Application

• To test, commit to the ec2 branch that we created. Once you do that, Go to the Actions tab, and there you will see both build deploy services are running.

• Once it is done, you will see both marked as green ticks, which means that everything went well and the docker is successfully up inside the ec2.

• To test, go to the instance we created and find the Public IPv4 address. Copy it, paste it into Postman, and run any endpoint in your application.

• As you can see it’s working now.

• Phew! I know that seems like a lot at first. But once you follow these steps, you'll get comfortable with it.

Wrapping Up

• If you've made it to the end of this guide, you truly deserve a big thank you. So, thank you for sticking with it and reading all the way through. I hope you've gained a lot of valuable insights from this single blog post, covering everything from creating a GitHub Workflow to understanding Actions, setting up a Runner, and working with AWS EC2.

• Keep diving deeper into these topics and explore other related concepts, as they are essential for cloud development in today's fast-paced tech world. Mastering these skills will undoubtedly enhance your ability to build and deploy applications efficiently.

• I look forward to sharing more knowledge with you in the next blog post! Until then, keep learning and experimenting with new technologies.

Reference: https://www.youtube.com/watch?v=sSAWMr_-Co4&t=409s

• Connect with me on LinkedIn, Github, and Twitter.

• Hey, Gophers! I hope you're doing well. I'm back with Part 2 of the series. In the last blog, we learned about Docker and how it makes development easier with Containers.

• We also created two containers: MongoDB and Redis.

• Now that we have both running in Docker containers, it's time to build a custom image for a Go application. In this part, we'll learn about the Dockerfile, docker-compose file, and how to dockerize the application. By the end of this tutorial, you'll have a fully functional dockerized application that you can pull onto your machine and use without installing any dependencies.

• Before that, I'll give you a quick overview of the project. I won't go into too much detail to keep the blog short. I'll just explain what the project is about, and then we'll get started with the main part. Let's do that!

Project Overview

• This is a simple project, but using all this tech together will be really fun. The project is about making a post anonymously, kind of like a tiny version of Reddit. It's just for learning, so it's okay if it's not very big. We'll only ask for a username and password, nothing else. The user signs up and logs in with these two things.

• Users can create posts, and they can also like and comment on other people's posts.

• Here's the mind map of what we'll use in our Go app and which APIs we'll implement.

• By the way, the project's name is Anonymous-GO.

Note: This blog requires basic knowledge of Go and how to work with APIs. I won't explain everything from the beginning. If you want to start from the basics, I already have a series on that. You might want to check it out first:

https://dhruvnakum.xyz/from-zero-to-go-hero-learn-go-programming-with-me-part-1

• Here is the final project if you want to go through.

Setting Up Go Project

Project Structure

• We will structure our project as follows:

• In config/, we have functions for setting up Redis.

• controllers/ contains all the main operations like auth, post, like, and comment. This is where we define all the route handlers.

• In database/, we set up MongoDB and have the User and Post collections.

• In middleware/, we define handlers for accessing private routes.

• In models/, we define different models like User, Post, Comment, and Like.

• In utils/, we have helper functions for JWT authentication, and for hashing and verifying passwords.

• We also have the .env file for storing all the project secrets.

• We will discuss docker-compose.yml, Dockerfile, and Dockerfile.dev in the next section.

Running MongoDB Containers

• If you read the previous blog, you might have a MongoDB container ready inside your docker. Let’s first run that.

  

• Before we start, we will use a MongoDB container for database storage and Redis Cloud for caching.

• To run the project, we first need to start the MongoDB container to connect the database with the application.

• When you run the project, you will see that both MongoDB and Redis are connected.

  

• Let’s now dockerize this application.

Dockerize GO Application

• Before we start, let’s recap things we learned about Docker in our previous blog.

What is Docker Image?

• As we saw in our previous blog, Docker Image is a package/bundle that is used as a template to create containers.

• It contains everything needed to run an application. The docker image contains the application code. In our case, it is our Go code.

• It also contains dependencies that we use in our application, like any libraries, frameworks, etc.

• Contains all the tools and settings.

• But the thing is Docker image cannot run on its own. As it’s just a template. To run the Image, we need to create a Container, just like Class and Object, Class is nothing without an Object, right!?

What is Docker Container?

• As Docker Image cannot run on its own, it requires Containers. Containers are basically running instances of a Docker Image.

• You can think Docker Image as Class and Docker Containers as Objects. Class is nothing but a template and we can make use of class by creating it’s objects.

• Every containers run in its isolated environment. And this containers runs exactly the same on every machine. Whether it’s Windows, Mac, or Linux.

• As you can create multiple objects from a defined class. You can create multiple containers from the same image.

• Now you might be asking that, Okay, I understand Image and Containers now. But how do we create an Image of our application right?

• Well that’s where Dockerfile comes into picture.

Dockerfile

• A Dockerfile is nothing but a set of instructions that tells Docker how to build a Docker Image.

• Let’s see how we can write a docker file

FROM golang:1.23-alpine AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -o main .

FROM alpine:latest
COPY --from=builder /app/main /main
COPY .env .
EXPOSE 3000
ENTRYPOINT ["/main"]

• Let’s understand what’s happening here

• First of all, we want Docker to compile our application, right? How do we do that? Well, Docker Hub already has the docker compiler image for us. It has all the tools, like go build go mod, etc.

• So, we are using this docker image as our base. You might ask why we are using this image only and not the other one. The simple answer to that is it’s lightweight.

  

FROM golang:1.23-alpine

• The line above does that task. We also added AS builder after it. This simply labels this stage as "builder," so we can refer to it later.

• You might ask why we need builder a stage.

  • Well, we don’t want the final application to have every Go tool. It’s unnecessary. So, instead of storing unnecessary files in the final image, we directly build the app in this stage and then copy only the final compiled binary file into a new lightweight image.

  • This keeps the docker image small, fast, and secure.

WORKDIR /app

• Here, we are creating a working directory inside the container. So, when we say /app, this means we are creating an app folder inside the container.

• You might ask why we are creating app directory, can’t we just push all the things directly. Well, you can, but it’s not recommended. Why? Because If we don’t do this, every time we need to execute cd command manually to go inside /app folder

COPY go.mod go.sum ./

• The COPY instruction is used to copy the files from your local machine to docker containers.

• It will copy it inside /app as we previously set the current working directory as /app

• You might ask why we are copying this. well, otherwise, our application won’t run. These files contain all the dependencies required by our application.

RUN go mod download

• This command is used to download and cache all dependencies specified in go.mod

COPY . .

• Now that we are done with the basic setup, we need to copy all the files and folders of our application into a docker container.

• The . (dot) here refers to the current directory on your local system and second . (dot) refers to the current working directory inside the container.

RUN CGO_ENABLED=0 GOOS=linux go build -o main .

• Now that we have copied our files, folders, and dependencies, we are ready to compile the application inside the container. This is what the above RUN instruction does.

• go build -o main . tells Go to compile the application and generate an executable file named main.

• The . at the end means compiling all Go files in the current directory (/app). which is main.go in this case.

• CGO allows Go programs to use C libraries, but it also makes the binary dependent on system libraries. And we don’t want that, so we are disabling it by assigning 0 value.

• GOOS=linux is used to build the application for Linux. As we are using Alpine Linux image, we need to do that to make it lightweight for the system.

FROM alpine:latest

• Now that we have compiled the Go application in the previous builder stage. We now need to run the final binary which is created inside container.

• To do that, we are using alpine container to create a lightweight container. This removes all the unnecessary libraries and tools.

• In the previous stage (golang:1.23-alpine AS builder), we compiled the Go application.

• Now, we only need to run the final binary, so we switch to a clean, minimal image.

So what exactly happens here:

• Docker starts with a fresh Alpine Linux image.

• No Go compiler or extra tools are installed, just a clean OS.

• The final container only contains what's needed to run the application.

Why Not Just Use golang:1.23-alpine for Everything?

• That’s a good question. It’s because The Go compiler and tools are only needed for building/compiling the app, not for running it. Keeping them in the final image wastes space and increases security risks. Using Alpine separately ensures a much smaller, optimized final container.

COPY --from=builder /app/main /main

• This line copies the compiled Go application from the builder stage into the final Alpine-based image.

• /app/main this is where the Go binary was built in the builder stage

• And /main this is the destination inside the Alpine container

EXPOSE 3000

• With this instruction, we are telling Docker that this container is expecting traffic at port 3000.

ENTRYPOINT ["/main"]

• ENTRYPOINT ["/main"] tells Docker to execute the compiled Go binary (/main) inside the container.

• So now, when you run the container, it automatically runs the /main binary.

How does this overall process work?

First Stage (builder)

• Uses golang:1.23-alpine

• Builds the Go application (main) inside /app

• This stage has all the necessary tools (Go compiler, dependencies)

Final Stage (alpine:latest)

• Starts with a clean Alpine Linux image

• Copies only the main binary from /app/main into /main

• The container is now ready to run the application

Does that mean we are using two Containers?

• No, we are not creating two containers. Instead, we are creating two separate environments during the build process

• Docker processes both stages during the image build phase.

• It uses the first stage (builder) to compile the application.

• It then discards the first stage and only keeps the final image from the second stage.

• The container is created from this final, lightweight image.

docker-compose

• Now our Dockerfile is ready. But right now, our app is only containerized. Remember, we are also using a MongoDB image in our app. If you try to run the app right now, it will throw an error because the app cannot connect to MongoDB.

Why Do We Need docker-compose.yml?

• Our app and MongoDB are running in separate containers.

• The app depends on MongoDB to function, but without a shared network, they cannot communicate.

• Running both services manually using docker run commands is inefficient.

• This is where Docker Compose comes into play.

• Docker Compose allows us to define and run multiple containers as a single service, ensuring that all required services (our app and MongoDB) start together, communicate seamlessly, and work as a unit.

• With a docker-compose.yml file, we can:

  • Define multiple services (e.g., app and mongo) in a single configuration.

  • Ensure MongoDB starts before the app using depends_on.

  • Use a shared network so the app can communicate with MongoDB by referring to it as mongo instead of localhost.

  • Set up environment variables for both services.

  • Use volumes to persist MongoDB data, preventing data loss when containers restart.

• Now let’s write docker-compose for our app:

services:
  app:
    build:
      context: .
    ports:
      - "3000:3000"
    depends_on:
      - mongo
    environment:
      - MONGODB_URI=mongodb://admin:admin@mongo:27017/
      - JWT_SECRET=secret
      - REDIS_SECRET=secret
      - REDIS_ADDR=redis-11068.c261.us-east-1-4.ec2.redns.redis-cloud.com:11068
      - REDIS_USERNAME=default
      - REDIS_PASSWORD=DM4iBq2pTYNQkweBZmwqGKyDYrj872M8

  mongo:
    image: mongo:8.0
    environment:
      MONGO_INITDB_ROOT_USERNAME: admin
      MONGO_INITDB_ROOT_PASSWORD: admin
    ports:
      - "27017:27017"
    volumes:
      - mongodata:/data/db

volumes:
  mongodata:
    driver: local

• So, as we know, we need two services: app and Mongo

• And that’s why we are defining those inside services

1. app Service:

   • This service is used to build the app using Docekerfile.

   • So, we are defining its context with the current directory by specifying . (dot).

   • Then, we want to map the container port to the local system’s port (called port mapping). We do that inside port tag.

   • Also, we want mongo-db to run before our app starts. To achieve that, we have to use depends_on tag and specify mongo image.

   • Also, if you look at the project, we have specified many environmental variables. And so we want this app service to use that. So, we specify all the environment variables inside ennvironment tag.

2. mongo Service:

   • We first need to pull up the Mongo image. We are doing it by specifying it inside the image tag.

   • As we did in our app, we are specifying env variables inside the environment tag.

   • Then, we need to expose port 27017 for database access.

   • Also, we want to keep the data in a persistent volume. So, we are using volumes

Running docker-compose

• Now that we are done with the docker-compose file, we can now run the container using the following command.

docker compose up -d

• It will start the container inside the docker.

• As you can see, our entire application is now dockerized.

• Now, let’s test our API to see if it’s working or not.

• Let’s create a user with a signup API

• Now, let’s test the login functionality with Redis caching.

• As you can see, Redis-session is added to cookies for accessing private routes. Let’s also test private routes to make sure they're working.

• 

  And yes, it is working. And now, if you try to access private routes after logging out. It will not work. Let’s check that.

  

  - We logged out.

• 

  As you can see, it’s throwing an error.

What’s Next?

• Currently, our APIs are working locally. But we want developers to use these APIs, right? To do that, we need to deploy this.

• Now, there are many deployment options available in the market.

• The most popular one is AWS.

• In the next part, we are going to push our custom image to Docker Hub, set up GitHub actions, and Upload and Run this docker container inside the AWS EC2 instance.

• It’s gonna be really good learning for anyone who is not familiar with AWS deployment. Because I learned a lot.

• Let’s meet in the next one, until then…

• Hey, Gophers! Thanks for stopping by and taking the time to read my blog. If you’ve been following my previous blogs, you might know that I recently started writing about Go. In my last article, we explored JWT authentication using the JWT and Fiber packages in Go.

• After covering the basics, I learned more advanced topics, such as working with databases, handling cookies and sessions, deploying Go applications on the cloud, using Docker, containerizing Go applications, etc.

• And to put all these concepts into practice, I built a project that combined them all. Honestly, it was quite a challenge since AWS, Dockerization, Redis, and GitHub Actions were new to me. But after overcoming everything, I’m excited to share everything I’ve learned with you.

• In this series, I’ll break down each concept in a simple and easy-to-understand way. If anything is unclear, feel free to ask in the comments!

• By the end of this series, you’ll have a fully deployed, containerized project running in the cloud. If you're a beginner, this project will be a great addition to your resume, as we’ll be implementing many industry standard practices used in real-world applications.

• We first need to learn a very important concept/tool called Docker. Because nowadays everyone is using it.

• So, let’s first start with Docker and understand what Docker is. Why do we and people in the software industry use it? And What problem does it resolve?

Why do we need Docker?

• Before diving into what Docker is, let's first understand the issues it helps to resolve.

• Imagine an organization with hundreds of developers, each working on different machines with different operating systems and configurations. Now, consider a project where a team of 50 developers is collaborating on the same GitHub repository.

• As the team grows, new developers are hired and provided with new systems to work on. To get started, they need to clone the project from the repository and set up the development environment. Let's take two developers as an example:

  • Developer 1 – A current employee who has already set up the project environment.

  • Developer 2 – A new employee who has just joined and needs to set up the project from scratch.

• Developer 1, who has been working on the project from the beginning, has already set up all the necessary environments. Since this is a Go project, he runs the server locally using the following versions of key dependencies:

  

• Now, the real challenge begins when Developer 2 (a new employee) joins the team and needs to run the project on his system.

  Challenges Faced by the New Developer

  1. Manual Setup Hassle

     • The new developer first clones the project from the repository.

     • He then needs to install multiple dependencies manually (Go, MongoDB, Redis, etc.).

     • In real-world projects, there are even more dependencies to install, making the setup tedious and error-prone.

  2. Operating System Differences

     • Developer 1 might be using Windows, while Developer 2 has a Mac.

     • Some project commands may work on Windows but not on Mac.

     • Certain OS-specific compatibility issues might arise during the setup.

  3. Version Mismatches

     • Developer 2 might install the latest versions of dependencies, which could be incompatible with the project.

     • Some dependencies may have breaking changes, causing unexpected issues.

  4. Cloud Deployment Challenges

     • Eventually, the project will need to be deployed on the Cloud.

     • The cloud server might have a different environment, requiring another round of installations and conflict resolution.

     • Even after setting everything up, there’s no guarantee that the project will work as expected.

  5. Scalability Issues

     • Today, it’s just one new developer-facing these challenges, but as the team grows, the setup process will have to be repeated for every new team member.

     • Constant communication and troubleshooting with each developer become inefficient and time-consuming.

• This is where Docker comes into play. It provides a way to package the entire environment, ensuring that everyone (developers and cloud servers) runs the same setup without manual installations or version mismatches.

• In the next section, we’ll dive into what Docker is, why it is useful, and how it solves these problems.

How does Docker resolve this problem?

Docker Container

• Docker is a platform that allows developers to create, deploy, and run applications inside containers.

• What is a Container?

  • You can think of a container as a box that holds everything needed to run an application: its code, runtime, dependencies (like Go, MongoDB, Redis), and any required configurations.

• With Docker, we can create a container that includes all the necessary tools, packages, and dependencies. This container ensures that:

  • Every developer gets the exact same versions of Go, MongoDB, Redis, and other dependencies.

  • The application runs identically on every system.

  • The container is lightweight, meaning it doesn’t take up unnecessary resources like a full virtual machine would.

• As you can see, this resolves the very big issue of installing and setting up the project environment again and again, and we can save our developers from the manual errors that they can make, as seen above.

To conclude, With Docker, we don’t have to install things like MongoDB or Redis manually again and again. Instead, we can run them inside containers and start using them instantly!

• Okay, so now that we understand containers, there is one more term that we need to understand, which is Images.

Docker Images

• A Docker image is like a recipe that contains all the necessary ingredients and instructions to prepare a dish. However, instead of food, a Docker image contains everything needed to set up and run a software application.

  Understanding Docker Images with an Example

  • Imagine you have a laptop. Without an operating system (Windows, Mac, or Linux), your laptop is just a piece of hardware that can’t do much. Similarly, a Docker container needs a Docker image to function.

  • Think of a Docker image as the "operating system" for a container. It includes:

    • The application code

    • All the dependencies (like Go, MongoDB, Redis, etc.)

    • The runtime environment

    • Any necessary configurations

  • Once a Docker image is created, it serves as a blueprint to generate multiple containers. Each container runs independently but follows the same instructions defined in the image, just like you can cook the same dish multiple times using the same recipe.

  • With Docker images, developers can ensure consistency across different environments, making application deployment seamless and reliable.

• I hope everything is clear now.

• If not, then don’t worry; we are now going to do some hands-on exercises and see everything that we have learned so far in action.

Docker Installation

Docker Desktop

• To run Docker containers on your machine, you need to install the Docker Desktop application.

  You can download it from the official website:

• Docker Desktop Installation

• Or simply search “Docker Desktop install” online, and it will take you to the installation page.

• Docker Desktop is a GUI (Graphical User Interface) that makes it easy to:

  • Manage containers

  • View and control Docker images

  • Monitor volumes and networks

  • Configure Docker settings

• It provides a user-friendly way to interact with Docker without using only command-line tools, making container management more accessible.

• Once you are done with the installation, open the terminal of your choice and run docker command

• It should show a similar result. This tells us that docker is successfully installed in our system. You can also verify using.

docker --version

• Now open your Docker desktop.

• If you open Docker Desktop, you’ll notice several sections on the left-hand side:

  • Containers – Shows running and stopped containers.

  • Images – Lists downloaded and built Docker images.

  • Volumes – Manages persistent data storage for containers.

  • Builds – Tracks built images and processes.

• We've already covered Containers and Images, and that’s all we need to get started!

• Okay, so for our upcoming project, we’ll need MongoDB and Redis as dependencies. Instead of manually installing them and setting up everything from scratch (which is boring and time-consuming 😩), we can use Docker to simplify the process.

• With Docker, we can:

  • Pull pre-built images of MongoDB and Redis.

  • Run them as containers.

  • Avoid dependency conflicts or setup issues.

• This means no more frustrating installations, we just run a command, and everything works!

• Before that, let’s understand what Docker hub is.

Docker Hub

• You might be wondering when I say we need to pull Docker images for MongoDB and Redis, where these images actually come from. It comes from Docker Hub. Docker hub is similar to Github but for docker images.

• It’s an online repository where developers can find and share Docker images.

• You can also create your custom image on your system and push it to the docker hub for others to use. We are going to create a custom image for our Go application and push it to the docker hub in the future.

• What we, as developers, have to do is just pull these images using the command docker pull, and we are all set.

Running MongoDB in a Docker Container

• Instead of manually installing MongoDB, we can use Docker to set it up in seconds with the following command:

docker run -d --name mongo-demo \
-e MONGO_INITDB_ROOT_USERNAME=admin \
-e MONGO_INITDB_ROOT_PASSWORD=admin \
-p 27017:27017
mongo:8.0

• Let’s understand this command:

  • docker run -d : It runs this container in a detached mode. This means that if we do not provide this tag, our terminal will be in use, and we won’t be able to perform any other operations. So, we need to run this container in the background

  • —name mongo-demo : We are naming our container as ‘mongo-demo’

  • Now, for security purposes, we are setting the Username and Password for our database using -e MONGO_INITDB_ROOT_USERNAME and -e MONGO_INITDB_ROOT_PASSWORD

  • We are also mentioning the port We want Mongo to run locally in our system.

PORT MAPPING

• When we run MongoDB inside a Docker container, it operates in an isolated environment. This means it does notautomatically connect to our local machine.

• If we try to access MongoDB from outside the container, it won’t work unless we explicitly expose its port.

• This is where Port Mapping comes in!

Here, -p 27017:27017 means:

• The first 27017 (before the :) is the port on your local machine (host).

• The second 27017 (after the :) is the port inside the container.

By default, services running inside a Docker container are not accessible from outside.

• MongoDB inside the container is running on port 27017.

• But our system (local machine) doesn’t know that unless we explicitly expose it.

• By mapping MongoDB’s internal port (27017) to our system’s port (27017), we make it accessible.

• At the end, we are specifying which image to pull, and that is mongo:8.0

• If you search Mongo in the docker hub, you will find its image shown below.

  

• What that above command will do is :

  • Pulls the MongoDB 8.0 image from Docker Hub (if not already downloaded).

  • Starts a MongoDB container with a default admin username and password.

  • Allows us to interact with MongoDB just like we would on a locally installed version, but without the manual setup!

• Once it is done, you will see this container running in your Docker Desktop.

• To check if it’s running or not, you can run the below command

docker ps

• This will list all the running containers

• Now, let’s run the Redis container

Running Redis in Docker Container

• Just like MongoDB, we can set up Redis instantly using Docker without manually installing anything. Use the following command:

docker run -d --name redis-demo redis:latest

• This will pull the latest Redis image (if not already available locally) and then start a Redis container that runs in the background.

  

• Again, to verify whether it’s running or not, let’s run docker ps

• As you can see, our Mongo and Redis containers are working pretty well.

Now that we have MongoDB and Redis running inside containers, we don’t need to install them manually on our system. Anytime we need them, we can just start the containers, and everything will work instantly.

Wrapping Up

• With Docker, we’ve reduced the hassle of manual installations and ensured a consistent and error-free development environment. No more dependency issues or version conflicts, just a simple setup in seconds!

What’s Next?

• This blog serves as the foundation for the next steps in our series. Now that you understand docker and how it works. We are ready to containerize our Go application.

• Stay tuned for the next part! 🚀 Until then.

• Connect with me on LinkedIn, Github, and Twitter.

• After the last series on Go, I've been learning some new ideas, like Secure Authentication. I discovered JWT authentication and how it helps secure routes.

• In this article, I'll explain how we can use JWT to authenticate users and secure routes.

• So, let's jump right in.

Creating a Base Folder Structure

• We'll start by setting up a project structure to clearly see how everything fits together.

• First, create a directory and open your favorite code editor. I prefer VS Code.

bmkdir go-auth
cd go-auth
code .

• So this is our project structure :

cmd/server/main.go

• This is the entry point of the application

controllers/auth_controller.go

• This is where we place our controllers/handlers. In our case, it will be auth_controller. Here we will implement all the functions related to authentication like Register, Login, and Logout.

database/db.go

• In this database, setup is done and the connection is made. We will be using MySQL as our database.

middleware/middleware.go

• Middleware is simply a function that wraps around our HTTP handlers to add extra features. For example, once a user is logged in, we use middleware to secure other routes like /dashboard and /cart. We will learn about middleware in this blog.

models/user.go

• The model represents the data structure of the application’s data. So for example, if you are building an application related to a library, models can be, a book, author, etc. In this project which we are building, we need user a model as we are dealing with the user’s authentication.

routes/routes.go

• We define all the HTTP routes here. We are using fiber a package for managing the application routes. If you are familiar with express in Node.js then this package is inspired by it and provides flexible ways and functionalities to manage routes.

types/types.go

• In this, we define our own custom struct types which we require in our application.

.env

• This is an environmental file where we put all the secure stuff.

Load .env File

• First of all, let’s load all the environmental variables in our application.

• For that, we are using a package called gotdotenv. This helps us achieve this.

go get github.com/joho/godotenv

• Let’s load this env file inside our application. Go to main.go and paste below code

import "github.com/joho/godotenv"

func main() {
    err := godotenv.Load()
    if err != nil {
        log.Fatal("Error loading .env file")
    }
}

• We place it at the top of the main function because we need all the variables loaded before the server starts.

Connect Database

• We are using MySQL to work with the database. I was going through some queries and everything as It’s been a while since I wrote any. And then I came across the term ORM (Object-Relation Mapping). And I found it super convenient as we can use objects oriented approach to execute the queries.

• So an ORM is a programming technique that allows us to interact with a relational database like MySQL using object-oriented programming principles, instead of writing raw SQL queries. It serves as a bridge between the object-oriented language and the database, mapping database tables to objects in the application.

• And researching more about it I found this package called gorm for Go. So we will be using this to work with our database.

• Let’s install it and also we need to install the mysql driver for it separately.

go get -u gorm.io/gorm
go get gorm.io/driver/mysql

• Let’s now create Connect function and write the logic to connect our application with the database.

package database

import (
    "fmt"
    "os"

    "gorm.io/driver/mysql"
    "gorm.io/gorm"
)

func Connect() error {
    dsn := fmt.Sprintf("%s:%s@tcp(127.0.0.1:3306)/go_auth?charset=utf8mb4&parseTime=True&loc=Local", os.Getenv("DB_USERNAME"), os.Getenv("DB_PASSWORD"))
}

• Inside Connect() we first initilized dsn (Data Source Name). dsn is nothing but a string that provides all the information to connect the database. It includes information like database type, server location, username password, database name, etc.

• Here we used fmt.Sprintf which returns a formatted string. We used two string %s placeholders for username and password. And we are getting those values from the .env file.

• Let’s first add those values to our .env

// .env
DB_USERNAME=root
DB_PASSWORD=root123

func Connect() error {
    dsn := fmt.Sprintf("%s:%s@tcp(127.0.0.1:3306)/go_auth?charset=utf8mb4&parseTime=True&loc=Local", os.Getenv("DB_USERNAME"), os.Getenv("DB_PASSWORD"))
    db, err := gorm.Open(mysql.Open(dsn), &gorm.Config{})
    if err != nil {
        return err
    }
}

• After that, we open the connection to the MySQL database using Gorm's Open() function. We use the mysql driver's Open function to connect with our dsn string. The second parameter is for gorm configuration if you have any.

• This gorm.Open function returns two things: db instance and error. So what we need to do is, first we need to create a global gorm.DB variable that our application can use from anywhere inside the project.

var DB *gorm.DB

func Connect() error {
    // ...
}

• And that’s why we create var DB *gorm.DB .It’s a pointer because we need whatever changes we made in this variable, it reflects the original one which is db inside the Connect function.

func Connect() error {
    dsn := fmt.Sprintf("%s:%s@tcp(127.0.0.1:3306)/go_auth?charset=utf8mb4&parseTime=True&loc=Local", os.Getenv("DB_USERNAME"), os.Getenv("DB_PASSWORD"))
    db, err := gorm.Open(mysql.Open(dsn), &gorm.Config{})
    if err != nil {
        return err
    }

    log.Info("Database connected successfully")
    DB = db

    db.AutoMigrate(&models.User{})
    return nil
}

• And that is why we assigned db to global DB variable.

• Then we have this db.AutoMigrate(&User{}) function. What this AutoMirgrate function does is, it creates the same table with all the values defined inside this model User inside the database.

• This means that GORM will ensure that the table corresponding to the User model in the database matches the structure defined in the model, creating or updating columns as necessary.

• And so when you run the application for the first time, it will create a table for you automatically.

• We haven’t created our User model yet so let’s create it first.

User Model

package models

type User struct {
    Id       
    Name    
    Email    
    Password
}

• Here we’ve created a simple User struct, which contains basic information like ID, Name Email, and Password.

• Now that we have the database file ready we can use the Connect function inside our main function to connect the database.

func main() {
    err := godotenv.Load()
    if err != nil {
        log.Fatal("Error loading .env file")
    }

    if err = database.Connect(); err != nil {
        log.Fatalf("Could not connect to the database, %v", err)
    }
}

• Now let’s run the project to see if it’s working

go run cmd/server/main.go

Setting Up the Router

• Okay so for route management, as we discussed earlier we are using fiber package. To install run the following command inside your terminal

go get github.com/gofiber/fiber/v2

• Let’s create an instance of fiber inside main.go to use it inside our application

func main() {
    //...

    app := fiber.New()
}

• Using this app we can now create our routes.

• To make things look clean and separate we will implement all the routes inside routes.go.

// routes.go
func SetupRoutes(app *fiber.App) {
    app.Post("/api/register", controllers.Register)
    app.Post("/api/login", controllers.Login)
    app.Post("/api/logout", controllers.Logout)
}

• Here we have created SetupRoutes a function that takes a app variable of type *fiber.App.

• Inside this, we created three routes, /register, login/ and /logout. (Don’t worry if you are seeing an error saying these functions are not found. We are going to implement them in a bit)

• Now we call this function in main.go like this

func main() {
    //...

    routes.SetupRoutes(app)
}

• Now let’s implement these handler functions inside controllers.

Implementing Route Handlers

Register

// controllers/auth_controller.go
func Register(c *fiber.Ctx) error {
    var regReq types.RegisterRequest
}

• Since we receive the submitted data in an HTTP request, we need to create a custom struct type to convert all that data into a format that Go can understand.

• Let’s create a RegisterRequest struct for the Register handler

// types/types.go

type RegisterRequest struct {
    Name    
    Email   
    Password
}

• Let’s now parse the request body of the register to this type

func Register(c *fiber.Ctx) error {
    var regReq types.RegisterRequest

    if err := c.BodyParser(&regReq); err != nil {
        return c.Status(fiber.StatusInternalServerError).JSON(fiber.Map{
            "message": "invalid request format",
        })
    }
}

• Function BodyParser helps bind the body fields to a struct field. It returns an error of type ErrUnprocessableEntity which means if BodyParser fails to bind the fields it will throw this error.

• So we handled it by returning an error message “invalid request format“ with a status code 500 (StatusInternalServerError)

• Now before we create our user and send it to the database to store, we can’t just send the password as it is. We need to secure it by converting it into some hash or something.

• Go has this package called bcrypt which helps us achieve that. First, let’s install the package

go get golang.org/x/crypto/bcrypt

• bcrypt has a function GenerateFromPassword that takes a byte of password string as the first parameter and cost as the second parameter (cost determines the computational complexity of the hashing process. It is a measure of how long it takes to hash a password, directly affecting the time it takes to verify the hash later)

• Let’s hash our password and then create a user object

func Register(c *fiber.Ctx) error {
    //...

    password, _ := bcrypt.GenerateFromPassword([]byte(regReq.Password), 10)

    user := models.User{
        Name:     regReq.Name,
        Email:    regReq.Email,
        Password: password,
    }
}

• This bcrypt function returns two values: encrypted password and error.

• And after that, we can construct our user model by using regReq struct and encrypted password.

• Now before we send this to the database, we need to first check whether this user already exists in the database or not.

func Register(c *fiber.Ctx) error {
    // ...   

    database.DB.Where("email = ?", user.Email).First(&user)
}

• We can do this using the Where function of the MySQL database. Since each email is unique in our application, we wrote a query that returns a user if they exist.

• Now we check if we found anything or not like this:

func Register(c *fiber.Ctx) error {
    // ...   

    database.DB.Where("email = ?", user.Email).First(&user)

    if user.Id != 0 {
        return c.Status(fiber.StatusConflict).JSON(fiber.Map{
            "message": "user already exists",
        })
    }
}

• If user.Id is not 0, it means we found a user, so we return an error message saying "user already exists."

• If we didn't find anything, we create a new user in the database using the Create() function of DB.

func Register(c *fiber.Ctx) error {
    // ...

    database.DB.Create(&user)
}

• And once everything is done, we send a message back to the API response saying "user created successfully" along with the user information.

func Register(c *fiber.Ctx) error {
    // ...

    return c.JSON(fiber.Map{
            "message": "user created successfully",
            "data":    user,
        })
}

• That was pretty straightforward, wasn’t it?

Running the Server

• To run the server, we use the app's Listen function and specify the port number like this:

// main.go

func main() {
    //...

    log.Println("Starting server at: ", 3000)
    err = app.Listen(":3000")
    if err != nil {
        log.Fatalln("Error Starting Server at: ", 3000)
    }
}

• Now let’s run the app

Login

// controllers/auth_controller.go

func Login(c *fiber.Ctx) error {
    var loginReq types.LoginRequest

    if err := c.BodyParser(&loginReq); err != nil {
        return err
    }

    if err := validate.Struct(&loginReq); err != nil {
        return c.Status(fiber.StatusBadRequest).JSON(fiber.Map{
            "message": "Validation failed",
            "errors":  err.Error(),
        })
    }
}

• This part of the Login remains the same as the Register. We have created a LoginRequest structure inside types.go

// types/types.go

package types

...

type LoginRequest struct {
    Email 
    Password
}

• Now, we need to check if the user exists in the database when the client tries to log in with their credentials.

• To do this, we will use the Where() function of DB to see if the email is in the database. If it exists (we check this by seeing if user.Id is not 0), we proceed with further operations otherwise, we return an error.

func Login(c *fiber.Ctx) error {    
     // ...

    var user models.User

    database.DB.Where("email = ?", loginReq.Email).First(&user)

    if user.Id == 0 {
        c.Status(fiber.StatusNotFound)
        return c.JSON(fiber.Map{
            "message": "user not found",
        })
    }
}

• If the user exists, we compare the incoming password with the one in the database. If they match, the credentials are correct. If they don't match, we return an error saying "incorrect password."

func Login(c *fiber.Ctx) error {    
     // ...
     if err := bcrypt.CompareHashAndPassword(user.Password, []byte(loginReq.Password)); err != nil {
        c.Status(fiber.StatusBadRequest)
        return c.JSON(fiber.Map{
            "message": "incorrect password",
        })
    }
}

• bcrypt has a function called CompareHashAndPassword that checks the incoming password against the one stored in the database.

• After that, we just send a JSON response with a success message and the data.

func Login(c *fiber.Ctx) error {    
     // ...
    return c.JSON(fiber.Map{
        "message": "success",
        "data":    loginReq,
    })
}

• Let’s check the Login handler by running the project.

Logout

• The logout feature is very simple. On the backend, there's not much to do. The frontend should handle the logout process.

• So, when this route is accessed, we just return a success message.

func Logout(c *fiber.Ctx) error {

    return c.JSON(fiber.Map{
        "message": "Logged out successfully",
    })
}

• One problem right now here is, that even if you aren’t logged in, you can still log out. And this issue we will resolve below by using JWT.

JSON Tags

• If you see the response of Login and Register…

• We notice that the field names are not in lowercase. To convert these fields into a JSON-friendly format, we use what are called Tags in Go.

• Tags specify how struct fields should be encoded into or decoded from JSON. So, even if we have a struct field like Name, we can encode and decode it as name using an annotation.

• To achieve this, we need to make changes inside types.go and user.go.

// user.go

package models

type User struct {
    Id       uint   `json:"id"`
    Name     string `json:"name"`
    Email    string `json:"email" gorm:"unique"`
    Password []byte `json:"-"`
}

• Here, we made the email a unique key in the database. This means the same email address can't be used again.

• Another thing to note is that we added - in front of Password because we don't want this field to appear in the response for security reasons.

• Now, let's modify types.go.

package types

type RegisterRequest struct {
    Name     string `json:"name"`
    Email    string `json:"email"`
    Password string `json:"password"`
}

type LoginRequest struct {
    Email    string `json:"email"`
    Password string `json:"password"`
}

• Now if you run the project and see the response, it should be fixed

JWT Authentication

• Let’s first understand what JWT is before we dive into code.

What is JWT?

• JWT (JSON Web Token) is a way to safely send data between the client and server.

• It is a token made up of three main parts:

  • Header: This includes information like the signing method and type of token.

  • Payload: This has Claims, which are details about the user data, such as user ID and expiration time.

  • Signature: This is made by signing the header and payload with a secret or private key. It checks if the token is real.

Flow of JWT Authentication

• The client sends credentials to the server.

• If the credentials are valid, the server generates a JWT containing the user's information and signs it.

• The server sends the JWT to the client, typically as part of the HTTP response.

• The client stores the token (e.g., in local storage or an HTTP-only cookie).

• The client includes the JWT in the Authorization header (ex, Bearer <token>).

• The server validates the token’s signature and checks its claims (e.g., expiration time, roles).

• If valid, the server processes the request; otherwise, it rejects it.

Implement JWT Authentication

• To use JWT in our app, let’s first add the package

go get github.com/dgrijalva/jwt-go/v4

• In our app, when a user logs in successfully, we create a JWT token. After checking the password, we start by creating a claim to build the token.

if err := bcrypt.CompareHashAndPassword(...); err != nil {//...}

claims := jwt.NewWithClaims(jwt.SigningMethodHS256, jwt.StandardClaims{
        Issuer: strconv.Itoa(int(user.Id)),
        ExpiresAt: &jwt.Time{
            Time: time.Now().Add(time.Hour * 24),
        },
    })

• jwt provides a function called NewWithClaims to create a Claim. It takes two parameters:

  • Signing Method: We can choose from several secure hashing algorithms, but we are using HS256.

  • Claim: We include the claims we mentioned earlier, like user ID and expiry time, using the jwt.StandardClaims interface. We set the user ID in Issuer as a string and ExpiresAt with a time value, set to 24 hours. This means the token will expire after 24 hours.

  • Now that we have the claims ready, we will create a token.

token, err := claims.SignedString([]byte(os.Getenv("JWT_SECRET")))
    if err != nil {
        c.Status(fiber.StatusInternalServerError)
        return c.JSON(fiber.Map{
            "message": "could not log in",
        })
    }

• As you can see claims have a method named SignedString which we use to sign the JWT secret. We typically store the secret inside .env file.

// .env

// ... other keys
JWT_SECRET=secret

• Use a strong secret key for better security. I have used secret for testing purposes.

• And after everything’s done we get the token.

• If you remember this line:

“ A JWT token is simply a mix of signed claims and secret/private keys. I hope this explanation makes it clear.”

• And as mentioned in the Flow of JWT part we send this token to the client so that they can send it back to the backend to access secure/private routes

// auth_controller.go

func Login(c *fiber.Ctx) error {

// ...

return c.JSON(fiber.Map{
        "message": "success",
        "token":   token,
        "data":    loginReq,
    })

}

• Okay so, now that we created the JWT token. Whenever a user tries to access private routes, we need to first validate the user, whether he is logged in or not. To do that we create a middleware

Middlewares

• In Go, we create middleware that acts as a layer between the HTTP request and the handler that processes it.

• Middleware is simply a function that takes an HTTP handler and returns an HTTP handler. In our app, we need middleware to ensure the user accessing the route is authorized to access private routes before entering.

• To implement it, go to middleware/middlewares.go and paste the code below.

package middleware

import (
    "os"

    jwtware "github.com/gofiber/contrib/jwt"
    "github.com/gofiber/fiber/v2"
)

func Protected() fiber.Handler {
    return jwtware.New(jwtware.Config{
        SigningKey: jwtware.SigningKey{
            JWTAlg: "HS256",
            Key:    []byte(os.Getenv("JWT_SECRET")),
        },
        TokenLookup:  "header:Authorization",
        AuthScheme:   "Bearer",
        ErrorHandler: jwtError,
    })
}

func jwtError(c *fiber.Ctx, err error) error {
    return c.Status(fiber.StatusUnauthorized).JSON(fiber.Map{
        "message": "Unauthorized",
    })
}

• Starting with imports, we are using "github.com/gofiber/contrib/jwt" packages and gave aliases as jwtware.

• Here we have created a function Protected() that returns an HTTP handler. Inside this we are returning jwtware.New() handler which is used to create a new middleware instance for validating JWT

• Here we pass different keys:

  • SigningKey: This specifies the algorithm and key for verifying JWT’s signature.

  • TokenLookup: Defines where the middleware should look for JWT

  • AuthScheme: It defines which scheme to use in Authorization

  • ErrorHandler: It is a custom function to handle errors. So whenever an unauthorized user access any routes which require authorization, this throws an error message: “Unauthorized “

• In our case, we need two routes: Log in and Register as Public routes and Logout as Private route. How do we do that?

// routes.go

func SetupRoutes(app *fiber.App) {
    // Public routes
    app.Post("/api/register", controllers.Register)
    app.Post("/api/login", controllers.Login)

    app.Use(middleware.Protected())

    // Protected routes
    app.Post("/api/logout", controllers.Logout)
}

• As you can see, to use any middleware, fiber provides a very handy method called Use() . So what will happen now is whatever routes defined after app.Use(middleware.Protected()) this will require a JWT Token to get access.

• Let’s see it in action.

1. Open Postman

2. Try to logout directly

As you can see, Now it is throwing an error saying “Unauthorized“.

3. Now login with valid credentials

4. 

   Copy the token and paste it inside the Authorization Bearer field and then try to logout.

5. 

Code

Conclusion

• In this article, we've explored how to implement secure user authentication in a Go application using JWTs and the Fiber framework.

• We saw how to set up a project structure, connect to a MySQL database with GORM, and define essential routes and middleware.

• We've built handlers for managing user registration, login, and protected routes. This approach not only enhances the security of your application but also provides a scalable framework for future development.

• I hope you liked this article and learned something from it. If you did consider leaving a like and feedback in the comment section.

• See you in the next one, until then….

• Connect with me on LinkedIn, Github, and Twitter.

• Welcome to the last part of this Learn Go Programming series. In the previous parts, we learned lots of new stuff about Go and saw how easy and fast it is to build a backend. In the last blog, we created a Read function for getting all the books that are there in the database.

• In this part, we are going to implement the remaining operations which are:

  • Create Book

  • Update Book

  • Delete Book and

  • Get a Specific Book

• So let’s get started without wasting any more time.

Create a Book (POST)

Setting up the route

• Now to create a book, first, let’s configure the route for it.

• Go to routes.go and add a new handler called CreateBookHandler

// routes.go
func SetupRouter() *chi.Mux {
    ...
    r.Post("/book", books.CreateBookHandler) // Yet to be created inside handler.go
    ...
}

• As we need to create something inside the database, we need to use the POST method of HTTP.

Creating a Handler

• Head over to handler.go and paste the below code:

// handler.go

...
func CreateBookHandler(w http.ResponseWriter, r *http.Request) {
    var book Book
    err := json.NewDecoder(r.Body).Decode(&book)
    if err != nil {
        http.Error(w, "Failed to decode book", http.StatusInternalServerError)
        return
    }

    err = CreateBook(book) // Yet to be created inside service.go
    if err != nil {
        http.Error(w, "Failed to create book", http.StatusInternalServerError)
        return
    }

    w.WriteHeader(http.StatusCreated)
}
...

• So when we are creating something, we are most probably getting the data from the client side. In our case where we are creating a book, let’s say the admin wants to add a book to the website. What he will do is, add the details of the book in text fields on the front side and then press the Submit button.

• So when he does that, all the data that were filled by the admin comes to the backend inside a request Body in a JSON format

• So what we need to do here is, we just need to decode this data into our Go structure format.

• For that, we used this json.Decoder() function which takes json string and then Decode() takes a struct in which we need to fill the converted data.

• Then we need to call CreateBook() service of database which will add a new entry for the new book.

• Let’s write this function inside service.go

Creating a CreateBook Service

func CreateBook(book Book) error {
    _, err := database.DB.Exec("INSERT INTO books (title, author, year) VALUES (?, ?, ?)", book.Title, book.Author, book.Year)
    if err != nil {
        return err
    }
    return nil
}

• In service.go file we need to create a function called CreateBook. This function takes a struct of type Book which we will pass from the handler. And it will return an error.

• Inside the function, we are writing a create query for MySQL database. It’s self-explanatory so I am not going into details.

• In the end, we are returning an error if anything goes wrong.

• It was pretty simple, right?

• Let’s run the project and test it inside Postman.

• I already created two books for testing before so you are also seeing those in output.

Update a Book (PUT)

Setting up the Route

• Now that we have created a book. We can now create an Update functionality. Let’s go to the routes.go and create a PUT router

...
r.Put("/book/{id}", books.UpdateBookHandler)
...

• We need an id to update a particular book, right? To do that we can use a unique identifier which is id for our book. So When the user/admin updates a particular book client sends the id of it with the request to our backend.

• So if you see, after /book/ we have id inside curly braces. This is a dynamic value. When a client hits a URL with this path like, /book/1 or /book/20, we can get this book dynamic id and perform update operations.

• Let’s now create UpdateBook handler function

Creating a Handler

func UpdateBookHandler(w http.ResponseWriter, r *http.Request) {
    id, err := strconv.Atoi(r.PathValue("id"))
    if err != nil {
        http.Error(w, "Failed to convert id", http.StatusInternalServerError)
        return
    }

    var book Book
    err = json.NewDecoder(r.Body).Decode(&book)

    if err != nil {
        http.Error(w, "Failed to decode book", http.StatusInternalServerError)
        return
    }

    err = UpdateBook(id, book)
    if err != nil {
        http.Error(w, "Failed to update book", http.StatusInternalServerError)
        return
    }

    w.WriteHeader(http.StatusOK)
}

• Okay so in the first line, we are getting id using request objects PathValue function. Here we pass the exact value that we passed inside the route which is id.

• But we need to convert them id to int because our database accepts the ID of type integer.

• For that, we are using strconv package. And it has Atoi (ASCII to Integer) function which takes a string and converts it to an integer. It also returns err if it isn’t able to convert. So we are handling that error too.

• Then we decode the sent JSON data to our book struct.

• In the end, we call the UpdateBook a function where we pass the id and book struct.

• Let’s create a database service for Updating a book in the database.

Creating a UpdateBook Service

func UpdateBook(id int, book Book) error {
    _, err := database.DB.Exec("UPDATE books SET title = ?, author = ?, year = ? WHERE id = ?", book.Title, book.Author, book.Year, id)
    if err != nil {
        return err
    }
    return nil
}

• Here we are just executing an update query.

• Let’s run the project and see if it’s working.

• Okay, so I changed my mind and thought to give you some exercise.

• I want you to create DELETE a route for deleting a book. And GET route for getting “a single book“ from the database.

• I will share the code with you but before that, I recommend trying it out on your own.

• No cheating…

.

.

.

.

.

• You didn’t try, didn’t you…

• Anyway here is the code..

Deleting a Book

// routes.go
r.Delete("/book/{id}", books.DeleteBookHandler)

// handler.go
func DeleteBookHandler(w http.ResponseWriter, r *http.Request) {
    id, err := strconv.Atoi(r.PathValue("id"))
    if err != nil {
        http.Error(w, "Failed to convert id", http.StatusInternalServerError)
        return
    }

    err = DeleteBook(id)
    if err != nil {
        http.Error(w, "Failed to delete book", http.StatusInternalServerError)
        return
    }

    w.WriteHeader(http.StatusOK)
}

//service.go
func DeleteBook(id int) error {
    _, err := database.DB.Exec("DELETE FROM books WHERE id = ?", id)
    if err != nil {
        return err
    }
    return nil
}

Getting Book by id

// routes.go
...
r.Get("/book/{id}", books.GetBookHandler)
...

//handler.go
func GetBookHandler(w http.ResponseWriter, r *http.Request) {
    id, err := strconv.Atoi(r.PathValue("id"))
    if err != nil {
        http.Error(w, "Failed to convert id", http.StatusInternalServerError)
        return
    }

    book, err := GetBook(id)
    if err != nil {
        http.Error(w, "Failed to fetch book", http.StatusInternalServerError)
        return
    }

    w.Header().Set("Content-Type", "application/json")
    json.NewEncoder(w).Encode(book)
}

// service.go
func GetBook(id int) (Book, error) {
    var book Book
    err := database.DB.QueryRow("SELECT id, title, author, year FROM books WHERE id = ?", id).Scan(&book.ID, &book.Title, &book.Author, &book.Year)
    if err != nil {
        return book, err
    }
    return book, nil
}

Modifications

• We did finish our project but there are a few things that need to be fixed in this project. For example, When we are Updating a book, we must pass the whole object with every value instead of just a value that we need to update.

• There are no clear messages in the console after we perform operations.

• We should add proper logs inside each error-handling block.

• I want you to do a little bit of research on these points and update the code.

• It is pretty easy and I think with a little bit of research and work you will be able to complete it.

GitHub Repository

• Here is the Git repository that contains all the code.

Wrapping Up

• That’s it. We have officially completed this series. I hope you learned something from this series.

• If you have any queries or concerns regarding any concepts, please feel free to comment it. I’ll try to answer it as per my knowledge.

• In future blogs, I am planning to experiment more with Golang and create new projects with it. So stay tuned and follow to get the notifications.

• See you in the next one, until then…

• Heyy Gophers!!! Welcome to the Part 4 of Learning Go with Me. Thanks for coming alone to till here. I hope you are learning and enjoying this series.

• Till now we almost covered the fundamentals of Go and now we are ready to write out the first API.

• It’s gonna be exciting as we will learn a lot of new things which will help you in your first programming career

• Let’s get started without wasting any more time.

• We already have the project structure ready for us

crud-api/
│
├── cmd/
│   └── main/
│       └── main.go           # Entry point for the application
│
├── config/
│   └── config.go             # Environment variable management
│
├── internal/
│   ├── routes/
│   │   └── routes.go         # API routes definition
│   ├── books/
│   │   ├── handler.go        # HTTP handlers for book operations
│   │   ├── model.go          # Book database model and queries
│   │   └── service.go        # Business logic for books
│   ├── database/
│   │   └── db.go             # Database connection logic
│
├── .env                      # Environment variables
├── go.mod                    # Go module dependencies
└── .gitignore                # Ignored files

• Also we have dependencies installed in our project from Part 1 of this series

go get github.com/go-chi/chi/v5
go get github.com/go-sql-driver/mysql
go get github.com/joho/godotenvfg

• And lastly, we have the environment file ready for us

// .env
PORT=8080
DB_USER=root
DB_PASSWORD=password
DB_HOST=127.0.0.1:3306
DB_NAME=crud_api

• Now let’s go step by step. As we already have the .env file ready for us, why not create a function to load it inside our project?

• Before that let’s see the main.go file

// main.go

package main

func main() {
    // TODO: Load .env file

    // TODO: Setup database connection

    // TODO: Setup router

    // TODO: Run server
}

• These are our TODOs which we need to complete

Loading .env Variables

• To load values inside our .go files we are using godotenv package. This package helps us get environmental values inside our file.

• We implement this function inside config.go file. So head over to config.go and let’s write LoadEnv function. But before that let’s import a few packages

// config/config.go

package config

import (
    "log"
    "os"

    "github.com/joho/godotenv"
)

• We are using log, os and godotenv packages here.

fmt and log Packages

• In Go we mostly use these two packages to print out values and logs in the terminal. At first, these two may look similar but they both serve different purposes.

• log is specifically designed for logging messages. This means we can get more information like time stamping and error reporting with this. So we mostly use it during application runtime like debugging, informational logs, and error logs.

  • It includes functions like log.Fatal(Exists the program after logging) and log.Panic (Logs and then panic)

• Whereas fmt is used for general-purpose printing text. It is commonly used for printing in a console with formatted strings. So we use it mostly when we are printing user-facing messages.

Example

fmt.Println("Hello World") // Hello World
fmt.Printf("Hello %s", "World") // Hello World

log.Fatalf("This is a fatal error.") // This is a fatal error. (After this program stops)
log.Panic("This is a panic error.") // This is a panic error. (After this program stops)

• More on fmt visit: https://pkg.go.dev/fmt

• More on log visit: https://pkg.go.dev/log

os Package

• The os package in Go is used for tasks related to the operating system. For example accessing the files, and environment variable.

• In our example, we are focused on accessing the .env file and it has Getenv() a function that receives the values of environment variables named by key

• Now let’s write LoadEnv() function

import (...)

func LoadEnv() {
    err := godotenv.Load("../.env")
    if err != nil {
        log.Fatal("Error loading .env file")
    }
}

• We get Load() function inside godotenv package. Load will read the .env file and load them into ENV for this process. Notice that you pass the path of your .env file.

• This function returns an error so we are collecting it inside err variable and then checking if the err has anything. If the err is not nil which means something went wrong. If so, we need to show the error message and exit the program. Because if the env file is not loaded then there is no point in starting the program.

• Now to get a specific value from the env file we need to make one function that will get us the value of the asked key from the env file

package config

import (...)

func LoadEnv() {
    ...
}

func GetEnv(key, fallback string) string {
    if value, ok := os.LookupEnv(key); ok {
        return value
    }
    return fallback
}

• We created the GetEnv function, which takes key and fallback strings as parameters and returns a string.

• The key is the string for which we need the value in our code. For example, inside our .env file, we have different values. If we need only DB_USER, we pass this key to the function, and it returns the related value.

• We also accept a fallback string, so if the requested value isn’t found, we use the default fallback value.

• Here, we are using the os package, which includes the LookupEnv function. This function takes a key as an argument and returns two things: a string and a bool. If the value is found, ok will be true; otherwise, it will be false. By checking ok, we can determine if a value was returned and decide whether to return the fallback value.

• Let’s check if it is working or not

package main

import (
    "fmt"

    "github.com/red-star25/crud-api/config"
)

func main() {
    config.LoadEnv()
    fmt.Println(config.GetEnvValue("DB_USER", "root"))

       // TODO: Setup database connection

    // TODO: Setup router

    // TODO: Run server
}

Output:

root

• That’s awesome. Let’s now set out the Database Connection

Setting Up the Database Connection

Importing Packages

• As you know we are using MySQL for storing our application’s data. We need to first set up the connection and create an instance of the SQL DB.

• Head over to db.go file and let’s first import all the required packages

package database

import (
    "database/sql"
    "fmt"
    "log"
    "os"

    _ "github.com/go-sql-driver/mysql"
)

• We have seen the usage of fmt, log and os. Let’s see what are the remaining ones.

• database/sql: This is a built-in library for SQL database interactions.

• _ "github.com/go-sql-driver/mysql": Imports the MySQL driver as a side effect to register it with database/sql. Here _ (underscore) represent that this import is used as a side effect. Which means this package is not directly used.

Creating Database

var DB *sql.DB

• Here type DB is a pointer to sql.DB an object. We are making it globally accessible so that anyone can reuse it without creating it again and again on each query

• You must be wondering Why it is a type of pointer.

  • So, If DB were a simple variable (not a pointer), any changes made to it inside a function, wouldn't affect the global variable itself. Instead, Go would create a copy of the value, and the global DB would remain unchanged.

  • By using a pointer, it updates the global DB directly. This makes the database connection immediately available to all parts of the application.

Connect function

• Now let’s create a function Connect that will be responsible for opening the database connection and making a connection.

func Connect() {
    dsn := fmt.Sprintf("%s:%s@tcp(%s)/%s",
        config.GetEnvValue("DB_USER", "root"),
        config.GetEnvValue("DB_PASSWORD", ""),
        config.GetEnvValue("DB_HOST", "localhost"),
        config.GetEnvValue("DB_NAME", "crud_api"),
    )
}

• Here we have created a variable dsn(Data Source Name) which is a type of string. So before we connect to MySQL we need to have a URL to which we want to connect.

• So here we have used a formatted string from fmt package. Here function Sprintf returns a formatted string without printing it to the console. %s is a placeholder for string values here.

• And we are getting all the values from our .env file using config’s GetEnvValue function.

• Example dsn

username:password@tcp(localhost)/mydatabase

Opening Connection

• Now that our data source URL is ready, we are ready to open our SQL connection.

• To do that mysql provides Open() function which takes

  • driverName (which is nothing but a database name, ex: MySQL, MongoDB, Postgres, etc). You can provide any driver here and open the connection and

  • dataSourceName is a URL of the database that we just created above.

var err error
DB, err = sql.Open("mysql", dsn)
if err != nil {
    log.Fatalf("Error connecting to database: %v", err)
}

• This function returns two things: db object and err.

• So what we are doing here is, we are assigning the returned SQL db object to our globally defined DB.

• And then we check if there is any error occurred or not.

You will see this type of error handling thoughtout any Go project. This is a convention in Go. It might seems little overdoing but it is a good practice that we are handling error just after calling function.

Also note that there is no try catch or finally keywords in Go. You can read why they are using this type of error handling in this article : https://go.dev/doc/faq#exceptions

Verifying the DB Connection

• To verify whether the connection is successful or not we have Ping function that verifies if the database connection is still alive, and it also establishes the connection if necessary.

if err := DB.Ping(); err != nil {
    log.Fatalf("Error verifying database connection: %v", err)
}

• That’s pretty much it. We can log the message after this that the Database is connected successfully.

Final db.go code

package database

import (
    "database/sql"
    "fmt"
    "log"

    _ "github.com/go-sql-driver/mysql"
    "github.com/red-star25/crud-api/config"
)

var DB *sql.DB

func Connect() {
    dsn := fmt.Sprintf("%s:%s@tcp(%s)/%s",
        config.GetEnvValue("DB_USER", "root"),
        config.GetEnvValue("DB_PASSWORD", ""),
        config.GetEnvValue("DB_HOST", "localhost"),
        config.GetEnvValue("DB_NAME", "crud_api"),
    )

    var err error
    DB, err = sql.Open("mysql", dsn)
    if err != nil {
        log.Fatalf("Error connecting to database: %v", err)
    }

    if err := DB.Ping(); err != nil {
        log.Fatalf("Error verifying database connection: %v", err)
    }

    log.Println("Database connected successfully!")

}

• And now call this Connect function inside main.go

package main

import (
    "github.com/red-star25/crud-api/config"
    "github.com/red-star25/crud-api/database"
)

func main() {
    config.LoadEnv()

    database.Connect()

    // TODO: Setup router

    // TODO: Run server

}

• If everything goes right then you will see this message after you run the program.

  

  Make sure you have MySQL up and running on your computer.

Creating Book model

• We are creating CRUD API for managing Books. So in order to view data in a structured manner with all the required fields we need to create a Book model using struct

In Go, the term "model" refers to a structured representation of data used within an application, typically to map data between a program and an external data source, such as a database or an API. Models are often implemented as structs in Go, which are a way to define and organize related data fields.

• Let’s create our Book model

// internal/books/model.go
package books

type Book struct {
    ID     int
    Title  string
    Author string
    Year   int
}

• Here we have four fields inside our Book struct. ID, Title, Author, Year.

• That’s easy, right? Yes. But there is one more thing which we need to do. So as we know that when we deal with APIs, we use JSON to serialize and deserialize the data that we are getting from the user and data we are sending to the user.

• And generally, they are represented in lowercase like id, name, author, year like that. But if you see our Book struct we have the first letter capital for each field. And that is intentional. Because we need those struct’s fields outside to use. And to make it globally accessible we capitalize the first letter of any variable.

• So to resolve this issue, we use what we call in go “ struct tags “. In other programming languages, we refer to this as “ annotations “.

• This helps us convert these struct fields to proper naming conventions. Let’s see how we do that

package books

type Book struct {
    ID     int    `json:"id"`
    Title  string `json:"title"`
    Author string `json:"author"`
    Year   int    `json:"year"`
}

• So what we do is after giving the type of the struct field we use backticks ` `. And in there we write json: and then whatever name we want.

Implement Database Queries (CRUD)

• Now that our database is up and running let’s write database queries for Create, Read, Update, and Delete.

• First of all, let’s import two required packages

package books

import (
    "crud-api/internal/database"
    "errors"
)

• We are importing a database for accessing the DB object. Using this we can perform all the queries.

• And errors we use for handling errors.

The GetAllBooks Function

• Firstly we are creating a function to get all the books from the database

func GetAllBooks() ([]Book, error) {
    rows, err := database.DB.Query("SELECT * FROM books")
    if err != nil {
        return nil, err
    }
    defer rows.Close()

    ...
}

• Here we created a function called GetAllBooks which is responsible for getting all the books from the database.

• It returns two values:

  • [ ]Book: A slice of struct Book.

  • error: Go’s error object.

• Inside the function body, we are making a query to the SQL database using the DB variable’s Query function. It takes a string where we write a raw SQL query.

• In this function, we need all the books with data. And to do that we wrote this query “SELECT * FROM books“. Which will get all the books from the database.

• And then we checked if it threw any errors

• After that, we defer call the rows.Close() to close it once this function is finished.

• This rows object allows us to iterate over the results we get from the Query. And to iterate over the results we have rows.Next() function which goes over all the results.

• We can take advantage of this function and create a for loop inside which we can fill all the data inside a slice of the book

func GetAllBooks() ([]Book, error) {    
    ...
    var books []Book
    for rows.Next() {
        var book Book
        if err := rows.Scan(&book.ID, &book.Title, &book.Author, &book.Year); err != nil {
            return nil, err
        }
        books = append(books, book)
    }
    return books, nil
}

• Here we have created a temporary slice of the book variable. In this, we will fill in all the data.

• Then we looped through rows.Next() which will go over all the rows and inside the for the body we are getting all mapping the data in a row to the Book struct using rows.Scan

• And after that, we are appending that book to the books slice

• And once every row is scanned we return it from the function.