But if you see, till now, we were dealing with a single host and a few containers. But in real life, that is not the case. In companies, they have so many products and services, and those services do not run on a single host. They are scattered around the world or in other places. Then you might ask, who manages those services? Is it like someone sitting in front of the display and checking all the time which services are working correctly or not?

If you think that is how they manage, you are mistaken, because managing so many services manually is not possible today. So what's the solution? Who manages it?

To answer that, you have to read this blog. But if you don't want to and just want to know, then here is the short answer:

Docker Swarm / Kubernetes

I bet you want more details than just these two words, right? And to get that, you'll have to read the blog, so, sorry, 😅 I can't give you the short answer. Because, like life, there are no shortcuts to achieving goals. Okay, enough of motivation, let's get to the real thing.

Docker Swarm

We now know how to install Docker, pull images, and work with containers. But as we discussed above, to work with that big scale, we need something that can manage these containers for us. And that's where Docker Swarm comes into the picture.

What is Docker Swarm?

Docker Swarm is a tool that helps you manage multiple Docker machines. Instead of running containers on just one computer, you can connect multiple machines and treat them like a big system. This group of machines is called a cluster.

In a Swarm cluster, each machine is called a node. A node can be a physical server, a virtual machine, a cloud server, or even a small device like a Raspberry Pi. The only thing required is that Docker must be installed, and the machines must be able to talk to each other over the network.

Types of Node

There are two types of nodes: managers and workers.

Managers control the cluster. They decide what tasks should run and which worker should run them.

Workers are the machines that actually run the containers and do the work. Managers tell them what to do, and workers follow those instructions.

Docker Swarm also keeps track of the cluster’s information, like which containers are running and which nodes are available. This information is stored on the manager nodes in a distributed database. The nice thing is that Docker sets this up automatically, so you don’t need to configure it yourself.

If you are concerned about Docker's security, then don't. Because security is built into Swarm. Communication between nodes is encrypted, and each node must prove its identity before joining the cluster. Docker also automatically manages and rotates security certificates, so the system stays secure without extra effort.

Swarm is also used to run and manage applications made of many small services called microservices. In Swarm, the main unit used to run applications is called a service. A service is similar to a container but with extra features. For example, you can easily run multiple copies of it, update it gradually without downtime, or roll back to an older version if something breaks.

There is also one tool called Kubernetes which is a competitor of Docker Swarm. They both orchestrate containerized applications. Well, it's true that Kubernetes is more popular and have more active community and ecosystem. Docker Swarm has it's own benefits too. It's a lot easier to configure and deploy. While it may not be the best solution for bigger companies, but its an excellent technology for small to medium businesses.

I think that's enough of the theory. Let's get to the practical part and see how we can implement this ourselves.

Creating our OWN Swarm Cluster

The example is highly inspired by the book (Docker Deep Dive - Nigel Poulton). The only thing missing there was that he didn't show how to implement it with a single host machine. Because hey, for just the example, we're not gonna have 10 different machines and run and test it, right? So what I did was, I implemented the same thing with the help of AWS. Let's see how we can implement it. The only prerequisite is to have an AWS account. So if you don't have one, I would suggest to make one. I am not gonna show it here otherwise the blog will become lengthy. So make one and then start from here.

Example:

So, in the example, we are going to make a secure swarm cluster with one manager nodes and two worker nodes.

As we saw earlier, that nodes can be virtual machines, physical servers, cloud instances (in our case), or Raspberry Pi systems. The only thing required on those system is to have Docker installed.

Creating 3 EC2 instances ( 1 manager, 2 worker)

Go to https://signin.aws.amazon.com/signin and sign in with the Root user. Once you are in, you will see a dashboard like this.

Now we need 3 machines, right? So let's create 3 EC2 instances. Go or search for EC2 and get in.

Now, click on Launch instance.

Creating Manager Node

Now, we need to name our machine and configure some options.

If you don't know anything about AWS EC2, I think it's the best time to learn. Because in today's time, it good to have some knowledge about it at-least if not a hands on experience. But anyways, you will not be needing to know much about it for this example as it's gonna be really simple.

Here, we are doing nothing but creating a Virtual instances (machines) for us to be able to test Docker Swarm capabilities.

Let's now finish the configuration part. Go ahead and name this instance "mgr1" (this will be our manager). Then select the Ubuntu image

After that, we need to add a key pair. This is required as we are gonna connect this instance to our laptop's terminal and use it. So go ahead and create an RSA (.pem) private key and store it somewhere safe. I named this key docker-swarm-test.

Once this is done, do nothing, just click on Launch Instance. As you can see below, we now have the running manager instance.

Now, we don't need to open these instances to the whole world, right? So what we do is we open a few ports between these instances for them to talk. We can add those ports by going into the Security section. Click on the Security Group link, it will redirect you to the new page where we can add the ports

Click the "Edit Inbound rules".

Add all the ports defined below. I've added the description for each port, what it is used for, so you can take a look at it too. Once done, save the rules, and you are good to go.

Creating Worker Nodes

Now we have one Manager node. We need two worker nodes. The process gonna be the same only except the name of the Instance. Go ahead and create wrk1 and wrk2. I am gonna show wrk1, and then you can create wrk2 on your own. It's pretty simple.

1. Name the instance (wrk1) and select the Ubuntu image.

Select the same key pair that we created.

Now here comes the most IMPORTANT part, so make sure you don't miss it. We need to put all these instances into the SAME VPC and Subnet in order for them to work. How will you make sure? Just open the mgr1 instance (on the right in the screenshot below), go to the Networking tag, and you will see the VPC ID and Subnet ID.

On the worker node (on the left), select the same Subnet ID and VPC ID.

Once done, launch that instance.

IMPORTANT: Make sure you add all the ports like we did for the Manager node.

As you can see now, I have all three instances running on my AWS.

SSH into each instance

Now, let's open these instances in our machine. Open three separate terminals and run below commands

ssh -i ~/Downloads/docker-swarm-test.pem ubuntu@107.21.169.169

Here, make sure you provide the correct path of the key you downloaded (this is the key which gets downloaded when we created the key-pair). And then specify the Public IP Address. Which you will find in the Details tab of the instance

Once done, your terminal will turn into that instance terminal, and you'll see something like this.

Do this for all three instances, and you will have all three instances running inside your host machine in your terminal.

(Top - Manager, Bottom Left - Worker 1, Bottom Right - Worker 2)

Installing Docker on all 3 instances

In order to run Docker Swarm, we first need to install Docker inside these instances. To do that run below commands

sudo apt-get update
sudo apt-get install -y docker.io
sudo systemctl enable --now docker

Run those commands in all three terminals, and you will have Docker installed.

Initializing a new swarm

Now that we have everything in place, we can go ahead and initialize our Docker swarm. One thing to note about is that, Docker nodes that are not part of swarm are said to be in single-engine mode. Once they're added to a swarm they're automatically switched to swarm mode.

Running docker swarm init on a Docker host in single-engine mode will switch that node into swarm mode, create a new swarm, and make the node the first manager of the swarm.

And all the additional nodes can then be joined to the swarm as workers and managers. Joining a Docker host to an exisiting swarm switches them into swarm mode as part of the operation.

Let's run the below command to our manager node to initialize a new swarm.

docker swarm init --advertise-addr <manager-ip-address>

Let's understand the command:

docker swarm init - This tells Docker to initialize a new swarm and make this node the first manager. It also enables swarm mode on the node.

--advertise-addr - As the name suggests, this is the swarm API endpoint that will be advertised to other nodes in the swarm. It will be one of the node's IP address, but can be an external load balancer address.

--listen-addr - This is the IP address that the node will accept swarm traffic on. If not explicitly set, it defaults to the same value as advertise-addr.

To list the nodes in the swarm run sudo docker node ls command.

As you can see, only the mgr1 node is the only node present in the swarm. It's also listed as Leader. Now it's time to add the worker nodes to join this swarm too. To do that run below command in manager node

sudo docker swarm join-token worker

If you want another manager to join the swarm, run:

sudo docker swarm join-token manager

Now go to the worker nodes (wrk1 and wrk2) and join the swarm using the command you got above:

That's it. We've just created 3 node swarm with 1 manager and 2 workers. And you can see that by running sudo docker node ls command into the manager node. Nodes with nothing in the Manager Status are workers.

Also if you notice the asterisk (*) in front of the node ID, it says which node you are logged on to and executing commands from.

Swarm manager high availability

You must be asking, why we added managers and workers, and how they work together, and what's the point of all these?

So in real worlds, it can happen that sometimes one or more nodes can fail, but if you are using Swarm, the survivors will keep the swarm running without interrupting anything.

Right now, we have only one manager, but if you have more than one managers, only one of them is active at any given moment. This active manager is called the "Leader". And the leader is the only one that will ever issue live commands against the swarm.

(Diagram from - Docker Deep Dive book by Nigel Poulton)

Swarm Services

A service in Docker Swarm is a way to run and manage containers across a cluster of machines. It is a feature that only works when Docker is running in swarm mode. A service lets you define how your application should run, and the swarm makes sure that state is always maintained

With services, you can set many of the same options you normally use for containers. For example, you can give the service a name, choose which image to use, map ports, and connect it to networks. But services also add extra features that are useful for modern cloud applications.

You can create services by running docker service create command

docker service create --name web -p 8080:8080 --replica 5 dhruvnakum/web

For example, a command above create a service named web, map port 8080, and run 5 replicas using a specific image. After the command runs, the manager node decides where those containers should run in the cluster. Each node that receives a task downloads the image and starts a container.

Swarm also constantly watches all services. It runs a background process that checks if the actual state matches the desired state. If everything matches, nothing changes. But if something goes wrong, the swarm fixes it automatically.

For example, if one of the machines running a container fails, the number of running containers might drop from 5 to 4. Swarm will notice this and automatically start another container on a different node to bring the total back to 5. This is called self-healing, and it helps keep applications running even when failures happen.

You can see all services in the swarm using the command

docker service ls

If you want to see the containers that belong to a specific service, you can use

docker service ps

Another useful feature is scaling. If your application gets more traffic, you can increase the number of running containers. For example, if traffic doubles, you can scale the service from 5 containers to 10 using the command docker service scale

Example:

docker service scale web=10

web scaled to 10
overall progress: 10 out of 10 tasks
1/10: running
2/10: running
3/10: running
4/10: running
5/10: running
6/10: running
7/10: running
8/10: running
9/10: running
10/10: running
verify: Service converged

When scaling happens, Docker tries to spread the containers evenly across the nodes in the cluster. This helps distribute the workload.

If traffic goes down again, you can scale the service back to a smaller number of containers. For example, you can reduce it from 10 back to 5.

docker service scale web=5

web scaled to 5
overall progress: 5 out of 5 tasks
1/5: running
2/5: running
3/5: running
4/5: running
5/5: running
verify: Service converged

If you no longer need the service, you can remove it using the command docker service rm. When you do this, the service and all of its running containers will be deleted from the swarm.

docker service rm web

Conclusion

Phew!! That was a lot, right? But I hope the practical implementation, with screenshots and everything, helped you get the idea of how swarm works.

Docker Swarm is Docker's native technology for managing clusters of Docker nodes and deploying and managing cloud-native apps.

It's similar to Kubernetes. So if you understood this right now. Understanding Kubernetes will become easier for you.

That's it. I hope you learned something from this. Just want to say this, understanding this only won't make you understand everything until you try it on your own. So make sure you practise this.

See you in the next one, until then....peace

https://giphy.com/gifs/fallontonight-jimmy-fallon-peace-out-tonightshow-2nlbKhgnvAK3sR8ffw?utm_source=iframe&utm_medium=embed&utm_campaign=Embeds&utm_term=https%3A%2F%2Fdhruvnakum.xyz%2F

When people hear Networking, it often sounds complex and scary. But the core ideas are actually simple once you break them down. Let's tackle down and understand the Docker Networking. I will try my best to simplify it, so let's get started.

Docker networking is divided into 3 main parts

1. CNM (Container Network Model) – the design or the plan, you can say

2. libnetwork – the implementation or the code that follows the plan

3. Drivers – the actual network types (bridge, overlay, macvlan, etc.)

A simple way to remember this is:

• CNM is like the blueprint of a house

• libnetwork is like the construction team that follows the blueprint

• Drivers are like the different types of houses you can build (apartment, villa, etc.)

Let’s go step by step and understand these in detail, and make it feel clear.

Part 1: CNM (Container Network Model)

CNM is not software you install. It’s a design document. Docker created CNM to define how container networking should be structured.

CNM says Docker networking is built using 3 building blocks

• Sandbox

• Endpoint

• Network

If you understand these three, you understand the base of Docker networking.

1) Sandbox

A sandbox is an isolated networking environment for a container. That sentence sounds heavy, but here’s what it really means:

Every container gets its own private networking setup, including:

• its own IP address

• its own network interfaces like eth0 inside the container

• its own routing table and rules about where traffic should go

• its own DNS settings

• its own ports

So the sandbox is basically, everything networking related that belongs only to that container.

Think of it like this

• Each container lives in its own room

• That room has its own Wi‑Fi setup, its own address, and its own rules

• Even if two containers are on the same machine, their “rooms” are still separate

On Linux, Docker achieves this separation using something called network namespaces. You don’t have to memorize that right now, just remember the outcome which is, containers behave like separate computers from a networking point of view.

The important thing to remember here is, for example, even when Container A and Container B run on the same Docker host, their networking is still isolated because each has its own sandbox.

2) Endpoint

Your laptop can connect to Wi‑Fi because it has a network card. Containers don’t have physical network cards. So Docker creates a virtual network interface for them. CNM calls this an endpoint.

An endpoint’s job is simple, which is to connect a container’s sandbox to a network.

There are two rules that make endpoints easy to understand

1. One endpoint connects to exactly one network.

   
   1. If a container needs to join two networks, it needs two endpoints.

For Example, If a container must talk to both Network A and Network B, Docker gives it

• Endpoint 1 → Network A

• Endpoint 2 → Network B

That’s how one container can be part of multiple networks at the same time.

3) Network

In Docker, a network is like a virtual switch. It groups endpoints. If two containers connect to the same Docker network, they can usually talk to each other.

If they are not on the same network, they can’t talk to each other by default (unless you set up routing manually).

A very simple analogy is that a Docker network is like a WhatsApp group. People inside the same group can message each other. If you’re not in the group, you can’t message the group members.

Same idea with containers: same network = communication is possible.

Part 2: libnetwork

So far, CNM is just a design. It describes what should exist: sandboxes, endpoints, networks. But Docker needs real code that actually creates and manages these things.

That code is called libnetwork.

So, CNM says what should exist, libnetwork actually creates it and manages it.

Whenever you run commands like:

• docker network create

• docker run

• docker network connect

Docker Engine uses libnetwork behind the scenes. You can think of libnetwork as Docker’s network manager.

Why Docker separated networking from the daemon?

Early on, a lot of Docker networking code lived inside the Docker daemon dockerd. But over time networking became bigger and more complicated:

• multiple network types (bridge, overlay, macvlan…)

• DNS and service discovery features

• load balancing

• multi-host networking

The daemon started becoming too large and messy. That’s why Docker engineers moved networking into a separate library called libnetwork.

This was good because networking could improve without constantly modifying the main daemon, and the design became more modular (cleaner separation), as well as other projects could reuse the networking library

So when someone says networking was ripped out of the daemon and refactored into libnetwork, they mean it was separated into its own component.

Control plane vs Data plane

These two terms show up a lot in networking. Here’s what it means.

Control plane (libnetwork)

The control plane is responsible for deciding how the network should behave and setting up the rules. It does not move packets itself. Instead, it programs the system so packets know where to go. In Docker networking, the control plane is mainly handled by libnetwork.

For Example, when you run docker network create mynet: The control plane performs multiple tasks.

1. Create a Network Object

Docker stores metadata like:

Network Name: mynet
Driver: bridge
Subnet: 172.18.0.0/16
Gateway: 172.18.0.1

This is stored in Docker's internal database.

2. IP Address Management (IPAM)

Docker assigns a subnet to the network.

Example:

Subnet: 172.18.0.0/16
Gateway: 172.18.0.1

Later, when containers join the network:

Container A → 172.18.0.2
Container B → 172.18.0.3

IP allocation is controlled by the IPAM system in the control plane.

3. Select Network Driver

Docker supports different drivers:

• bridge

• overlay

• macvlan

• host

• none

Example:

Driver = bridge

The control plane decides which driver will implement the network.

4. Maintain Network State

Control plane tracks:

• which containers belong to the network

• assigned IPs

• driver configuration

• DNS entries

Example state:

Network: mynet
Containers:
   web -> 172.18.0.2
   db  -> 172.18.0.3

5. Configure System Networking

The control plane tells Linux:

• create bridges

• configure routes

• configure iptables rules

But it doesn't move packets itself. Instead, it programs the data plane.

Data plane (driver)

Data plane means where the real packets move.

Imagine two containers:

Container A
172.18.0.2

Container B
172.18.0.3

They are connected to:

docker0 bridge

now the network structure is like this

Container A
   |
 vethA
   |
 docker0 bridge
   |
 vethB
   |
Container B

Now when A sends a packet to B:

ping 172.18.0.3

The data plane handles the packet movement.

Implementation

Let’s walk through a simple example:

Step 1: Create a bridge network

You run:

docker network create -d bridge mynet

Docker Engine receives the command. It asks libnetwork to create a new network object. libnetwork stores, name=mynet, driver=bridge, settings, IP ranges, etc. And then it calls the bridge driver and says build the real network for this.

The bridge driver then sets up Linux-level things like:

• a Linux bridge device

• NAT / iptables rules

• isolation rules

Step 2: Run a container on that network

You run:

docker run --network mynet nginx

In this case, Docker asks libnetwork to attach this container to “mynet”. So what libnetwork will do is create, a sandbox which is a container’s private network environment and an endpoint.

Then libnetwork connects the endpoint to the network and assigns IP + DNS settings. And finally asks the bridge driver that “connect this endpoint to the bridge”

So a very clean way to remember:

• libnetwork organizes and manages

• the driver does the real network wiring

Single-host Bridge Networks

The simplest Docker network type is the single-host bridge network. The name itself tells you the meaning of it. Single-host works only on one Docker machine. Bridge behaves like a Layer 2 switch.

Every Docker host gets a default bridge network. On Linux it’s called bridge. On Windows it’s called nat.

If you don’t specify a network, containers join this default one

On Linux, this default bridge network is backed by a real Linux bridge called docker0.

Creating your own bridge network

Run the command below to create your own bridge network.

docker network create -d bridge localnet

Then run a container attached to it. For example we have two container c1 and c2.

• run c1 on localnet

docker container run -d --name c1 \
--network localnet \
alpine sleep 1d

• run c2 on localnet

docker container run -d --name c2 \
--network localnet \
ubuntu:latest

Now c2 can reach c1, and very importantly, it can resolve it by name, not just IP.

That leads to an important concept: Service discovery.

Service discovery

Container IP addresses are often not stable. For example, containers restart and may get a new IP, or scaling up/down creates new containers with new IPs.

So you don’t want your app to hardcode IP addresses.

Instead, you want to say: “talk to db”, “talk to api”, “talk to redis”

That is service discovery, automatic mapping from name → IP.

Docker supports this on the same network using an internal DNS system:

Docker runs an internal DNS server that knows container names and IPs, and containers have a local DNS resolver that forwards requests to Docker’s DNS

So if you run ping c1 from inside c2, Docker DNS helps c2 translate “c1” into the correct IP.

This name-based discovery works only within the same Docker network. If two containers aren’t on the same network, Docker DNS won’t resolve them by name for each other.

Port mapping

Containers on a bridge network are mainly meant to talk to each other inside the same host/network. But what if you want a browser, something outside Docker, to reach a container web server? That’s where port mapping comes in.

For example, you have seen this --publish 5000:80 In a command, we write when we run the container. What does that mean?

It simply means that container listens on port 80 and Docker host opens port 5000

your machine : docker container (5000:80)

So the outside world talks to the Docker host, and Docker forwards the traffic to the container. This is extremely common for local development.

Multi-host networking

Bridge networks are great when everything is on one machine. But real systems often run across multiple machines. For example, Host 1 runs some containers. Host 2 runs other containers, and you want containers across hosts to talk like they are on one shared network

By default, containers on different hosts are like they’re in different houses, they’re not on the same local network.

An overlay network solves this by creating a virtual network that spans multiple Docker hosts.

Overlay network

An overlay network is a virtual network built on top of the real network.

For example, you can consider like this:

• real network = normal roads

• overlay network = secret tunnels built on top of those roads

Containers use the tunnels so that, even across different machines, they behave like they’re on the same LAN.

How overlay “tunneling” works

When Container A on Host 1 sends data to Container B on Host 2:

• A sends a normal packet, it thinks B is nearby.

• Docker on Host 1 wraps that packet inside another packet like putting a letter in an envelope

• the outer packet travels over the real network to Host 2

• Docker on Host 2 unwraps it and delivers the original packet to B

Docker has a built-in overlay driver, so you can create an overlay network like:

docker network create -d overlay mynet

Usually this is used with Swarm or other orchestrators

MACVLAN: making containers look like real devices on your LAN

Overlay is about connecting containers across hosts inside a Docker-managed virtual network. Macvlan is a different goal.

Macvlan is used when you want containers to join the real physical network directly.

In other words, instead of hiding containers behind the host like bridge / NAT often does, macvlan makes each container appear like a real machine on your network.

With macvlan, each container gets its own MAC address, its own IP address from your real LAN subnet

So your switch/router sees the container as it sees: your laptop, a server, a VM, and now… the container

macvlan often requires the host network card to be in promiscuous mode. Promiscuous mode is sometimes blocked in corporate networks or cloud providers

So, macvlan is commonly a good fit in controlled environments like data centers.

Overlay vs Macvlan

Overlay:

• Best for container-to-container communication across multiple Docker hosts.

• Containers feel like they are on one shared Docker network.

• Common in clusters, Swarm or microservices

Macvlan:

• Best when containers must be first-class citizens on the physical network

• Containers get real LAN IPs and MACs

• Common when containers must talk to physical servers directly and be reachable from outside without port mapping

Swarm publishing: Ingress mode vs Host mode

In Swarm, you usually deploy services and not individual containers. A service can have replicas running on different nodes. And to allow outside users to access a service, you publish a port.

For example,

• published port (outside) = 5000

• target port (inside container) = 80

Swarm supports two publishing modes:

1) Ingress mode (default)

• Any node can accept traffic, even if it’s not running the container. And the cluster routes it internally to the right node.

• Ingress mode is the default. This means any time you publish a service with -p or --publish it will default to ingress mode.

• To publish a service in host mode you need to use the long format of the --publish flag and add mode=host. Let’s see an example using host mode.

docker service create -d --name c1 \
--publish published=5000,target=80,mode=host \
nginx

2) Host mode

• In this case, only nodes that are actually running the service accept traffic on that port.

In practice, people often talk about routing mesh and load balancing here. If multiple replicas exist, Swarm can spread incoming requests across them.

Conclusion

• So in summary, Docker networking boils down to three things.

  • CNM defines the model - sandbox, endpoint, network.

  • libnetwork is the manager that creates and connects those pieces, and drivers - bridge/overlay/macvlan, do the real packet-moving work.

  • Bridge is for containers on one host, overlay connects containers across hosts like one network, and macvlan puts containers directly onto the physical LAN with their own MAC/IP.

We saw how we can containerize our application and push it to the Docker Hub registry for anyone to use.

In real world, applications are not that simple, right? There are loads of things like, web server, database, cache, some background worker, etc. And things get complicated quickly. If we try to manage this using docker run command and custom scripts then things become messy and hard to maintain.

This is exactly where Docker Compose helps. Docker Compose lets you describe your entire multi-container application in one simple configuration file, and then deploy everything with a single command. Instead of remembering a dozen commands, you maintain one declarative file. Once the app is running, Compose also gives you a small set of commands to manage the whole app’s lifecycle, like start, stop, restart, view status, and tear it down cleanly. Because the configuration is just a text file, you can also store it in Git and treat it like normal code.

Docker Compose works across platforms, and if you use Docker Desktop on Mac, Compose is typically installed automatically. You can quickly verify it by checking the version with docker-compose --version.

Understanding Compose Files (YAML)

Docker Compose uses a YAML file to define a multi-service application. YAML is popular because it’s human-readable and clean for configuration. Technically, YAML is a superset-like format that can represent the same structures as JSON, and Compose also supports JSON, but YAML is the standard and most common format you’ll see in real projects.

By default, Compose looks for a file named docker-compose.yml. If you want to use a different name, for example, a production-specific file, you can pass the filename using the -f flag when you run Compose.

To make this more understandable, imagine a small app built with three services:

• backend

• frontend

• db

Compose lets you define all the services together so they can be started and managed as one unit.

What’s Inside a docker-compose.yml

Just look at the below docker-compose file.

version: "3.8"
services:
  web-fe:
   ...
  redis:
   ...
networks:
  counter-net:

volumes:
  counter-volume:

A Compose file is structured into top-level sections (keys). The file includes mostly these four important top-level keys:

1. version

2. services

3. networks

4. volumes

Other top-level keys also exist (like secrets and configs), but these four are the core ones you’ll see often.

1) version: Compose file format version

The version key is required in this style of Compose file and appears at the top. It defines the version of the Compose file format, you can think of it like the schema/API version for the YAML format. A common mistake is assuming this refers to your Docker Engine version or the Docker Compose binary version it does not. It only describes the format rules for the YAML file itself.

In your example, the Compose file uses version 3.8, which is a widely used format.

2) services: The containers that make up your app

The services section is the heart of Compose. This is where you define each microservice that will run as a container. In the example, there are two services: web-fe and redis.

A really important detail is that Compose uses these service names when it generates container names. So if you define services named web-fe and redis, the containers created will include those names (often combined with the project name, like counter-app_web-fe_1). This naming becomes useful when you’re troubleshooting, checking logs, or inspecting containers later.

3) networks: How containers talk to each other

The networks section tells Docker what networks to create. By default, Compose typically creates bridge networks, which are single-host networks. That means the network connects containers running on the same Docker host. If you need more advanced networking, Compose can reference other drivers, but the default bridge model is perfect for local development and many single-host deployments.

We have the whole dedicated Docker Networking blog upcoming, so stay tuned

In the example, a network named counter-net is defined. Both the service attach to this network, which means they can talk to each other using service names as DNS names.

4) volumes: Persistent data storage

The volumes section defines Docker volumes that should exist for the application. Volumes are used when you want data to survive container restarts and container deletion. This is especially important for databases, but it’s also helpful for development workflows where you want to persist files, cached dependencies, or app state.

In the example, a volume called counter-vol is defined and mounted into the web-fe container. This means some directory inside the container maps to a real persistent storage location managed by Docker.

Breaking Down the Example Services

version: "3.8"
services:
  web-fe:
    build: .
    command: python app.py
    ports:
      - target: 5000
        published: 5000
    networks:
      - counter-net
    volumes:
      - type: volume
        source: counter-vol
        target: /code
  redis:
    image: "redis:alpine"
    networks:
      counter-net:

networks:
  counter-net:

volumes:
  counter-vol:

The web-fe service

The web-fe service is a custom application container. Instead of pulling a prebuilt image from Docker Hub, it uses build: .. That tells Docker: Build an image from the Dockerfile in the current directory. This is a very common development pattern because you keep the application code and Dockerfile together in one repo.

Next, the service defines a command like python app.py. This tells the container what process to run when it starts. In this case, the main process is a Python program called app.py your Flask web app.

It also defines ports, mapping port 5000 inside the container to port 5000 on the host machine. This is what makes the app accessible from your browser. Without a port mapping, the service might run fine but would be isolated inside Docker networking and not reachable directly from your host.

The service attaches to the counter-net network. This is critical because it allows web-fe to communicate with redis over an isolated internal network and it also keeps the service topology tidy instead of throwing everything onto the default network.

Finally, the service mounts a volume. The Compose file mounts counter-vol into /code inside the container. This means files written into /code in the container are actually stored in a Docker managed volume, so they persist beyond the lifecycle of that one container instance.

Putting it all together, Compose deploys one container for web-fe, built from your local Dockerfile, running app.py, exposed on port 5000, connected to the app network, and using persistent storage for the mounted path.

The redis service

The redis service is simpler because it uses an existing image: redis:alpine. This tells Docker to pull the Redis image from Docker Hub if it’s not already present locally.

It also attaches to the same counter-net network, so the web service can talk to it. There’s no need to publish Redis ports to the host unless you specifically want to connect to Redis from outside Docker. Many apps keep Redis internal, accessible only to other containers.

Deploying a Multi-Container App with One Command

Let's clone an example app repo and start it using Compose.

• git clone <https://github.com/red-star25/docker_tutorial_compose_example>

• docker-compose up &

The docker-compose up command is the standard way to bring up an app. When you run it, Compose does several things for you automatically:

1. Builds images that require building (like the web-fe service)

2. Pulls images that need downloading (like redis:alpine)

3. Creates required networks counter-net)

4. Creates required volumes ( counter-volume)

5. Starts containers in the correct configuration

By default, docker-compose up expects the Compose file to be named docker-compose.yml. If your file is named something else, you must pass it with -f. For example, if your file is something-else.yml, you would run docker-compose -f something-else.yml up.

Many people also start apps in “detached mode” using -d, which runs containers in the background and returns your terminal immediately. So instead of using &, you can run docker-compose up -d. Both approaches give you your terminal back, but -d is the standard Compose way.

After the app starts, you can confirm what happened by listing images, containers, and networks. You’ll notice a new image for the web app (created from your Dockerfile) and a container for each service.

docker container ls

docker image ls

At this point, you’ve successfully deployed a multi-container app using Compose.

Managing the App Lifecycle with Compose

Once the app is running, Compose becomes your control panel.

Stopping and removing: docker-compose down

If you want to stop the app and remove the containers and network it created, you use docker-compose down. This is like the tear down the environment command. It shuts down the services and removes the containers (and usually the default network created for the app).

docker-compose down

A key detail from here is that volumes are not deleted by default when you run down. This is intentional. Volumes are meant to store persistent data, so Docker treats them as long-lived resources. That means if your app wrote data to a volume, the data will still be there the next time you bring the app up.

Also, images you built or pulled remain on the system. This is another reason redeployments become faster because Docker doesn’t need to download or rebuild everything again unless something changed.

Bringing it back quickly: docker-compose up -d

If you run docker-compose up -d again after bringing it down, you’ll often see it start faster. The images are already present and the volume still exists, so Compose only needs to recreate containers and the network, then start everything.

Checking status: docker-compose ps

To see what containers belong to the Compose app and their status (running/stopped), you use docker-compose ps. This is more convenient than docker container ls because it focuses on the services in the Compose project.

Seeing processes inside containers: docker-compose top

If you want to see which processes are running inside each service container, you can use docker-compose top. This is useful when debugging “Is my app actually running?” situations.

Stop without deleting resources: docker-compose stop

Sometimes you just want to pause the app without removing containers, networks, and configuration. docker-compose stopstops all containers in the Compose app but keeps them around. You can then confirm their status using docker-compose ps.

Restarting: docker-compose restart

If you’ve stopped the app (or if something is acting strange), docker-compose restart restarts the services. It’s a convenient way to bounce the entire application without destroying and recreating everything.

Key Docker Compose Commands

To wrap everything up, here’s what the common commands mean in simple terms:

docker-compose up starts the whole app (building/pulling images, creating networks/volumes, starting containers).

docker-compose up -d does the same but runs in the background.

docker-compose ps shows the status of the app’s containers.

docker-compose top shows the processes running inside each container.

docker-compose stop stops the containers but keeps them and their resources.

docker-compose restart restarts containers that belong to the app.

docker-compose down stops and removes the app’s containers

docker-compose rm removes stopped containers

Final Thoughts

Docker Compose is powerful because it turns a messy set of manual Docker commands into one clean, version-controlled application definition. It makes it easy to bring up complex multi-service applications on your machine or on a server, and it gives you a consistent way to manage the application over time.

Once you get comfortable reading and writing Compose files especially understanding services, networks, and volumes you’ll find deploying and managing multi-container apps becomes much simpler and far less error-prone.

See you in the next one, until then...

https://giphy.com/gifs/fallontonight-jimmy-fallon-peace-out-tonightshow-2nlbKhgnvAK3sR8ffw?utm_source=iframe&utm_medium=embed&utm_campaign=Embeds&utm_term=https%3A%2F%2Fdhruvnakum.xyz%2F

When you containerize an app, you are basically preparing it so Docker can turn it into an image, and then run that image as a container.

What does “containerizing” actually mean?

Think of containerizing like packing your app into a travel bag:

• Your app code is inside the bag.

• Your app dependencies, like Node.js packages, etc, are inside the bag.

• The instructions on how to start your app are also inside the bag.

Once the bag is packed, anyone can run it the same way, as long as they have Docker.

The usual flow of containerizing an app

Most Docker container workflows follow the same steps:

First, you start with your application code and whatever it depends on (libraries, packages, config files). Next, you write a Dockerfile. This Dockerfile is a set of instructions that tells Docker how to build your app into an image.

After that, you run a Docker build command. Docker reads your Dockerfile line by line and creates an image. Once you have an image, you can store it in an image registry (optional, but very common). Finally, you run a container using that image.

So the flow is:

You have code → you create a Dockerfile → you build an image → you push it to a registry (this is optional) → you run a container.

Example: containerizing a simple single-container app

Let’s say you want to containerize a small Node.js web app.

You copy the code from GitHub and go into the project folder:

• https://github.com/red-star25/docker_tutorial_sample_nodejs_app.git clone this repository

Inside that folder, you’ll find the Dockerfile.

What is a Dockerfile?

A Dockerfile is the starting point for creating a container image. It describes:

• What base system should your app start from

• What software needs to be installed

• What files to copy into the image

• Which command should run when the container starts

One important detail: the folder that contains your app code and files you want to copy into the image is called the build context. Most people keep the Dockerfile at the root of this folder, so it is easy to build.

Also, the name matters: it must be exactly Dockerfile with a capital D, and it must be one word.

Walking through the example Dockerfile

Here is the Dockerfile from the page, and what each line is doing:

The first line is:

• FROM alpine

This means: start with Alpine Linux as the base image. Alpine is a very small Linux image, so it’s popular for small containers. Every Dockerfile must start with a FROM instruction because it sets the base layer.

Then we have:

• LABEL maintainer="dhruv.nakum25@gmail.com"

A label is metadata. It doesn’t install anything or change files in a big way. It’s just extra information added to the image. Adding a maintainer label is a nice practice because people know who to contact about the image.

Next is:

• RUN apk add --update nodejs nodejs-npm

This installs Node.js and npm in the image using Alpine’s package manager apk. This step creates a new image layer because it adds software to the image.

Then:

• COPY . /src

This copies everything in your build context (your current folder) into the image at /src. This also creates a new layer, because you are adding files to the image.

Next:

• WORKDIR /src

This sets the working directory inside the image. From this point onward, commands run as if /src is the current folder. This is usually metadata, not a layer that adds content.

After that:

• RUN npm install

This runs npm install inside /src (because of the WORKDIR). It installs dependencies listed in package.json. This creates another new layer because it adds installed packages into the image.

Then:

• EXPOSE 8080

This tells that the app listens on port 8080 inside the container. This is important for humans and tools, but it’s metadata, not a layer.

Finally:

• ENTRYPOINT ["node", "./app.js"]

This tells Docker what command should run when the container starts. In this case, it runs the Node app. This is also metadata.

Building the Docker image

Once you have the Dockerfile, you build the image using the build command. In this example, the image is called web:latest.

The command is:

• docker image build -t web:latest .

The dot (.) at the end is very important. It tells Docker use my current folder as the build context.

After the build finishes, you can list images to confirm it exists:

• docker image ls

You can also inspect the image to see what settings Docker stored from your Dockerfile:

• docker image inspect web:latest

This is a good way to confirm things like the entry point and the image layers.

Pushing images to a registry (Docker Hub)

Pushing is optional, but it’s very useful. If your image is only on your laptop, no one else can pull it easily. If you push it to a registry, you can download and run it from anywhere.

Docker Hub is the most common public registry and is the default place Docker pushes to.

First you login:

• docker login

Before pushing, you need to tag the image properly. Docker needs three parts:

• Registry (example: docker.io for Docker Hub, even if you don’t write it)

• Repository (often includes your username)

• Tag (like latest, v1, etc.)

If you try to push web:latest directly, Docker will attempt to push it to a repository named web, which you probably don’t own.

So instead, you tag it with your Docker Hub username (Docker ID). Example from the page:

• docker image tag web:latest dhruvnakum/web:latest

This does not remove the old tag. It just adds another tag pointing to the same image.

Now you can push:

• docker image push dhruvnakum/web:latest

Once it is pushed, you can pull it later from anywhere.

Running the container

After the image is built, you run it as a container.

Example command:

• docker container run -d --name mycontainer -p 80:8080 web:latest

Let’s break that down simply:

• -d means run in the background (detached mode)

• --name c1 gives the container a friendly name

• -p 80:8080 maps port 80 on your computer to port 8080 inside the container

• web:latest is the image you want to run

So if your app listens on 8080 inside the container, you can visit http://localhost in your browser because port 80 on your machine is connected to port 8080 in the container.

A closer look: layers vs metadata

Docker reads the Dockerfile from top to bottom, one line at a time. Some lines create real filesystem layers, and some lines only create metadata.

In general:

• Instructions like FROM, RUN, and COPY create layers because they add or change files.

• Instructions like WORKDIR, EXPOSE, ENV, and ENTRYPOINT usually add metadata instead of creating layers.

If you want to see the image build steps and layers, you can use:

• docker image history web:latest

And to verify the final result including entrypoint and layers, use:

• docker image inspect web:latest

Why image size matters

When it comes to Docker images, big is bad.

Smaller images are usually:

• faster to build

• faster to push and pull

• easier to store

• less risky

A common problem is that we install build tools and temporary dependencies, but then we forget to remove them. That makes production images bigger than they need to be.

Multi-stage builds solve this in a clean way. A multi-stage Dockerfile has multiple FROM lines. Each FROM starts a new stage. You can build in one stage, and then copy only the final needed output into the last stage (which stays small and clean).

Build cache best practices

Docker tries to speed up builds using cache.

For each Dockerfile instruction, Docker checks if it already has a cached layer for that exact step. If yes, it reuses it, and the build is faster. If no, Docker builds a new layer.

But here’s the key: once the cache is broken at some step, Docker usually has to rebuild everything after that step too.

That’s why ordering your Dockerfile matters a lot.

Also, Docker checks files you copy. Even if the Dockerfile line didn’t change, if the content of files in the folder changed, the checksum will change and Docker will rebuild that layer.

If you ever want to ignore cache completely, you can build with:

• docker image build --no-cache=true -t web:latest .

Squashing images

Sometimes images end up with many layers. Squashing can combine layers into fewer layers.

This can be helpful when you want fewer layers, but it has a downside: squashed images don’t share layers as efficiently. That can increase storage usage and make pushes/pulls bigger in some cases.

If you want to squash during build, you can add --squash to the build command.

Conclusion

So we learned that containerizing an app is mainly about writing a good Dockerfile and using it to build a clean image. Once you have an image, you can run it the same way on any Docker host, and you can share it by pushing to a registry. That's it.

Next up, we have docker-compose. It's the most fundamental topic for multi-stage builds. And we use it in production. So it's gonna be really fun learning that. But for now, create a simple app, containerize it using Dockerfile, and try it on your own and see if you understood the concept till now. Until then...

We learned about Images. What image is, and how it works internally. In this blog, we are gonna dive deep into Containers and learn what they are, how they differ from VMs, their use cases, their lifecycle, and much more. So let's get right into it.

What is a container?

Containers are one of the main ideas behind Docker. If you understand what a container is and how to run one, most Docker commands will start to make sense.

A container is a running copy of an image. What does that mean!!?

In the previous blog, we learned about images and saw that an image is like a blueprint/class that has everything needed to run an app. And to make use of the class what do we do in programming!!?? Yes, we make objects out of it.

That's what a container is. A container is what you get when you start that image. And you can start many containers from the same image, as you would with a class and an object.

Let's now see the difference between Containers and VMs, because hey, we had VMs earlier, which solved our problems, right? Then why container?

Containers vs Virtual Machines (VMs)

Containers and VMs both run on a host machine (your laptop, a server, or a cloud instance). But they work differently.

How VMs work

VMs use a hypervisor. You might or might not know about a hypervisor, so let me give you the basic idea about it.

Hypervisor

A hypervisor is software that lets you run multiple virtual machines (VMs) on one physical computer. It creates virtual machines, Gives each VM its own CPU, memory and storage. It keeps VMs isolated from each other and shares the physical hardware safely.

So instead of one computer running one OS, you can run: Window, Linux, Another Linux, etc all on the same machine

So each VM has, its own OS and kernel. This is why we say "Hypervisors virtualize hardware."

How do containers work?

Unlike VMs, containers do not create an OS for every app. It shares the host operating system and also shares its kernel.

What it does is split OS resources into isolated containers, meaning each container has its own filesystem, processes, and network space.

This is why we say, “Containers virtualize the operating system.”

One downside of containers is that they are less secure by default than VMs, because isolation is not as strong as that of VMs. There are ways to improve security, but they can add complexity.

Running your first container

Let's now create our first container, and the easiest way to start a container is:

docker container run <image> <app>

Example:

docker container run -it ubuntu /bin/bash

What this does:

• docker container run creates and starts a new container

• -it connects your terminal to the container (interactive mode)

• ubuntu Is the image

• /bin/bash is the app (a shell) that runs inside the container

Your terminal prompt will change because you are now inside the container.

What's happening behind the scenes?

So when you hit enter after running the above command, the Docker client will send this command to the API server running on the Docker Daemon. Docker Daemon will search for the Docker host's local image repository to see if it already has a copy of the requested image. If it finds the image, it will pull it locally, otherwise, it will go to the Docker hub (image repository) and see if it can find it there.

Once the image is pulled, Docker Daemon tells containerd and runc to create and start the container (remember the Docker architecture we discussed in Part -1). Once its done you will be able to get inside the container.

A container runs as long as the main app is running. If the app exists, the container stops. If you stop the container, it stops. If you remove the container, it is gone.

Stopping, starting, and deleting containers

Common lifecycle commands:

• Stop a running container:

docker container stop <container>

• Start a stopped container:

docker container start <container>

• Remove a container forever:

docker container rm <container>

Working with container processes

If you are inside a container running /bin/bash, and you kill that main bash process, the container stops.

To exit the container but keep it running:

• Press: Ctrl-PQ

Now you are back on your host machine, but the container is still running in the background.

Check running containers:

docker container ls

Re-attach using exec

You can open a new shell inside the running container:

docker container exec -it <container_id> bash

This creates a new process inside the container.

So if you type exit here, the container can still stay alive because the original main process is still running.

Container lifecycle

You can give a custom name to your container like this

docker container run --name boo -it ubuntu:latest /bin/bash

After running it, we will be inside the container. Let's create a file here

cd tmp
echo "Hello World" > newfile
ls -l
cat newfile

Detach without stopping it:

• Ctrl-PQ

Stop it:

docker container stop boo

Start it again:

docker container start boo

Go back inside:

docker container exec -it boo bash

Check the file:

cd tmp
ls -l
cat newfile

You will see the file is still there.

So what did we learn from this?

Two important notes

1. That the data lives on the Docker host. If the host fails, the data is lost.

2. Containers are meant to be immutable, so writing data inside them is not a good habit.

That is why Docker has volumes, which store data outside the container. We will look into volumes in the upcoming blog.

Stopping containers gracefully

There is a big difference between stopping and force-killing.

• docker container stop is polite.

  It sends a SIGTERM to the main process (PID 1) and gives it time (usually 10 seconds) to shut down cleanly.

  If it does not stop in time, Docker sends SIGKILL.

• docker container rm -f is not polite.

  It kills the container right away (no clean shutdown).

Restart policies

It’s often a good idea to run containers with a restart policy. Why do we care? Because in the real world, we might have some container that crashes or stops suddenly, and we want them to restart automatically. This is where Restart policies come into play.

Restart policies enable Docker to automatically restart them after certain events or failures have occurred.

Common policies are:

• always - If we attach this policy to the container, Docker will always restart the stopped container if it's closed by Docker or fails.

• unless-stopped

• on-failed (often written as on-failure)

Example: Let's create a container with the always policy attached.

docker container run --name zoo -it --restart always ubuntu:lastest bash

Now If you type exit, and get out of the container, the container stops, but Docker starts it again because the policy is always. Try docker container ls and see.

See how the container was created 19 seconds ago, but has only been up for 11 seconds. This is because the exit command killed it and Docker restarted it. Be aware that Docker has restarted the same container and not created a new one.

Difference between always and unless-stopped:

• always: restarts even after Docker daemon restarts

• unless-stopped: does not restart after daemon restart if it was already in a stopped state

Let's see it in action: Create two container with name always and unless-stopped with respective policies attached

docker container run -d --name always \                                                               
--restart always \
ubuntu:latest sleep 1d

docker container run -d --name unless-stopped \                                                               
--restart always \
ubuntu:latest sleep 1d

Now check with docker container ls

Now lets run, and restart the Docker

docker container stop always unless-stopped

Now run

docker container ls -a

Notice that the “always” container has been restarted but the "unless-stopped" container has not.

The on-failure policy will restart a container if it exits with a non-zero exit code.

Let's clean up everything and remove all the container now.

docker container rm $(docker container ls -aq) -f

Be careful with this command as this command will remove every container running forcefully.

Quick list of useful container commands

• docker container run — start a new container

• docker container ls — list running containers

• docker container ls -a — list all containers (including stopped)

• docker container exec — run a command inside a running container

• docker container stop — stop a container

• docker container start — start a stopped container

• docker container rm — delete a container

• docker container inspect — detailed container info

• Ctrl-PQ — detach without stopping the container

Conclusion

We saw how container is beneficial and works with all the important commands to start, stop, and remove. Also we learned how restart policies plays an important role in Docker ecosystem.

In the next blog, we will implement everything we learned so far by containerizing the application. Untill then...

We learned about some fundamental concepts of Docker and its architecture. We saw why we need Docker in the first place and how it is solving the problems and making things easier than the good old VMs.

In this blog, we are going to understand the core part of Docker, Images. We will see the difference between Images and Containers, and also learn about the Image registries.

We will see the commands that help us pull, create, and run the Docker image and much more. So let's get started

What is a Docker image?

A Docker image is a ready-to-use package that has everything an app needs to run, such as:

• The application code

• The app’s required libraries

• Basic operating system files

If you are a developer, you can think of an image like a class blueprint. You use it to create real running things.

Docker images are mainly used at build time, while containers are what you use at run time when the app is actually running.

Where do Docker images come from?

Most of the time, you do not create images from scratch. You pull them from a place called an image registry, and the most common registry is Docker Hub.

So when we pull an image from the Docker Hub, the Docker daemon downloads it to your computer (your Docker host). After that, Docker can use it to start one or more containers

Images are made of layers

A Docker image is not one single file. It is built from many read-only layers stacked on top of each other.

Inside these layers, we have a small cut-down operating system and all the required files and dependencies for the app. Which means you don't get the full system with all the things.

Docker stacks the layers and shows them as one unified image.

When you pull an image like this:

docker image pull ubuntu:latest

Each line that ends with “Pull complete” is basically one layer being downloaded.

You can also inspect layers using:

docker image inspect ubuntu:latest

The idea here is, every image starts with a base layer, and when we add things (like installing Python or adding your app code), Docker adds new layers on top. For example, we have this Ubuntu image. And then we add Python to it, and then we add source code.

Now, the final image becomes a stack of layers in that order.

(Source: Docker Deep Dive - Nigel Poulton)

Docker also has a system called a storage driver that handles stacking these layers. There are many drivers, but the experience feels the same to you.

Images and containers: how they connect

We will learn about containers in the next blog, but for now, just understand that you can start containers from an image using commands like, docker container run and docker service create. Once a container is created from an image, the image and container are linked, and we cannot delete the image until all containers using it are stopped and deleted. If you try, Docker will give an error.

Also, the purpose of a container is usually to run one app or one service, so the image should only include what that app needs.

For example:

• If the app does not need a shell, the image should not include many shells

• Docker images also do not include a kernel. All containers on a machine share the host machine’s kernel

Image registries and repositories

• An image registry is a central place to store images (for example, Docker Hub).

• Image registries contain one or more image repositories. In turn, image repositories contain one or more images.

Docker Hub has two types of repositories:

• Official repositories

  • Checked and curated by Docker

• Unofficial repositories

  • Can be risky

  • Do not assume they are safe, well-documented, or built correctly

Image names and tags (how you pull the right one)

For official images, pulling is usually:

docker image pull <repository>:<tag>

Examples:

docker image pull alpine:latest
docker image pull redis:latest
docker image pull mongo:4.2.6
docker image pull busybox:latest

If you run:

docker image pull alpine

Docker assumes you mean:

• alpine:latest

Two important notes about latest

If you do not specify a tag, Docker assumes latest. And If the repo does not have a latest tag, the pull will fail.

latest does not mean “newest”. It is just a label. For example, in Alpine, the newest is often tagged edge. So be careful when using latest.

Pulling images from unofficial repos

Same idea, but you include the username or org name:

docker image pull someusername/imagename:version

Pulling from other registries (not Docker Hub)

If the image is in another registry, like there are google registries too, so to get that we include the registry’s DNS name:

docker image pull gcr.io/google-containers/git-sync:v3.1.5

Searching Docker Hub from the command line

You can search Docker Hub using:

docker search alpine

• It searches for repos that match a string in the NAME field

• NAME is the repository name

To show only official repos:

docker search alpine --filter "is-official=true"

Pulling images by digest

Pulling by tag is common, but there is a problem: Tags can change. Someone can push a new version using the same tag. Then you might not know which exact version your running systems are using

For example, you have exampleimage:1.5 with a known bug. You fix it and push it again as exampleimage:1.5Now two different images have used the same tag name over time. It becomes hard to know what is running where

Docker uses a content-based ID called a digest, which is a cryptographic hash.

The idea here is that if the image content changes, the digest changes. So digests are unchangeable, and they uniquely identify the exact image

When you pull an image, Docker often shows the digest:

docker image pull alpine

You can list digests locally:

docker image ls --digests ubuntu

And you can pull the exact same image again using the digest:

docker image pull ubuntu@sha256:d1e2e92...7e495eff4f9

This makes sure you get exactly the image you expect.

Multi-architecture images

Different machines can have different Operating systems (Linux, Windows), different CPU types (x64/amd64, ARM, etc.).

For example, your laptop might be Linux on x64, A Raspberry Pi is Linux on ARM, A Windows server might be Windows on x64

Before multi-architecture support, you had to manually pick the right image, which was confusing.

So, when you run

docker pull golang:latest

Docker automatically detects your OS and CPU, pulls the correct version for your system. You don’t need to specify anything.

How does Docker do this?

Docker uses two important things in the registry:

1. Manifest List: This is nothing but a list of supported platforms for that image tag

   • Example: Linux/amd64, Linux/arm, Windows/amd64

2. Manifests: Each platform has its own Manifest. And that Manifest lists the layers and configuration for that platform

So the Flow is:

docker pull golang:latest
        |
Docker checks manifest list
        |
Finds match for your OS + CPU
        |
Downloads correct layers

Docker also lets you build images for other CPU types using buildx.

Example:

docker buildx build --platform linux/arm/v7 -t myimage:arm-v7 .

You can even build ARM images while working on an x64 machine. How cool is that!!

Deleting Docker images

To delete an image:

docker image rm 02674b9cb179

Delete multiple images:

docker image rm f70734b6a266 a4d3716dbb72

Important rules:

• Deleting an image removes the image and its layers.

• But if a layer is shared by multiple images, it will not be removed until all those images are deleted.

• If an image is used by a running container, Docker will not let you delete it

  • Stop and delete the container first

Shortcut to delete all images (force):

docker image rm $(docker image ls -q) -f

# Here $(docker image ls -q) will list all the images. Its a shortcut.

Conclusion

This is more than enough for you to understand everything about Docker images right now. In the next blog, we will dive into Containers and see what a container is and how it actually works under the hood. Until then...

https://media.giphy.com/media/v1.Y2lkPTc5MGI3NjExbjFtZG8yY3V6azR5ejRoY2p4eXltcWdzcDVrMXVhcjlzdnVrcG5zayZlcD12MV9naWZzX3NlYXJjaCZjdD1n/2nlbKhgnvAK3sR8ffw/giphy.gif

So, I've been reading this book called Docker Deep Dive Zero to Docker by Nigel Poulton. Why? Because I've been working on mobile, web, and cloud applications for quite a long time now. And I really wanted to know how Docker works internally and how it does the magic.

And then I came across this book and thought to give it a try because I love reading, and to be honest, I hardly read technical books to learn about something personally (well, except college and study books, of course lol). So while reading it, I was also making notes for myself.

I will be sharing everything I learned from this book. I will be using the book to explain and rewrite things, making things simpler to understand, because some of the things later on in this book took a while for me to understand. I will also be using Nano Banana to create visuals to help you better understand because I don't want to waste time on graphics too much. And obiously AI does better job than me now in designing these things ;)

So, let's get started without further ado.

The Big Picture

A few years ago, running apps looked very different. Most companies ran one application per server. If the business needed a new app, the IT team often had to buy a new server. This cost a lot of money and wasted resources, because many servers had unused CPU and RAM.

Then virtual machines (VMs) became popular.

VMs Solved One Problem, But Created Another

VMs made it possible to run multiple applications on one server. That was a big improvement. But there was a downside too, which were

• Each VM needs its own full operating system.

• That OS uses CPU and RAM even when the app is small.

• Sometimes each OS also needs its own license.

• VMs can be slow to boot.

• Moving VMs across machines is not always smooth.

So even though VMs helped, they still had a lot of overhead.

Containers: A Lighter Way To Run Apps

To solve the above problem, containers come into the picture. What is a container? For now, just think that a container is similar to a VM in one way. It runs an application in an isolated environment.

But the key difference is that Containers share the host machine’s kernel. They do not need a full OS per container

And because of this, containers are Faster to start, more lightweight, and Easier to move around, which was the problem in VMs, remember?

Google has used container tech for a long time. But for many companies, containers were still too complex to use directly. To address this complexity and make things easier for everyone, Docker Inc. began developing Docker.

What Does “Docker” Mean?

When people say “Docker”, they might mean two things:

1. Docker, Inc. - the company

2. Docker, the technology - the tool that creates and runs containers

Docker, the technology runs on Linux and Windows, and helps you build, run, and manage containers.

Docker Architecture

There are three things that we need to be aware of when we talk about Docker.

1. Docker Runtime

2. Docker Daemon

3. Docker Orchestrator

Let's see each one of them in detail now.

Docker Runtime

Docker uses a “tiered runtime” setup. Low-level and High-level runtime.

When I say Docker uses a tiered runtime architecture, I mean that Docker splits responsibilities between different components instead of having one big program do everything.

Low-level runtime: runc

• This talks to the operating system. Starts and stops containers. Each container typically has a runc instance managing it.

High-level runtime: containerd

• This manages the whole container lifecycle. It pulls images. Sets up networking and calls runc when needed

You do not need to memorize this on day one, but it helps to know Docker has layers.

Docker Daemon

Docker Daemon sits above containerd and performs higher-level tasks, such as exposing the Docker remote API, managing images, volumes, and networks, and more.

The Docker daemon’s main job is to provide an easy-to-use standard interface that abstracts the lower levels.

Docker Orchestrator

Before we understand this layer, we need to understand what orchestration means.

You see, running one container is easy. But in real apps, you often need many containers, like web apps, databases, caches, and background workers.

Orchestration means managing all of those automatically. If you have seen any performance in an orchestra where the orchestrator manages all the musicians, the same concept applies to the Docker too.

• The orchestrator, Start the containers, restart if they crash, scale up when traffic increases, and connect them properly

Docker has its own orchestrator called Docker Swarm, but today most teams use Kubernetes.

Kubernetes in one line (because we are not going into the details right now)

Kubernetes is the most popular platform for deploying and managing containerized apps.

OCI: Why Standards Matter

There is also something called the Open Container Initiative (OCI). In simple terms, it's nothing but a government council that sets standards for the image and runtime formats.

An analogy often used to describe these two standards is rail trails.

This is useful, and it’s fair to say that the two OCI specifications have had a major impact on the architecture and design of the core Docker product.

Installing Docker

Docker is available everywhere to download. There are Windows, Mac, and Linux applications. You can install it in the cloud, on premises, and on your laptop. And there are manual, scripted, and wizard-based installs, too. There literally are loads of ways and places to install Docker.

Docker Desktop

This is the main application we use for everything. Go to Docker Desktop website and download it on your respective operating system. Once installed, you will see the interface something like this:

Docker’s Main Parts

So when you install Docker, you mainly work with two pieces: Docker Client and Docker Daemon

1) Docker Client

This is what you interact with, usually from the terminal.

Open your terminal and run the command below. You will see the current Docker version installed in your system.

Example:

docker version

2) Docker Daemon

This is the background service that does the real work:

• pulls images

• runs containers

• manages networks and volumes

• exposes the Docker API

The client talks to the daemon. This is the high-level explanation. Now, let's get into the details

The Two Core Docker Objects: Images and Containers

Docker Image

A Docker image is like a package that contains:

• a small filesystem like Linux files, or it can be your app, or the dependencies needed to run the app

The important thing here is that it's an image, not a running instance. It is more like a blueprint. You can think of a Docker Image as a Class in the world of programming. And when you create an Object of that Class, it becomes a real thing and gets memory allocated in the system. In this case, that object is Container.

Getting an image onto your Docker host is called pulling. So whenever you want to get the image, you run the command below,

docker image pull ubuntu:latest

And from where this image will come from, you might ask? We will talk about it in the upcoming article, don't worry. For now, just know that it comes somewhere from the internet. And so, after pulling the image, you will see something like this.

To check whether you successfully installed that image, there are two ways: First, to run the command below in your terminal

docker image ls

Each image gets its own unique ID. When referencing images, you can refer to them using either IDs or names

And the second way is to check the Docker Desktop

Docker Container

A container is a running instance created from an image.

So:

• Image = blueprint

• Container = running app

Now that we have an image pulled locally, we can use the Docker container run command to launch a container from it.

docker container run -it ubuntu:latest /bin/bash

Let's understand the command here first:

docker container run tells the Docker daemon to start a new container.

-it flag tells Docker to make the container interactive and to attach the current shell to the container’s terminal.

ubuntu:latest command tells Docker that we want the container to be based on the ubuntu:latest image.

/bin/bash tell Docker which process we want to run inside of the container. For linux we are running a Bash shell.

As you can see, after running the command, we got inside the Ubuntu container. And it's a real Ubuntu image. So you can run the command you run on Ubuntu. But not all of them right now. Why? We will come to it later, don't worry.

You can exit the container without terminating it by Ctr - PQ command

If you want to see all the containers, you can run

docker container ls

Now, if you have exited the container, you can again get back into it using the exec command

docker container exec -it 0acc2d5f2fef bash

We used the -it options to attach our shell to the container’s shell

We can stop and kill the container using the commands below

docker container stop <name/id>

docker container rm <name/id>

IMPORTANT NOTE:

• If you are like this right now...

https://media.giphy.com/media/v1.Y2lkPTc5MGI3NjExcjE1MmVtZ3o4bXF1NzJ2aXMzZ3prYmd1dWU3NnY3ZmZ5ZDcwZ3pjbiZlcD12MV9naWZzX3NlYXJjaCZjdD1n/GPg6PuL5RkeyVCD5wk/giphy.gif

Conclusion

We learned about VMs and saw why it's not efficient to use them anymore. We saw how containers can be so much lighter and faster than VMs.

We also saw what Docker is and how it works using its architecture. We learned about OCI and how it helps maintain the standard practice of images and containers.

We also went through some basic commands of Docker, like:

This big picture view should help you with the upcoming article, where we will dig deeper into images and containers.

See you in the next article, until then....

https://media.giphy.com/media/v1.Y2lkPTc5MGI3NjExbjFtZG8yY3V6azR5ejRoY2p4eXltcWdzcDVrMXVhcjlzdnVrcG5zayZlcD12MV9naWZzX3NlYXJjaCZjdD1n/2nlbKhgnvAK3sR8ffw/giphy.gif

• I've been learning a lot about AWS lately, and to be honest, there's a lot to understand and learn. I decided to take a little break and try to implement what I've learned so far. At the same time, I'm working on a project that's still in progress, and I'll be announcing it soon. I have a backend app in Node.js for this project and wanted to deploy it. So, I thought, why not deploy it on AWS to test my knowledge and see if I've learned anything?

• A little bit about my Node.js app: I'm using JavaScript, and I have MongoDB running. It's on Atlas in the cloud. I think that's all you need to know. If you have a Node.js application running, you can follow similar steps to make your backend accessible to your frontend application or any third-party consumer.

• So, without further ado, let's get started.

Set up an AWS Account

• First, we need an AWS account, so I'm assuming you already have one. If not, it's really easy to create one. Go to https://aws.amazon.com and create an account.

• It might ask for your credit card information, but don't worry, they won't charge you until you exceed the free-tier limit. In this guide, we won't exceed the free tier, so there's no need to worry. Just add your details, and you're all set.

AWS EC2 Instance

• So, we are going to use the AWS EC2 service to deploy our application to AWS. If you're not familiar with EC2, think of it as renting a virtual machine, similar to your laptop or PC. There are many benefits, such as storing data, choosing your operating system, deciding how much computing power you need, selecting the number of CPU cores, RAM, and the type of network you want to use, and more.

• You can customize your virtual machine as you like and rent it from AWS, which is the power of the cloud.

• What we'll do is copy our local app, with all its files and folders, directly to this virtual machine and then run the app there. The advantage is that anyone with the link can access it, not just us on our local machine.

Creating EC2 Instance

• Simply search for EC2 in the search bar and click on it to open the service page.

• Once it's open, you will see a page like this.

  

  • Simply click on a Launch Instance on Dashboard and give it a name.

• It will also ask you to select the operating system image. Choose Ubuntu and scroll down.

• We will use the t2.micro instance because it is part of the free tier. If you need a higher configuration CPU, you can check other instances and select the one that meets your needs.

• To connect to the instance from our local machine or anywhere else, we need access. To do this, we generate a key pair, which we can use to connect to our AWS instance and launch it in our local environment.

• Click on "Generate Key Pair," give it a relevant name, and make sure to select the RSA key pair type. Finally, create the key pair, and it will download to your local machine.

• In the network settings, select:

  • Allow SSH traffic from - My IP - Because we only want to access the instance from our local machine.

  • Allow HTTPS traffic from the internet (so our application can access our Node.js server using an endpoint or URL with HTTPS)

  • Allow HTTP traffic from the internet (so our application can access our Node.js server using an endpoint or URL with HTTP)

• That's it. Now, review the summary and launch the instance.

Connecting via SSH

• Once it’s created, click on Connect to your instance

• This will redirect you to the interface shown below. Here, you can run the instance within AWS Cloud itself and access it, or you can use the SSH Client to access this instance from your local machine, which is what we want.

• Before setting up SSH, remember that we downloaded the Key-Pair. Let's first place it in the correct location.

• I am using a Mac, and I usually store my SSH keys inside the .ssh folder located in the home directory. Let's first navigate to the home directory.

cd ~/

• Now, move the key pair from the download folder to the .ssh folder using the mv command below:

• Once it’s done, we are good to go. So let’s follow these instructions.

• Run chmod A command to change the permission to ensure your key is not publicly viewable.

• Now run

•   ssh -i "<your-key-name>.pem" ubuntu@<your-ip>
  

  to connect to the instance we created.

• Type yes and press Enter

• As you can see, we are now inside the instance we created. Isn't it cool that we can access it from our local machine?

• Now you can perform any tasks that you would normally do on your local machine.

Installing/Updating Packages

• Now that we have our instance connected, before starting anything, we need to ensure we are using the latest packages in Ubuntu and install any available updates. To do this, we run two commands:

sudo apt update

• Which refreshes package lists (checks for updates) and then

sudo apt upgrade

• Which Installs available updates for installed packages

Installing Node.js in an EC2 instance

• Now that we have all the latest packages, we need to install Node.js (or the backend framework/library you need).

• To do this, we need to download and run the NodeSource setup script, which configures the system to install Node.js version 20 from their official repository.

• Use the command below to do that:

curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash -

• Once the NodeSource repository is added, we need to install Node.js and npm (Node.js’s package manager) on our system to use Node.js.

• To do this, run the command below

sudo apt-get install -y nodejs

Pushing Local App to EC2

• Now that we have the Node environment set up, we can upload our local Node.js app to this instance. We can do this using the SSH connection we already set up.

rsync -avz --exclude 'node_modules' --exclude '.git' --exclude '.env' \
  -e "ssh -i ~/.ssh/your-key.pem" \
  . ubuntu@ip-address:~/app

• In simple terms, this command copies the current folder (.) to the remote server’s ~/app directory using SSH. It skips unnecessary files like node_modules, .git, and .env because we don't need those.

NOTE: Be sure to replace ‘your-key’ with your SSH key name in the command above, and replace ‘ip-address’ with your actual IP address.

• To execute this command, go to your code directory and run the command above.

• If you now run the ls command in your instance's home directory, you will see the app folder has been created.

  

• 

  And if you use cd to enter it, you will see all your source code.

Setting up Environment (.env) File.

• If you have a database running locally, like PostgreSQL or MySQL, you can install those on this instance just like we did for Node.js.

• I have MongoDB running on MongoDB Atlas. So, to allow this instance to access the database, we need to add the instance's IP to the whitelist. Simply copy the instance's IP address and paste it into the IP Access List entry.

• Now that everything is set up, we need one more file: the .env file. Remember, we excluded it when we transferred our files to the instance.

• So let’s create the .envfile

cd app/
sudo vim .env # This will create and open .env file

# .env
# paste your environment variables

• You can exit the Vim editor by pressing esc, then : (colon), typing wq, and pressing enter.

• As you can see, we now have the .env file in place.

• Before we run our app, let’s make sure we have downloaded all the required packages by running

npm i

• Now, if we run the app, it should run without any errors.

Configuring Security Rules

• To check if it is running, go to the instance dashboard, select your instance, copy the Public IP address, and paste it into your browser.

• We can't see anything, and it didn't work. Why? Do you remember the security group we set up earlier?

• We haven't specified our IP address and because we are accessing it from our machine we must add our IP address with port numner. So, let's add our IP address and the port number to the inbound rules and see if it works now.

• Click on Add Rules and add Custom TCP and specify your IP by selecting My IP.

• Now, if you run it, it should work fine. I am hitting /test the endpoint, which shows Hello World text.

Running App in Background (systemd)

• But here we have two major problems, and I hope you already know what they are.

• One big issue is that it's running on PORT 7575 and doesn't have a domain name. The second issue is that it's running on our local terminal, so if we close this terminal, we won't be able to access our server using that public IP address.

• First, let's address the second issue. To fix that, we need to find a way to run the application in the background so that no matter what we do in the terminal, the server will stay up the whole time.

• If you want to run your Node app as a background service, we use the systemd command.

• We need to create a service file. This file tells systemd (Linux’s init system) how to start, monitor, and restart your Node.js app as a background service, similar to how SSH or MongoDB are managed.

• Run the command below to create a new service file.

sudo vim /etc/systemd/system/myapp.service

• And paste lines below in it once Vim is opened

[Unit]
Description=Node.js App 
After=network.target multi-user.target

[Service]
User=ubuntu
WorkingDirectory=/home/ubuntu/app
ExecStart=/usr/bin/npm start
Restart=always
Environment=NODE_ENV=production
EnvironmentFile=/home/ubuntu/app/.env
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=myapp

[Install]
WantedBy=multi-user.target

• In the [Unit] section, we include our app description and ensure our app starts after networking is ready (so Node can bind to a port).

• In the [Service] section, we specify how the app will run by including the working directory, environment, and other settings.

Now run

sudo systemctl daemon-reload

• Which Reloads systemd so it notices the new service file.

sudo systemctl enable myapp.service

• Enables the service to auto-start on boot.

sudo systemctl start myapp.service

• Start it right now (without reboot).

sudo systemctl status myapp.service

• Shows current status, logs, and whether it’s running or failed.

• And now, if you kill your terminal/app and still hit the URL, it should work.

Setting up Reverse Proxy (using Caddy)

• The next step is to allow HTTP access directly, without using PORT 7575. Then, we will add a domain name and an SSL certificate.

• To keep your actual web server, localhost:7575, hidden from the world and exposing only the HTTP and HTTPS ports, which are 80 and 443, we use reverse proxies like Nginx or Caddy.

• A reverse proxy is a server that sits in front of your application servers and manages all incoming client requests before sending them to your backend app, such as your Node.js service.

• We use this for security purposes. It allows us to add a firewall before requests reach the actual server. It also enables load balancing, as the reverse proxy can distribute requests to multiple servers.

http://54.86.58.33:7575/

We do

https://api.mydomain.com

• To implement this, there are many services like Nginx and Caddy. For this example, we'll use Caddy because it's easy to set up.

• Caddy is a modern, lightweight reverse proxy and web server, similar to Nginx but with a major advantage:

• Visit this URL and run all the commands.

sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https curl
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list
chmod o+r /usr/share/keyrings/caddy-stable-archive-keyring.gpg
chmod o+r /etc/apt/sources.list.d/caddy-stable.list
sudo apt update
sudo apt install caddy

• Once it is done, let’s remove the custom TCP rule we added in the security group for our IP

• And now, if we run

sudo systemctl start caddy

• And then visit the IP address 54.86.58.33 Without mentioning any port, you should see a Caddy page

• But what we want is to forward the traffic to our application instead of showing this page. So, we will edit the Caddy file by opening it using the command below:

sudo vim /etc/caddy/Caddyfile

• Comment down the root and file_server line and uncomment the reverse_proxy line and add your port.

• And now let’s restart Caddy

sudo systemctl restart caddy

• If you again run this URL, you should be able to see your app.

• That’s it, now you can use this URL anywhere to access the backend APIs.

• You can also set up a custom domain name and attach it to this current IP address (for example, example.com) or something like this using the Route53 service.

Conclusion

• Thank you for taking the time to read my blog. I hope you enjoyed it and learned something new. I'll be sharing more about AWS soon as I continue to learn. Keep learning!!

• Until then…

• Connect with me on LinkedIn, Github, and Twitter.

• In the last part, we successfully put the Go application in a Docker container. Now, we need to deploy it so others can access it.

• First, we need to upload our Docker application to Docker Hub. We'll start with that and then set up a GitHub workflow action to help us deploy our application to AWS EC2.

• You might wonder why we use GitHub Actions. In real-world projects, things change often, and you can't manually deploy the application every time. To automate this, we use a tool like GitHub Actions.

• Let's begin with Docker Hub, and then we'll create the workflow file.

Upload Go Application to DockerHub

• • DockerHub is like GitHub but for Docker images. It's an online place where developers can find and share Docker images.

    * You make your custom image on your computer and push it to DockerHub for others to use.

    * As developers, we just need to pull these images using the command docker pull, and we're ready to go.

    * Now, let's upload our Go application to DockerHub.Step 1: Login to Dockerhub

• First, we need to log into the docker hub in order to upload.

• Run the below command.

docker login

• This takes you to the browser where you can log in or sign up. Once you do that, you'll see a message confirming your login was successful.

  

  • Once you are logged in, go to hub.docker.com

Create Repository

• Just like GitHub needs a repository to store code, Docker Hub needs a repository to store Docker images. Go to the Repositories tab in the Navbar.

• And create a public repository. I already have one, so I won't create it again.

Push Image

• Once you've created the repository, you can push the image we made using the docker push command. But first, we need to tag our local image as latest, as shown below.

docker tag anonymous-go dhruvnakum/anonymous-go:latest

• And now let’s push it

 docker push dhruvnakum/anonymous-go:latest

• Replace dhruvnakum with your username, and if you named the repository differently, use that name instead.

• After you run this, the image will be successfully pushed to the repository we just created.

• 

  Now, let's create a workflow to automate deployment.

Creating the GitHub Actions Workflow

• To automate deployment, we will set up a GitHub Actions workflow.

Create the Workflow File

• In the project, create a new file at .github/workflows/cicd.yml.

Define the CI/CD Pipeline

name: Deploy Go Application

on:
  push:
    branches: 
      - ec2

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Source
        uses: actions/checkout@v4
      - name: Create .env file
        run: echo "PORT=${{ secrets.PORT }}" >> .env
      - name: Login to docker hub
        run: sudo docker login -u ${{ secrets.DOCKER_USERNAME }} -p ${{ secrets.DOCKER_PASSWORD }}
      - name: Build docker image
        run: sudo docker build -t dhruvnakum/anonymous-go .
      - name: Push image to docker hub
        run: sudo docker push dhruvnakum/anonymous-go:latest
  deploy:
    needs: build
    runs-on: self-hosted
    steps:
      - name: Delete old anonymous-go container
        run: sudo docker rm -f anonymous-go  
      - name: Pull new anonymous-go docker image
        run: sudo docker pull dhruvnakum/anonymous-go
      - name: Delete old mongo container
        run: sudo docker rm -f mongo-demo
      - name: Pull new mongo docker image
        run: sudo docker pull mongo:8.0
      - name: Delete old network
        run: sudo docker network rm anonymous-go-nw
      - name: Create networks
        run: sudo docker network create anonymous-go-nw
      - name: Run mongo container
        run: sudo docker run --name mongo-demo --network anonymous-go-nw -d mongo:8.0
      - name: Run docker container
        run: |
          sudo docker run --name anonymous-go \
          --network anonymous-go-nw \
          -p 3000:3000 \
          -e MONGODB_URI="mongodb://mongo-demo:27017" \
          -e JWT_SECRET="secret" \
          -e REDIS_SECRET="secret" \
          -e REDIS_ADDR="redis-11068.c261.us-east-1-4.ec2.redns.redis-cloud.com:11068" \
          -e REDIS_PASSWORD="DM4iBq2pTYNQkweBZmwqGKyDYrj872M8" \
          -e PORT=3000 \
          -d dhruvnakum/anonymous-go:latest

Let me explain what's happening here:

• First, we named the workflow Deploy to Go Application.

• To be safe, I created a new branch called ec2 and pushed all the code to it. We want the workflow to run automatically whenever this branch is updated. Here's how we do it:

on:
  push:
    branches: 
      - ec2

• We have two jobs: Build and Deploy.

• In the Build job, we download our project's code, build a Docker image from it, and push it to Docker Hub.

• In the Deploy job, we pull the Docker image of our project and the Mongo image from Docker Hub, run them inside one container as we did before, and finally, we run that container.

Build Job

• We want this job to run on the latest Ubuntu operating system.

jobs:
  build:
    runs-on: ubuntu-latest

• Now we want to download our project’s code from GitHub inside this OS, so we write this:

name: Checkout source code
uses: actions/checkout@v2

• Since we have environment variables in our code, we need to make them available here. To do this, we must add these secrets in GitHub Action Secrets. Go to the project's settings, and under secrets and variables, add these secrets.

• Now, to get this from there, we create .env inside this OS and push everything:

- name: Create .env file
  run: echo "PORT=${{ secrets.PORT }}" >> .env

• Now that we have all the secrets. We Login to the docker:

- name: Login to docker hub
  run: sudo docker login -u ${{ secrets.DOCKER_USERNAME }} -p ${{ secrets.DOCKER_PASSWORD }}

• Then we build the docker image of our project:

- name: Build docker image
  run: sudo docker build -t dhruvnakum/anonymous-go .

• Finally, we push the image to the docker hub:

- name: Push image to docker hub
  run: sudo docker push dhruvnakum/anonymous-go:latest

Deploy Job

• In this job, everything we write will run on the AWS EC2 instance provided by GitHub.

• We want this job to start after the build job finishes successfully.

deploy:
    needs: build

• In this job, we first want to delete all the old containers and pull the latest ones. Once that is done, we run the container using docker run command.

deploy:
    needs: build
    runs-on: self-hosted
    steps:
      - name: Delete old anonymous-go container
        run: sudo docker rm -f anonymous-go  
      - name: Pull new anonymous-go docker image
        run: sudo docker pull dhruvnakum/anonymous-go
      - name: Delete old mongo container
        run: sudo docker rm -f mongo-demo
      - name: Pull new mongo docker image
        run: sudo docker pull mongo:8.0
      - name: Delete old network
        run: sudo docker network rm anonymous-go-nw
      - name: Create networks
        run: sudo docker network create anonymous-go-nw
      - name: Run mongo container
        run: sudo docker run --name mongo-demo --network anonymous-go-nw -d mongo:8.0
      - name: Run docker container
        run: |
          sudo docker run --name anonymous-go \
          --network anonymous-go-nw \
          -p 3000:3000 \
          -e MONGODB_URI="mongodb://mongo-demo:27017" \
          -e JWT_SECRET="secret" \
          -e REDIS_SECRET="secret" \
          -e REDIS_ADDR="redis-11068.c261.us-east-1-4.ec2.redns.redis-cloud.com:11068" \
          -e REDIS_PASSWORD="DM4iBq2pTYNQkweBZmwqGKyDYrj872M8" \
          -e PORT=3000 \
          -d dhruvnakum/anonymous-go:latest

Before we move ahead I forgot to change the URL in the main from localhost to 0.0.0.0 . So make sure you have this inside main.go at the end.

log.Fatal(r.Run("0.0.0.0:" + os.Getenv("PORT")))

Setting Up AWS EC2 Instance

• Before deploying the project, we need to set up an EC2 instance.

• To do this, log in to the AWS Management Console and go to the EC2 dashboard.

• Use this link: https://aws.amazon.com/ec2/

• It might ask for your card details, but don't worry; they won't charge you unless you exceed the limit, which you won't be charged for for this purpose.

• Search for EC2 and click on it.

  

• Now click on the “Launch Instance.”

• Now, give the instance name and select the Ubuntu image

• Also, select Key pair. If you don’t have one. You can create it by clicking on the Create new key pair button.

• Once it is done, click on the Launch Instance button on the right

• Since our app is running on port 3000, go to the Security Group settings and allow inbound traffic on port 3000 to make sure the application is accessible.

• To do that, click on the Instance ID

• Then click on the Security tab and there click on Security Group

• Click on Edit inbound rules and then add custom TCP with Port range 3000 as shown below and save the rules.

• Now, we are ready to run the instance.

Setting up Docker inside EC2 Instance,

• Our instance is ready. First, we need to make sure Docker is installed. To do this, let's connect to the instance. Click the Connect button after selecting the instance.

• Once you do that you will see a window like this:

• To confirm whether docker is there or not, run docker command, you will most likely see this:

  

• Now, to install everything, run the below commands

sudo apt-get update && sudo apt-get install docker.io -y && sudo systemct| start docker && sudo chmod 666 /var/run/docker.sock && sudo systemcti enable docker && docker --version

• Once it is done, run docker version to see if it’s installed correctly.

Register a Self-Hosted GitHub Runner

• A self-hosted runner is simply a computer or server that you own (like an AWS EC2 instance) that runs GitHub Actions instead of using GitHub’s default machines.

• To set up, go to the project settings, and inside Actions, you will see this runner tab.

• Click on `New self-hosted runner:

• Select Linux

• And run all the commands you see there inside your EC2.

• Once you run the last step ./run.sh command, you will see a runner with the status idle :

• But there is one problem, that runner does not run on detached mode in our instance, If you kill it by pressing CTRL+C, you wont be able to run it.

• To resolve this issue, just run the below command

sudo ./svc.sh install

sudo ./svc.sh start

• Now, your running is running in the background, and you can perform other operations inside your instance.

• Finally, we are done with it. Now, we can test our application.

Testing Your Application

• To test, commit to the ec2 branch that we created. Once you do that, Go to the Actions tab, and there you will see both build deploy services are running.

• Once it is done, you will see both marked as green ticks, which means that everything went well and the docker is successfully up inside the ec2.

• To test, go to the instance we created and find the Public IPv4 address. Copy it, paste it into Postman, and run any endpoint in your application.

• As you can see it’s working now.

• Phew! I know that seems like a lot at first. But once you follow these steps, you'll get comfortable with it.

Wrapping Up

• If you've made it to the end of this guide, you truly deserve a big thank you. So, thank you for sticking with it and reading all the way through. I hope you've gained a lot of valuable insights from this single blog post, covering everything from creating a GitHub Workflow to understanding Actions, setting up a Runner, and working with AWS EC2.

• Keep diving deeper into these topics and explore other related concepts, as they are essential for cloud development in today's fast-paced tech world. Mastering these skills will undoubtedly enhance your ability to build and deploy applications efficiently.

• I look forward to sharing more knowledge with you in the next blog post! Until then, keep learning and experimenting with new technologies.

Reference: https://www.youtube.com/watch?v=sSAWMr_-Co4&t=409s

• Connect with me on LinkedIn, Github, and Twitter.

• Hey, Gophers! I hope you're doing well. I'm back with Part 2 of the series. In the last blog, we learned about Docker and how it makes development easier with Containers.

• We also created two containers: MongoDB and Redis.

• Now that we have both running in Docker containers, it's time to build a custom image for a Go application. In this part, we'll learn about the Dockerfile, docker-compose file, and how to dockerize the application. By the end of this tutorial, you'll have a fully functional dockerized application that you can pull onto your machine and use without installing any dependencies.

• Before that, I'll give you a quick overview of the project. I won't go into too much detail to keep the blog short. I'll just explain what the project is about, and then we'll get started with the main part. Let's do that!

Project Overview

• This is a simple project, but using all this tech together will be really fun. The project is about making a post anonymously, kind of like a tiny version of Reddit. It's just for learning, so it's okay if it's not very big. We'll only ask for a username and password, nothing else. The user signs up and logs in with these two things.

• Users can create posts, and they can also like and comment on other people's posts.

• Here's the mind map of what we'll use in our Go app and which APIs we'll implement.

• By the way, the project's name is Anonymous-GO.

Note: This blog requires basic knowledge of Go and how to work with APIs. I won't explain everything from the beginning. If you want to start from the basics, I already have a series on that. You might want to check it out first:

https://dhruvnakum.xyz/from-zero-to-go-hero-learn-go-programming-with-me-part-1

• Here is the final project if you want to go through.

Setting Up Go Project

Project Structure

• We will structure our project as follows:

• In config/, we have functions for setting up Redis.

• controllers/ contains all the main operations like auth, post, like, and comment. This is where we define all the route handlers.

• In database/, we set up MongoDB and have the User and Post collections.

• In middleware/, we define handlers for accessing private routes.

• In models/, we define different models like User, Post, Comment, and Like.

• In utils/, we have helper functions for JWT authentication, and for hashing and verifying passwords.

• We also have the .env file for storing all the project secrets.

• We will discuss docker-compose.yml, Dockerfile, and Dockerfile.dev in the next section.

Running MongoDB Containers

• If you read the previous blog, you might have a MongoDB container ready inside your docker. Let’s first run that.

  

• Before we start, we will use a MongoDB container for database storage and Redis Cloud for caching.

• To run the project, we first need to start the MongoDB container to connect the database with the application.

• When you run the project, you will see that both MongoDB and Redis are connected.

  

• Let’s now dockerize this application.

Dockerize GO Application

• Before we start, let’s recap things we learned about Docker in our previous blog.

What is Docker Image?

• As we saw in our previous blog, Docker Image is a package/bundle that is used as a template to create containers.

• It contains everything needed to run an application. The docker image contains the application code. In our case, it is our Go code.

• It also contains dependencies that we use in our application, like any libraries, frameworks, etc.

• Contains all the tools and settings.

• But the thing is Docker image cannot run on its own. As it’s just a template. To run the Image, we need to create a Container, just like Class and Object, Class is nothing without an Object, right!?

What is Docker Container?

• As Docker Image cannot run on its own, it requires Containers. Containers are basically running instances of a Docker Image.

• You can think Docker Image as Class and Docker Containers as Objects. Class is nothing but a template and we can make use of class by creating it’s objects.

• Every containers run in its isolated environment. And this containers runs exactly the same on every machine. Whether it’s Windows, Mac, or Linux.

• As you can create multiple objects from a defined class. You can create multiple containers from the same image.

• Now you might be asking that, Okay, I understand Image and Containers now. But how do we create an Image of our application right?

• Well that’s where Dockerfile comes into picture.

Dockerfile

• A Dockerfile is nothing but a set of instructions that tells Docker how to build a Docker Image.

• Let’s see how we can write a docker file

FROM golang:1.23-alpine AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -o main .

FROM alpine:latest
COPY --from=builder /app/main /main
COPY .env .
EXPOSE 3000
ENTRYPOINT ["/main"]

• Let’s understand what’s happening here

• First of all, we want Docker to compile our application, right? How do we do that? Well, Docker Hub already has the docker compiler image for us. It has all the tools, like go build go mod, etc.

• So, we are using this docker image as our base. You might ask why we are using this image only and not the other one. The simple answer to that is it’s lightweight.

  

FROM golang:1.23-alpine

• The line above does that task. We also added AS builder after it. This simply labels this stage as "builder," so we can refer to it later.

• You might ask why we need builder a stage.

  • Well, we don’t want the final application to have every Go tool. It’s unnecessary. So, instead of storing unnecessary files in the final image, we directly build the app in this stage and then copy only the final compiled binary file into a new lightweight image.

  • This keeps the docker image small, fast, and secure.

WORKDIR /app

• Here, we are creating a working directory inside the container. So, when we say /app, this means we are creating an app folder inside the container.

• You might ask why we are creating app directory, can’t we just push all the things directly. Well, you can, but it’s not recommended. Why? Because If we don’t do this, every time we need to execute cd command manually to go inside /app folder

COPY go.mod go.sum ./

• The COPY instruction is used to copy the files from your local machine to docker containers.

• It will copy it inside /app as we previously set the current working directory as /app

• You might ask why we are copying this. well, otherwise, our application won’t run. These files contain all the dependencies required by our application.

RUN go mod download

• This command is used to download and cache all dependencies specified in go.mod

COPY . .

• Now that we are done with the basic setup, we need to copy all the files and folders of our application into a docker container.

• The . (dot) here refers to the current directory on your local system and second . (dot) refers to the current working directory inside the container.

RUN CGO_ENABLED=0 GOOS=linux go build -o main .

• Now that we have copied our files, folders, and dependencies, we are ready to compile the application inside the container. This is what the above RUN instruction does.

• go build -o main . tells Go to compile the application and generate an executable file named main.

• The . at the end means compiling all Go files in the current directory (/app). which is main.go in this case.

• CGO allows Go programs to use C libraries, but it also makes the binary dependent on system libraries. And we don’t want that, so we are disabling it by assigning 0 value.

• GOOS=linux is used to build the application for Linux. As we are using Alpine Linux image, we need to do that to make it lightweight for the system.

FROM alpine:latest

• Now that we have compiled the Go application in the previous builder stage. We now need to run the final binary which is created inside container.

• To do that, we are using alpine container to create a lightweight container. This removes all the unnecessary libraries and tools.

• In the previous stage (golang:1.23-alpine AS builder), we compiled the Go application.

• Now, we only need to run the final binary, so we switch to a clean, minimal image.

So what exactly happens here:

• Docker starts with a fresh Alpine Linux image.

• No Go compiler or extra tools are installed, just a clean OS.

• The final container only contains what's needed to run the application.

Why Not Just Use golang:1.23-alpine for Everything?

• That’s a good question. It’s because The Go compiler and tools are only needed for building/compiling the app, not for running it. Keeping them in the final image wastes space and increases security risks. Using Alpine separately ensures a much smaller, optimized final container.

COPY --from=builder /app/main /main

• This line copies the compiled Go application from the builder stage into the final Alpine-based image.

• /app/main this is where the Go binary was built in the builder stage

• And /main this is the destination inside the Alpine container

EXPOSE 3000

• With this instruction, we are telling Docker that this container is expecting traffic at port 3000.

ENTRYPOINT ["/main"]

• ENTRYPOINT ["/main"] tells Docker to execute the compiled Go binary (/main) inside the container.

• So now, when you run the container, it automatically runs the /main binary.

How does this overall process work?

First Stage (builder)

• Uses golang:1.23-alpine

• Builds the Go application (main) inside /app

• This stage has all the necessary tools (Go compiler, dependencies)

Final Stage (alpine:latest)

• Starts with a clean Alpine Linux image

• Copies only the main binary from /app/main into /main

• The container is now ready to run the application

Does that mean we are using two Containers?

• No, we are not creating two containers. Instead, we are creating two separate environments during the build process

• Docker processes both stages during the image build phase.

• It uses the first stage (builder) to compile the application.

• It then discards the first stage and only keeps the final image from the second stage.

• The container is created from this final, lightweight image.

docker-compose

• Now our Dockerfile is ready. But right now, our app is only containerized. Remember, we are also using a MongoDB image in our app. If you try to run the app right now, it will throw an error because the app cannot connect to MongoDB.

Why Do We Need docker-compose.yml?

• Our app and MongoDB are running in separate containers.

• The app depends on MongoDB to function, but without a shared network, they cannot communicate.

• Running both services manually using docker run commands is inefficient.

• This is where Docker Compose comes into play.

• Docker Compose allows us to define and run multiple containers as a single service, ensuring that all required services (our app and MongoDB) start together, communicate seamlessly, and work as a unit.

• With a docker-compose.yml file, we can:

  • Define multiple services (e.g., app and mongo) in a single configuration.

  • Ensure MongoDB starts before the app using depends_on.

  • Use a shared network so the app can communicate with MongoDB by referring to it as mongo instead of localhost.

  • Set up environment variables for both services.

  • Use volumes to persist MongoDB data, preventing data loss when containers restart.

• Now let’s write docker-compose for our app:

services:
  app:
    build:
      context: .
    ports:
      - "3000:3000"
    depends_on:
      - mongo
    environment:
      - MONGODB_URI=mongodb://admin:admin@mongo:27017/
      - JWT_SECRET=secret
      - REDIS_SECRET=secret
      - REDIS_ADDR=redis-11068.c261.us-east-1-4.ec2.redns.redis-cloud.com:11068
      - REDIS_USERNAME=default
      - REDIS_PASSWORD=DM4iBq2pTYNQkweBZmwqGKyDYrj872M8

  mongo:
    image: mongo:8.0
    environment:
      MONGO_INITDB_ROOT_USERNAME: admin
      MONGO_INITDB_ROOT_PASSWORD: admin
    ports:
      - "27017:27017"
    volumes:
      - mongodata:/data/db

volumes:
  mongodata:
    driver: local

• So, as we know, we need two services: app and Mongo

• And that’s why we are defining those inside services

1. app Service:

   • This service is used to build the app using Docekerfile.

   • So, we are defining its context with the current directory by specifying . (dot).

   • Then, we want to map the container port to the local system’s port (called port mapping). We do that inside port tag.

   • Also, we want mongo-db to run before our app starts. To achieve that, we have to use depends_on tag and specify mongo image.

   • Also, if you look at the project, we have specified many environmental variables. And so we want this app service to use that. So, we specify all the environment variables inside ennvironment tag.

2. mongo Service:

   • We first need to pull up the Mongo image. We are doing it by specifying it inside the image tag.

   • As we did in our app, we are specifying env variables inside the environment tag.

   • Then, we need to expose port 27017 for database access.

   • Also, we want to keep the data in a persistent volume. So, we are using volumes

Running docker-compose

• Now that we are done with the docker-compose file, we can now run the container using the following command.

docker compose up -d

• It will start the container inside the docker.

• As you can see, our entire application is now dockerized.

• Now, let’s test our API to see if it’s working or not.

• Let’s create a user with a signup API

• Now, let’s test the login functionality with Redis caching.

• As you can see, Redis-session is added to cookies for accessing private routes. Let’s also test private routes to make sure they're working.

• 

  And yes, it is working. And now, if you try to access private routes after logging out. It will not work. Let’s check that.

  

  - We logged out.

• 

  As you can see, it’s throwing an error.

What’s Next?

• Currently, our APIs are working locally. But we want developers to use these APIs, right? To do that, we need to deploy this.

• Now, there are many deployment options available in the market.

• The most popular one is AWS.

• In the next part, we are going to push our custom image to Docker Hub, set up GitHub actions, and Upload and Run this docker container inside the AWS EC2 instance.

• It’s gonna be really good learning for anyone who is not familiar with AWS deployment. Because I learned a lot.

• Let’s meet in the next one, until then…

• Hey, Gophers! Thanks for stopping by and taking the time to read my blog. If you’ve been following my previous blogs, you might know that I recently started writing about Go. In my last article, we explored JWT authentication using the JWT and Fiber packages in Go.

• After covering the basics, I learned more advanced topics, such as working with databases, handling cookies and sessions, deploying Go applications on the cloud, using Docker, containerizing Go applications, etc.

• And to put all these concepts into practice, I built a project that combined them all. Honestly, it was quite a challenge since AWS, Dockerization, Redis, and GitHub Actions were new to me. But after overcoming everything, I’m excited to share everything I’ve learned with you.

• In this series, I’ll break down each concept in a simple and easy-to-understand way. If anything is unclear, feel free to ask in the comments!

• By the end of this series, you’ll have a fully deployed, containerized project running in the cloud. If you're a beginner, this project will be a great addition to your resume, as we’ll be implementing many industry standard practices used in real-world applications.

• We first need to learn a very important concept/tool called Docker. Because nowadays everyone is using it.

• So, let’s first start with Docker and understand what Docker is. Why do we and people in the software industry use it? And What problem does it resolve?

Why do we need Docker?

• Before diving into what Docker is, let's first understand the issues it helps to resolve.

• Imagine an organization with hundreds of developers, each working on different machines with different operating systems and configurations. Now, consider a project where a team of 50 developers is collaborating on the same GitHub repository.

• As the team grows, new developers are hired and provided with new systems to work on. To get started, they need to clone the project from the repository and set up the development environment. Let's take two developers as an example:

  • Developer 1 – A current employee who has already set up the project environment.

  • Developer 2 – A new employee who has just joined and needs to set up the project from scratch.

• Developer 1, who has been working on the project from the beginning, has already set up all the necessary environments. Since this is a Go project, he runs the server locally using the following versions of key dependencies:

  

• Now, the real challenge begins when Developer 2 (a new employee) joins the team and needs to run the project on his system.

  Challenges Faced by the New Developer

  1. Manual Setup Hassle

     • The new developer first clones the project from the repository.

     • He then needs to install multiple dependencies manually (Go, MongoDB, Redis, etc.).

     • In real-world projects, there are even more dependencies to install, making the setup tedious and error-prone.

  2. Operating System Differences

     • Developer 1 might be using Windows, while Developer 2 has a Mac.

     • Some project commands may work on Windows but not on Mac.

     • Certain OS-specific compatibility issues might arise during the setup.

  3. Version Mismatches

     • Developer 2 might install the latest versions of dependencies, which could be incompatible with the project.

     • Some dependencies may have breaking changes, causing unexpected issues.

  4. Cloud Deployment Challenges

     • Eventually, the project will need to be deployed on the Cloud.

     • The cloud server might have a different environment, requiring another round of installations and conflict resolution.

     • Even after setting everything up, there’s no guarantee that the project will work as expected.

  5. Scalability Issues

     • Today, it’s just one new developer-facing these challenges, but as the team grows, the setup process will have to be repeated for every new team member.

     • Constant communication and troubleshooting with each developer become inefficient and time-consuming.

• This is where Docker comes into play. It provides a way to package the entire environment, ensuring that everyone (developers and cloud servers) runs the same setup without manual installations or version mismatches.

• In the next section, we’ll dive into what Docker is, why it is useful, and how it solves these problems.

How does Docker resolve this problem?

Docker Container

• Docker is a platform that allows developers to create, deploy, and run applications inside containers.

• What is a Container?

  • You can think of a container as a box that holds everything needed to run an application: its code, runtime, dependencies (like Go, MongoDB, Redis), and any required configurations.

• With Docker, we can create a container that includes all the necessary tools, packages, and dependencies. This container ensures that:

  • Every developer gets the exact same versions of Go, MongoDB, Redis, and other dependencies.

  • The application runs identically on every system.

  • The container is lightweight, meaning it doesn’t take up unnecessary resources like a full virtual machine would.

• As you can see, this resolves the very big issue of installing and setting up the project environment again and again, and we can save our developers from the manual errors that they can make, as seen above.

To conclude, With Docker, we don’t have to install things like MongoDB or Redis manually again and again. Instead, we can run them inside containers and start using them instantly!

• Okay, so now that we understand containers, there is one more term that we need to understand, which is Images.

Docker Images

• A Docker image is like a recipe that contains all the necessary ingredients and instructions to prepare a dish. However, instead of food, a Docker image contains everything needed to set up and run a software application.

  Understanding Docker Images with an Example

  • Imagine you have a laptop. Without an operating system (Windows, Mac, or Linux), your laptop is just a piece of hardware that can’t do much. Similarly, a Docker container needs a Docker image to function.

  • Think of a Docker image as the "operating system" for a container. It includes:

    • The application code

    • All the dependencies (like Go, MongoDB, Redis, etc.)

    • The runtime environment

    • Any necessary configurations

  • Once a Docker image is created, it serves as a blueprint to generate multiple containers. Each container runs independently but follows the same instructions defined in the image, just like you can cook the same dish multiple times using the same recipe.

  • With Docker images, developers can ensure consistency across different environments, making application deployment seamless and reliable.

• I hope everything is clear now.

• If not, then don’t worry; we are now going to do some hands-on exercises and see everything that we have learned so far in action.

Docker Installation

Docker Desktop

• To run Docker containers on your machine, you need to install the Docker Desktop application.

  You can download it from the official website:

• Docker Desktop Installation

• Or simply search “Docker Desktop install” online, and it will take you to the installation page.

• Docker Desktop is a GUI (Graphical User Interface) that makes it easy to:

  • Manage containers

  • View and control Docker images

  • Monitor volumes and networks

  • Configure Docker settings

• It provides a user-friendly way to interact with Docker without using only command-line tools, making container management more accessible.

• Once you are done with the installation, open the terminal of your choice and run docker command

• It should show a similar result. This tells us that docker is successfully installed in our system. You can also verify using.

docker --version

• Now open your Docker desktop.

• If you open Docker Desktop, you’ll notice several sections on the left-hand side:

  • Containers – Shows running and stopped containers.

  • Images – Lists downloaded and built Docker images.

  • Volumes – Manages persistent data storage for containers.

  • Builds – Tracks built images and processes.

• We've already covered Containers and Images, and that’s all we need to get started!

• Okay, so for our upcoming project, we’ll need MongoDB and Redis as dependencies. Instead of manually installing them and setting up everything from scratch (which is boring and time-consuming 😩), we can use Docker to simplify the process.

• With Docker, we can:

  • Pull pre-built images of MongoDB and Redis.

  • Run them as containers.

  • Avoid dependency conflicts or setup issues.

• This means no more frustrating installations, we just run a command, and everything works!

• Before that, let’s understand what Docker hub is.

Docker Hub

• You might be wondering when I say we need to pull Docker images for MongoDB and Redis, where these images actually come from. It comes from Docker Hub. Docker hub is similar to Github but for docker images.

• It’s an online repository where developers can find and share Docker images.

• You can also create your custom image on your system and push it to the docker hub for others to use. We are going to create a custom image for our Go application and push it to the docker hub in the future.

• What we, as developers, have to do is just pull these images using the command docker pull, and we are all set.

Running MongoDB in a Docker Container

• Instead of manually installing MongoDB, we can use Docker to set it up in seconds with the following command:

docker run -d --name mongo-demo \
-e MONGO_INITDB_ROOT_USERNAME=admin \
-e MONGO_INITDB_ROOT_PASSWORD=admin \
-p 27017:27017
mongo:8.0

• Let’s understand this command:

  • docker run -d : It runs this container in a detached mode. This means that if we do not provide this tag, our terminal will be in use, and we won’t be able to perform any other operations. So, we need to run this container in the background

  • —name mongo-demo : We are naming our container as ‘mongo-demo’

  • Now, for security purposes, we are setting the Username and Password for our database using -e MONGO_INITDB_ROOT_USERNAME and -e MONGO_INITDB_ROOT_PASSWORD

  • We are also mentioning the port We want Mongo to run locally in our system.

PORT MAPPING

• When we run MongoDB inside a Docker container, it operates in an isolated environment. This means it does notautomatically connect to our local machine.

• If we try to access MongoDB from outside the container, it won’t work unless we explicitly expose its port.

• This is where Port Mapping comes in!

Here, -p 27017:27017 means:

• The first 27017 (before the :) is the port on your local machine (host).

• The second 27017 (after the :) is the port inside the container.

By default, services running inside a Docker container are not accessible from outside.

• MongoDB inside the container is running on port 27017.

• But our system (local machine) doesn’t know that unless we explicitly expose it.

• By mapping MongoDB’s internal port (27017) to our system’s port (27017), we make it accessible.

• At the end, we are specifying which image to pull, and that is mongo:8.0

• If you search Mongo in the docker hub, you will find its image shown below.

  

• What that above command will do is :

  • Pulls the MongoDB 8.0 image from Docker Hub (if not already downloaded).

  • Starts a MongoDB container with a default admin username and password.

  • Allows us to interact with MongoDB just like we would on a locally installed version, but without the manual setup!

• Once it is done, you will see this container running in your Docker Desktop.

• To check if it’s running or not, you can run the below command

docker ps

• This will list all the running containers

• Now, let’s run the Redis container

Running Redis in Docker Container

• Just like MongoDB, we can set up Redis instantly using Docker without manually installing anything. Use the following command:

docker run -d --name redis-demo redis:latest

• This will pull the latest Redis image (if not already available locally) and then start a Redis container that runs in the background.

  

• Again, to verify whether it’s running or not, let’s run docker ps

• As you can see, our Mongo and Redis containers are working pretty well.

Now that we have MongoDB and Redis running inside containers, we don’t need to install them manually on our system. Anytime we need them, we can just start the containers, and everything will work instantly.

Wrapping Up

• With Docker, we’ve reduced the hassle of manual installations and ensured a consistent and error-free development environment. No more dependency issues or version conflicts, just a simple setup in seconds!

What’s Next?

• This blog serves as the foundation for the next steps in our series. Now that you understand docker and how it works. We are ready to containerize our Go application.

• Stay tuned for the next part! 🚀 Until then.

• Connect with me on LinkedIn, Github, and Twitter.

• After the last series on Go, I've been learning some new ideas, like Secure Authentication. I discovered JWT authentication and how it helps secure routes.

• In this article, I'll explain how we can use JWT to authenticate users and secure routes.

• So, let's jump right in.

Creating a Base Folder Structure

• We'll start by setting up a project structure to clearly see how everything fits together.

• First, create a directory and open your favorite code editor. I prefer VS Code.

bmkdir go-auth
cd go-auth
code .

• So this is our project structure :

cmd/server/main.go

• This is the entry point of the application

controllers/auth_controller.go

• This is where we place our controllers/handlers. In our case, it will be auth_controller. Here we will implement all the functions related to authentication like Register, Login, and Logout.

database/db.go

• In this database, setup is done and the connection is made. We will be using MySQL as our database.

middleware/middleware.go

• Middleware is simply a function that wraps around our HTTP handlers to add extra features. For example, once a user is logged in, we use middleware to secure other routes like /dashboard and /cart. We will learn about middleware in this blog.

models/user.go

• The model represents the data structure of the application’s data. So for example, if you are building an application related to a library, models can be, a book, author, etc. In this project which we are building, we need user a model as we are dealing with the user’s authentication.

routes/routes.go

• We define all the HTTP routes here. We are using fiber a package for managing the application routes. If you are familiar with express in Node.js then this package is inspired by it and provides flexible ways and functionalities to manage routes.

types/types.go

• In this, we define our own custom struct types which we require in our application.

.env

• This is an environmental file where we put all the secure stuff.

Load .env File

• First of all, let’s load all the environmental variables in our application.

• For that, we are using a package called gotdotenv. This helps us achieve this.

go get github.com/joho/godotenv

• Let’s load this env file inside our application. Go to main.go and paste below code

import "github.com/joho/godotenv"

func main() {
    err := godotenv.Load()
    if err != nil {
        log.Fatal("Error loading .env file")
    }
}

• We place it at the top of the main function because we need all the variables loaded before the server starts.

Connect Database

• We are using MySQL to work with the database. I was going through some queries and everything as It’s been a while since I wrote any. And then I came across the term ORM (Object-Relation Mapping). And I found it super convenient as we can use objects oriented approach to execute the queries.

• So an ORM is a programming technique that allows us to interact with a relational database like MySQL using object-oriented programming principles, instead of writing raw SQL queries. It serves as a bridge between the object-oriented language and the database, mapping database tables to objects in the application.

• And researching more about it I found this package called gorm for Go. So we will be using this to work with our database.

• Let’s install it and also we need to install the mysql driver for it separately.

go get -u gorm.io/gorm
go get gorm.io/driver/mysql

• Let’s now create Connect function and write the logic to connect our application with the database.

package database

import (
    "fmt"
    "os"

    "gorm.io/driver/mysql"
    "gorm.io/gorm"
)

func Connect() error {
    dsn := fmt.Sprintf("%s:%s@tcp(127.0.0.1:3306)/go_auth?charset=utf8mb4&parseTime=True&loc=Local", os.Getenv("DB_USERNAME"), os.Getenv("DB_PASSWORD"))
}

• Inside Connect() we first initilized dsn (Data Source Name). dsn is nothing but a string that provides all the information to connect the database. It includes information like database type, server location, username password, database name, etc.

• Here we used fmt.Sprintf which returns a formatted string. We used two string %s placeholders for username and password. And we are getting those values from the .env file.

• Let’s first add those values to our .env

// .env
DB_USERNAME=root
DB_PASSWORD=root123

func Connect() error {
    dsn := fmt.Sprintf("%s:%s@tcp(127.0.0.1:3306)/go_auth?charset=utf8mb4&parseTime=True&loc=Local", os.Getenv("DB_USERNAME"), os.Getenv("DB_PASSWORD"))
    db, err := gorm.Open(mysql.Open(dsn), &gorm.Config{})
    if err != nil {
        return err
    }
}

• After that, we open the connection to the MySQL database using Gorm's Open() function. We use the mysql driver's Open function to connect with our dsn string. The second parameter is for gorm configuration if you have any.

• This gorm.Open function returns two things: db instance and error. So what we need to do is, first we need to create a global gorm.DB variable that our application can use from anywhere inside the project.

var DB *gorm.DB

func Connect() error {
    // ...
}

• And that’s why we create var DB *gorm.DB .It’s a pointer because we need whatever changes we made in this variable, it reflects the original one which is db inside the Connect function.

func Connect() error {
    dsn := fmt.Sprintf("%s:%s@tcp(127.0.0.1:3306)/go_auth?charset=utf8mb4&parseTime=True&loc=Local", os.Getenv("DB_USERNAME"), os.Getenv("DB_PASSWORD"))
    db, err := gorm.Open(mysql.Open(dsn), &gorm.Config{})
    if err != nil {
        return err
    }

    log.Info("Database connected successfully")
    DB = db

    db.AutoMigrate(&models.User{})
    return nil
}

• And that is why we assigned db to global DB variable.

• Then we have this db.AutoMigrate(&User{}) function. What this AutoMirgrate function does is, it creates the same table with all the values defined inside this model User inside the database.

• This means that GORM will ensure that the table corresponding to the User model in the database matches the structure defined in the model, creating or updating columns as necessary.

• And so when you run the application for the first time, it will create a table for you automatically.

• We haven’t created our User model yet so let’s create it first.

User Model

package models

type User struct {
    Id       
    Name    
    Email    
    Password
}

• Here we’ve created a simple User struct, which contains basic information like ID, Name Email, and Password.

• Now that we have the database file ready we can use the Connect function inside our main function to connect the database.

func main() {
    err := godotenv.Load()
    if err != nil {
        log.Fatal("Error loading .env file")
    }

    if err = database.Connect(); err != nil {
        log.Fatalf("Could not connect to the database, %v", err)
    }
}

• Now let’s run the project to see if it’s working

go run cmd/server/main.go

Setting Up the Router

• Okay so for route management, as we discussed earlier we are using fiber package. To install run the following command inside your terminal

go get github.com/gofiber/fiber/v2

• Let’s create an instance of fiber inside main.go to use it inside our application

func main() {
    //...

    app := fiber.New()
}

• Using this app we can now create our routes.

• To make things look clean and separate we will implement all the routes inside routes.go.

// routes.go
func SetupRoutes(app *fiber.App) {
    app.Post("/api/register", controllers.Register)
    app.Post("/api/login", controllers.Login)
    app.Post("/api/logout", controllers.Logout)
}

• Here we have created SetupRoutes a function that takes a app variable of type *fiber.App.

• Inside this, we created three routes, /register, login/ and /logout. (Don’t worry if you are seeing an error saying these functions are not found. We are going to implement them in a bit)

• Now we call this function in main.go like this

func main() {
    //...

    routes.SetupRoutes(app)
}

• Now let’s implement these handler functions inside controllers.

Implementing Route Handlers

Register

// controllers/auth_controller.go
func Register(c *fiber.Ctx) error {
    var regReq types.RegisterRequest
}

• Since we receive the submitted data in an HTTP request, we need to create a custom struct type to convert all that data into a format that Go can understand.

• Let’s create a RegisterRequest struct for the Register handler

// types/types.go

type RegisterRequest struct {
    Name    
    Email   
    Password
}

• Let’s now parse the request body of the register to this type

func Register(c *fiber.Ctx) error {
    var regReq types.RegisterRequest

    if err := c.BodyParser(&regReq); err != nil {
        return c.Status(fiber.StatusInternalServerError).JSON(fiber.Map{
            "message": "invalid request format",
        })
    }
}

• Function BodyParser helps bind the body fields to a struct field. It returns an error of type ErrUnprocessableEntity which means if BodyParser fails to bind the fields it will throw this error.

• So we handled it by returning an error message “invalid request format“ with a status code 500 (StatusInternalServerError)

• Now before we create our user and send it to the database to store, we can’t just send the password as it is. We need to secure it by converting it into some hash or something.

• Go has this package called bcrypt which helps us achieve that. First, let’s install the package

go get golang.org/x/crypto/bcrypt

• bcrypt has a function GenerateFromPassword that takes a byte of password string as the first parameter and cost as the second parameter (cost determines the computational complexity of the hashing process. It is a measure of how long it takes to hash a password, directly affecting the time it takes to verify the hash later)

• Let’s hash our password and then create a user object

func Register(c *fiber.Ctx) error {
    //...

    password, _ := bcrypt.GenerateFromPassword([]byte(regReq.Password), 10)

    user := models.User{
        Name:     regReq.Name,
        Email:    regReq.Email,
        Password: password,
    }
}

• This bcrypt function returns two values: encrypted password and error.

• And after that, we can construct our user model by using regReq struct and encrypted password.

• Now before we send this to the database, we need to first check whether this user already exists in the database or not.

func Register(c *fiber.Ctx) error {
    // ...   

    database.DB.Where("email = ?", user.Email).First(&user)
}

• We can do this using the Where function of the MySQL database. Since each email is unique in our application, we wrote a query that returns a user if they exist.

• Now we check if we found anything or not like this:

func Register(c *fiber.Ctx) error {
    // ...   

    database.DB.Where("email = ?", user.Email).First(&user)

    if user.Id != 0 {
        return c.Status(fiber.StatusConflict).JSON(fiber.Map{
            "message": "user already exists",
        })
    }
}

• If user.Id is not 0, it means we found a user, so we return an error message saying "user already exists."

• If we didn't find anything, we create a new user in the database using the Create() function of DB.

func Register(c *fiber.Ctx) error {
    // ...

    database.DB.Create(&user)
}

• And once everything is done, we send a message back to the API response saying "user created successfully" along with the user information.

func Register(c *fiber.Ctx) error {
    // ...

    return c.JSON(fiber.Map{
            "message": "user created successfully",
            "data":    user,
        })
}

• That was pretty straightforward, wasn’t it?

Running the Server

• To run the server, we use the app's Listen function and specify the port number like this:

// main.go

func main() {
    //...

    log.Println("Starting server at: ", 3000)
    err = app.Listen(":3000")
    if err != nil {
        log.Fatalln("Error Starting Server at: ", 3000)
    }
}

• Now let’s run the app

Login

// controllers/auth_controller.go

func Login(c *fiber.Ctx) error {
    var loginReq types.LoginRequest

    if err := c.BodyParser(&loginReq); err != nil {
        return err
    }

    if err := validate.Struct(&loginReq); err != nil {
        return c.Status(fiber.StatusBadRequest).JSON(fiber.Map{
            "message": "Validation failed",
            "errors":  err.Error(),
        })
    }
}

• This part of the Login remains the same as the Register. We have created a LoginRequest structure inside types.go

// types/types.go

package types

...

type LoginRequest struct {
    Email 
    Password
}

• Now, we need to check if the user exists in the database when the client tries to log in with their credentials.

• To do this, we will use the Where() function of DB to see if the email is in the database. If it exists (we check this by seeing if user.Id is not 0), we proceed with further operations otherwise, we return an error.

func Login(c *fiber.Ctx) error {    
     // ...

    var user models.User

    database.DB.Where("email = ?", loginReq.Email).First(&user)

    if user.Id == 0 {
        c.Status(fiber.StatusNotFound)
        return c.JSON(fiber.Map{
            "message": "user not found",
        })
    }
}

• If the user exists, we compare the incoming password with the one in the database. If they match, the credentials are correct. If they don't match, we return an error saying "incorrect password."

func Login(c *fiber.Ctx) error {    
     // ...
     if err := bcrypt.CompareHashAndPassword(user.Password, []byte(loginReq.Password)); err != nil {
        c.Status(fiber.StatusBadRequest)
        return c.JSON(fiber.Map{
            "message": "incorrect password",
        })
    }
}

• bcrypt has a function called CompareHashAndPassword that checks the incoming password against the one stored in the database.

• After that, we just send a JSON response with a success message and the data.

func Login(c *fiber.Ctx) error {    
     // ...
    return c.JSON(fiber.Map{
        "message": "success",
        "data":    loginReq,
    })
}

• Let’s check the Login handler by running the project.

Logout

• The logout feature is very simple. On the backend, there's not much to do. The frontend should handle the logout process.

• So, when this route is accessed, we just return a success message.

func Logout(c *fiber.Ctx) error {

    return c.JSON(fiber.Map{
        "message": "Logged out successfully",
    })
}

• One problem right now here is, that even if you aren’t logged in, you can still log out. And this issue we will resolve below by using JWT.

JSON Tags

• If you see the response of Login and Register…

• We notice that the field names are not in lowercase. To convert these fields into a JSON-friendly format, we use what are called Tags in Go.

• Tags specify how struct fields should be encoded into or decoded from JSON. So, even if we have a struct field like Name, we can encode and decode it as name using an annotation.

• To achieve this, we need to make changes inside types.go and user.go.

// user.go

package models

type User struct {
    Id       uint   `json:"id"`
    Name     string `json:"name"`
    Email    string `json:"email" gorm:"unique"`
    Password []byte `json:"-"`
}

• Here, we made the email a unique key in the database. This means the same email address can't be used again.

• Another thing to note is that we added - in front of Password because we don't want this field to appear in the response for security reasons.

• Now, let's modify types.go.

package types

type RegisterRequest struct {
    Name     string `json:"name"`
    Email    string `json:"email"`
    Password string `json:"password"`
}

type LoginRequest struct {
    Email    string `json:"email"`
    Password string `json:"password"`
}

• Now if you run the project and see the response, it should be fixed

JWT Authentication

• Let’s first understand what JWT is before we dive into code.

What is JWT?

• JWT (JSON Web Token) is a way to safely send data between the client and server.

• It is a token made up of three main parts:

  • Header: This includes information like the signing method and type of token.

  • Payload: This has Claims, which are details about the user data, such as user ID and expiration time.

  • Signature: This is made by signing the header and payload with a secret or private key. It checks if the token is real.

Flow of JWT Authentication

• The client sends credentials to the server.

• If the credentials are valid, the server generates a JWT containing the user's information and signs it.

• The server sends the JWT to the client, typically as part of the HTTP response.

• The client stores the token (e.g., in local storage or an HTTP-only cookie).

• The client includes the JWT in the Authorization header (ex, Bearer <token>).

• The server validates the token’s signature and checks its claims (e.g., expiration time, roles).

• If valid, the server processes the request; otherwise, it rejects it.

Implement JWT Authentication

• To use JWT in our app, let’s first add the package

go get github.com/dgrijalva/jwt-go/v4

• In our app, when a user logs in successfully, we create a JWT token. After checking the password, we start by creating a claim to build the token.

if err := bcrypt.CompareHashAndPassword(...); err != nil {//...}

claims := jwt.NewWithClaims(jwt.SigningMethodHS256, jwt.StandardClaims{
        Issuer: strconv.Itoa(int(user.Id)),
        ExpiresAt: &jwt.Time{
            Time: time.Now().Add(time.Hour * 24),
        },
    })

• jwt provides a function called NewWithClaims to create a Claim. It takes two parameters:

  • Signing Method: We can choose from several secure hashing algorithms, but we are using HS256.

  • Claim: We include the claims we mentioned earlier, like user ID and expiry time, using the jwt.StandardClaims interface. We set the user ID in Issuer as a string and ExpiresAt with a time value, set to 24 hours. This means the token will expire after 24 hours.

  • Now that we have the claims ready, we will create a token.

token, err := claims.SignedString([]byte(os.Getenv("JWT_SECRET")))
    if err != nil {
        c.Status(fiber.StatusInternalServerError)
        return c.JSON(fiber.Map{
            "message": "could not log in",
        })
    }

• As you can see claims have a method named SignedString which we use to sign the JWT secret. We typically store the secret inside .env file.

// .env

// ... other keys
JWT_SECRET=secret

• Use a strong secret key for better security. I have used secret for testing purposes.

• And after everything’s done we get the token.

• If you remember this line:

“ A JWT token is simply a mix of signed claims and secret/private keys. I hope this explanation makes it clear.”

• And as mentioned in the Flow of JWT part we send this token to the client so that they can send it back to the backend to access secure/private routes

// auth_controller.go

func Login(c *fiber.Ctx) error {

// ...

return c.JSON(fiber.Map{
        "message": "success",
        "token":   token,
        "data":    loginReq,
    })

}

• Okay so, now that we created the JWT token. Whenever a user tries to access private routes, we need to first validate the user, whether he is logged in or not. To do that we create a middleware

Middlewares

• In Go, we create middleware that acts as a layer between the HTTP request and the handler that processes it.

• Middleware is simply a function that takes an HTTP handler and returns an HTTP handler. In our app, we need middleware to ensure the user accessing the route is authorized to access private routes before entering.

• To implement it, go to middleware/middlewares.go and paste the code below.

package middleware

import (
    "os"

    jwtware "github.com/gofiber/contrib/jwt"
    "github.com/gofiber/fiber/v2"
)

func Protected() fiber.Handler {
    return jwtware.New(jwtware.Config{
        SigningKey: jwtware.SigningKey{
            JWTAlg: "HS256",
            Key:    []byte(os.Getenv("JWT_SECRET")),
        },
        TokenLookup:  "header:Authorization",
        AuthScheme:   "Bearer",
        ErrorHandler: jwtError,
    })
}

func jwtError(c *fiber.Ctx, err error) error {
    return c.Status(fiber.StatusUnauthorized).JSON(fiber.Map{
        "message": "Unauthorized",
    })
}

• Starting with imports, we are using "github.com/gofiber/contrib/jwt" packages and gave aliases as jwtware.

• Here we have created a function Protected() that returns an HTTP handler. Inside this we are returning jwtware.New() handler which is used to create a new middleware instance for validating JWT

• Here we pass different keys:

  • SigningKey: This specifies the algorithm and key for verifying JWT’s signature.

  • TokenLookup: Defines where the middleware should look for JWT

  • AuthScheme: It defines which scheme to use in Authorization

  • ErrorHandler: It is a custom function to handle errors. So whenever an unauthorized user access any routes which require authorization, this throws an error message: “Unauthorized “

• In our case, we need two routes: Log in and Register as Public routes and Logout as Private route. How do we do that?

// routes.go

func SetupRoutes(app *fiber.App) {
    // Public routes
    app.Post("/api/register", controllers.Register)
    app.Post("/api/login", controllers.Login)

    app.Use(middleware.Protected())

    // Protected routes
    app.Post("/api/logout", controllers.Logout)
}

• As you can see, to use any middleware, fiber provides a very handy method called Use() . So what will happen now is whatever routes defined after app.Use(middleware.Protected()) this will require a JWT Token to get access.

• Let’s see it in action.

1. Open Postman

2. Try to logout directly

As you can see, Now it is throwing an error saying “Unauthorized“.

3. Now login with valid credentials

4. 

   Copy the token and paste it inside the Authorization Bearer field and then try to logout.

5. 

Code

Conclusion

• In this article, we've explored how to implement secure user authentication in a Go application using JWTs and the Fiber framework.

• We saw how to set up a project structure, connect to a MySQL database with GORM, and define essential routes and middleware.

• We've built handlers for managing user registration, login, and protected routes. This approach not only enhances the security of your application but also provides a scalable framework for future development.

• I hope you liked this article and learned something from it. If you did consider leaving a like and feedback in the comment section.

• See you in the next one, until then….

• Connect with me on LinkedIn, Github, and Twitter.

• Welcome to the last part of this Learn Go Programming series. In the previous parts, we learned lots of new stuff about Go and saw how easy and fast it is to build a backend. In the last blog, we created a Read function for getting all the books that are there in the database.

• In this part, we are going to implement the remaining operations which are:

  • Create Book

  • Update Book

  • Delete Book and

  • Get a Specific Book

• So let’s get started without wasting any more time.

Create a Book (POST)

Setting up the route

• Now to create a book, first, let’s configure the route for it.

• Go to routes.go and add a new handler called CreateBookHandler

// routes.go
func SetupRouter() *chi.Mux {
    ...
    r.Post("/book", books.CreateBookHandler) // Yet to be created inside handler.go
    ...
}

• As we need to create something inside the database, we need to use the POST method of HTTP.

Creating a Handler

• Head over to handler.go and paste the below code:

// handler.go

...
func CreateBookHandler(w http.ResponseWriter, r *http.Request) {
    var book Book
    err := json.NewDecoder(r.Body).Decode(&book)
    if err != nil {
        http.Error(w, "Failed to decode book", http.StatusInternalServerError)
        return
    }

    err = CreateBook(book) // Yet to be created inside service.go
    if err != nil {
        http.Error(w, "Failed to create book", http.StatusInternalServerError)
        return
    }

    w.WriteHeader(http.StatusCreated)
}
...

• So when we are creating something, we are most probably getting the data from the client side. In our case where we are creating a book, let’s say the admin wants to add a book to the website. What he will do is, add the details of the book in text fields on the front side and then press the Submit button.

• So when he does that, all the data that were filled by the admin comes to the backend inside a request Body in a JSON format

• So what we need to do here is, we just need to decode this data into our Go structure format.

• For that, we used this json.Decoder() function which takes json string and then Decode() takes a struct in which we need to fill the converted data.

• Then we need to call CreateBook() service of database which will add a new entry for the new book.

• Let’s write this function inside service.go

Creating a CreateBook Service

func CreateBook(book Book) error {
    _, err := database.DB.Exec("INSERT INTO books (title, author, year) VALUES (?, ?, ?)", book.Title, book.Author, book.Year)
    if err != nil {
        return err
    }
    return nil
}

• In service.go file we need to create a function called CreateBook. This function takes a struct of type Book which we will pass from the handler. And it will return an error.

• Inside the function, we are writing a create query for MySQL database. It’s self-explanatory so I am not going into details.

• In the end, we are returning an error if anything goes wrong.

• It was pretty simple, right?

• Let’s run the project and test it inside Postman.

• I already created two books for testing before so you are also seeing those in output.

Update a Book (PUT)

Setting up the Route

• Now that we have created a book. We can now create an Update functionality. Let’s go to the routes.go and create a PUT router

...
r.Put("/book/{id}", books.UpdateBookHandler)
...

• We need an id to update a particular book, right? To do that we can use a unique identifier which is id for our book. So When the user/admin updates a particular book client sends the id of it with the request to our backend.

• So if you see, after /book/ we have id inside curly braces. This is a dynamic value. When a client hits a URL with this path like, /book/1 or /book/20, we can get this book dynamic id and perform update operations.

• Let’s now create UpdateBook handler function

Creating a Handler

func UpdateBookHandler(w http.ResponseWriter, r *http.Request) {
    id, err := strconv.Atoi(r.PathValue("id"))
    if err != nil {
        http.Error(w, "Failed to convert id", http.StatusInternalServerError)
        return
    }

    var book Book
    err = json.NewDecoder(r.Body).Decode(&book)

    if err != nil {
        http.Error(w, "Failed to decode book", http.StatusInternalServerError)
        return
    }

    err = UpdateBook(id, book)
    if err != nil {
        http.Error(w, "Failed to update book", http.StatusInternalServerError)
        return
    }

    w.WriteHeader(http.StatusOK)
}

• Okay so in the first line, we are getting id using request objects PathValue function. Here we pass the exact value that we passed inside the route which is id.

• But we need to convert them id to int because our database accepts the ID of type integer.

• For that, we are using strconv package. And it has Atoi (ASCII to Integer) function which takes a string and converts it to an integer. It also returns err if it isn’t able to convert. So we are handling that error too.

• Then we decode the sent JSON data to our book struct.

• In the end, we call the UpdateBook a function where we pass the id and book struct.

• Let’s create a database service for Updating a book in the database.

Creating a UpdateBook Service

func UpdateBook(id int, book Book) error {
    _, err := database.DB.Exec("UPDATE books SET title = ?, author = ?, year = ? WHERE id = ?", book.Title, book.Author, book.Year, id)
    if err != nil {
        return err
    }
    return nil
}

• Here we are just executing an update query.

• Let’s run the project and see if it’s working.

• Okay, so I changed my mind and thought to give you some exercise.

• I want you to create DELETE a route for deleting a book. And GET route for getting “a single book“ from the database.

• I will share the code with you but before that, I recommend trying it out on your own.

• No cheating…

.

.

.

.

.

• You didn’t try, didn’t you…

• Anyway here is the code..

Deleting a Book

// routes.go
r.Delete("/book/{id}", books.DeleteBookHandler)

// handler.go
func DeleteBookHandler(w http.ResponseWriter, r *http.Request) {
    id, err := strconv.Atoi(r.PathValue("id"))
    if err != nil {
        http.Error(w, "Failed to convert id", http.StatusInternalServerError)
        return
    }

    err = DeleteBook(id)
    if err != nil {
        http.Error(w, "Failed to delete book", http.StatusInternalServerError)
        return
    }

    w.WriteHeader(http.StatusOK)
}

//service.go
func DeleteBook(id int) error {
    _, err := database.DB.Exec("DELETE FROM books WHERE id = ?", id)
    if err != nil {
        return err
    }
    return nil
}

Getting Book by id

// routes.go
...
r.Get("/book/{id}", books.GetBookHandler)
...

//handler.go
func GetBookHandler(w http.ResponseWriter, r *http.Request) {
    id, err := strconv.Atoi(r.PathValue("id"))
    if err != nil {
        http.Error(w, "Failed to convert id", http.StatusInternalServerError)
        return
    }

    book, err := GetBook(id)
    if err != nil {
        http.Error(w, "Failed to fetch book", http.StatusInternalServerError)
        return
    }

    w.Header().Set("Content-Type", "application/json")
    json.NewEncoder(w).Encode(book)
}

// service.go
func GetBook(id int) (Book, error) {
    var book Book
    err := database.DB.QueryRow("SELECT id, title, author, year FROM books WHERE id = ?", id).Scan(&book.ID, &book.Title, &book.Author, &book.Year)
    if err != nil {
        return book, err
    }
    return book, nil
}

Modifications

• We did finish our project but there are a few things that need to be fixed in this project. For example, When we are Updating a book, we must pass the whole object with every value instead of just a value that we need to update.

• There are no clear messages in the console after we perform operations.

• We should add proper logs inside each error-handling block.

• I want you to do a little bit of research on these points and update the code.

• It is pretty easy and I think with a little bit of research and work you will be able to complete it.

GitHub Repository

• Here is the Git repository that contains all the code.

Wrapping Up

• That’s it. We have officially completed this series. I hope you learned something from this series.

• If you have any queries or concerns regarding any concepts, please feel free to comment it. I’ll try to answer it as per my knowledge.

• In future blogs, I am planning to experiment more with Golang and create new projects with it. So stay tuned and follow to get the notifications.

• See you in the next one, until then…

• Heyy Gophers!!! Welcome to the Part 4 of Learning Go with Me. Thanks for coming alone to till here. I hope you are learning and enjoying this series.

• Till now we almost covered the fundamentals of Go and now we are ready to write out the first API.

• It’s gonna be exciting as we will learn a lot of new things which will help you in your first programming career

• Let’s get started without wasting any more time.

• We already have the project structure ready for us

crud-api/
│
├── cmd/
│   └── main/
│       └── main.go           # Entry point for the application
│
├── config/
│   └── config.go             # Environment variable management
│
├── internal/
│   ├── routes/
│   │   └── routes.go         # API routes definition
│   ├── books/
│   │   ├── handler.go        # HTTP handlers for book operations
│   │   ├── model.go          # Book database model and queries
│   │   └── service.go        # Business logic for books
│   ├── database/
│   │   └── db.go             # Database connection logic
│
├── .env                      # Environment variables
├── go.mod                    # Go module dependencies
└── .gitignore                # Ignored files

• Also we have dependencies installed in our project from Part 1 of this series

go get github.com/go-chi/chi/v5
go get github.com/go-sql-driver/mysql
go get github.com/joho/godotenvfg

• And lastly, we have the environment file ready for us

// .env
PORT=8080
DB_USER=root
DB_PASSWORD=password
DB_HOST=127.0.0.1:3306
DB_NAME=crud_api

• Now let’s go step by step. As we already have the .env file ready for us, why not create a function to load it inside our project?

• Before that let’s see the main.go file

// main.go

package main

func main() {
    // TODO: Load .env file

    // TODO: Setup database connection

    // TODO: Setup router

    // TODO: Run server
}

• These are our TODOs which we need to complete

Loading .env Variables

• To load values inside our .go files we are using godotenv package. This package helps us get environmental values inside our file.

• We implement this function inside config.go file. So head over to config.go and let’s write LoadEnv function. But before that let’s import a few packages

// config/config.go

package config

import (
    "log"
    "os"

    "github.com/joho/godotenv"
)

• We are using log, os and godotenv packages here.

fmt and log Packages

• In Go we mostly use these two packages to print out values and logs in the terminal. At first, these two may look similar but they both serve different purposes.

• log is specifically designed for logging messages. This means we can get more information like time stamping and error reporting with this. So we mostly use it during application runtime like debugging, informational logs, and error logs.

  • It includes functions like log.Fatal(Exists the program after logging) and log.Panic (Logs and then panic)

• Whereas fmt is used for general-purpose printing text. It is commonly used for printing in a console with formatted strings. So we use it mostly when we are printing user-facing messages.

Example

fmt.Println("Hello World") // Hello World
fmt.Printf("Hello %s", "World") // Hello World

log.Fatalf("This is a fatal error.") // This is a fatal error. (After this program stops)
log.Panic("This is a panic error.") // This is a panic error. (After this program stops)

• More on fmt visit: https://pkg.go.dev/fmt

• More on log visit: https://pkg.go.dev/log

os Package

• The os package in Go is used for tasks related to the operating system. For example accessing the files, and environment variable.

• In our example, we are focused on accessing the .env file and it has Getenv() a function that receives the values of environment variables named by key

• Now let’s write LoadEnv() function

import (...)

func LoadEnv() {
    err := godotenv.Load("../.env")
    if err != nil {
        log.Fatal("Error loading .env file")
    }
}

• We get Load() function inside godotenv package. Load will read the .env file and load them into ENV for this process. Notice that you pass the path of your .env file.

• This function returns an error so we are collecting it inside err variable and then checking if the err has anything. If the err is not nil which means something went wrong. If so, we need to show the error message and exit the program. Because if the env file is not loaded then there is no point in starting the program.

• Now to get a specific value from the env file we need to make one function that will get us the value of the asked key from the env file

package config

import (...)

func LoadEnv() {
    ...
}

func GetEnv(key, fallback string) string {
    if value, ok := os.LookupEnv(key); ok {
        return value
    }
    return fallback
}

• We created the GetEnv function, which takes key and fallback strings as parameters and returns a string.

• The key is the string for which we need the value in our code. For example, inside our .env file, we have different values. If we need only DB_USER, we pass this key to the function, and it returns the related value.

• We also accept a fallback string, so if the requested value isn’t found, we use the default fallback value.

• Here, we are using the os package, which includes the LookupEnv function. This function takes a key as an argument and returns two things: a string and a bool. If the value is found, ok will be true; otherwise, it will be false. By checking ok, we can determine if a value was returned and decide whether to return the fallback value.

• Let’s check if it is working or not

package main

import (
    "fmt"

    "github.com/red-star25/crud-api/config"
)

func main() {
    config.LoadEnv()
    fmt.Println(config.GetEnvValue("DB_USER", "root"))

       // TODO: Setup database connection

    // TODO: Setup router

    // TODO: Run server
}

Output:

root

• That’s awesome. Let’s now set out the Database Connection

Setting Up the Database Connection

Importing Packages

• As you know we are using MySQL for storing our application’s data. We need to first set up the connection and create an instance of the SQL DB.

• Head over to db.go file and let’s first import all the required packages

package database

import (
    "database/sql"
    "fmt"
    "log"
    "os"

    _ "github.com/go-sql-driver/mysql"
)

• We have seen the usage of fmt, log and os. Let’s see what are the remaining ones.

• database/sql: This is a built-in library for SQL database interactions.

• _ "github.com/go-sql-driver/mysql": Imports the MySQL driver as a side effect to register it with database/sql. Here _ (underscore) represent that this import is used as a side effect. Which means this package is not directly used.

Creating Database

var DB *sql.DB

• Here type DB is a pointer to sql.DB an object. We are making it globally accessible so that anyone can reuse it without creating it again and again on each query

• You must be wondering Why it is a type of pointer.

  • So, If DB were a simple variable (not a pointer), any changes made to it inside a function, wouldn't affect the global variable itself. Instead, Go would create a copy of the value, and the global DB would remain unchanged.

  • By using a pointer, it updates the global DB directly. This makes the database connection immediately available to all parts of the application.

Connect function

• Now let’s create a function Connect that will be responsible for opening the database connection and making a connection.

func Connect() {
    dsn := fmt.Sprintf("%s:%s@tcp(%s)/%s",
        config.GetEnvValue("DB_USER", "root"),
        config.GetEnvValue("DB_PASSWORD", ""),
        config.GetEnvValue("DB_HOST", "localhost"),
        config.GetEnvValue("DB_NAME", "crud_api"),
    )
}

• Here we have created a variable dsn(Data Source Name) which is a type of string. So before we connect to MySQL we need to have a URL to which we want to connect.

• So here we have used a formatted string from fmt package. Here function Sprintf returns a formatted string without printing it to the console. %s is a placeholder for string values here.

• And we are getting all the values from our .env file using config’s GetEnvValue function.

• Example dsn

username:password@tcp(localhost)/mydatabase

Opening Connection

• Now that our data source URL is ready, we are ready to open our SQL connection.

• To do that mysql provides Open() function which takes

  • driverName (which is nothing but a database name, ex: MySQL, MongoDB, Postgres, etc). You can provide any driver here and open the connection and

  • dataSourceName is a URL of the database that we just created above.

var err error
DB, err = sql.Open("mysql", dsn)
if err != nil {
    log.Fatalf("Error connecting to database: %v", err)
}

• This function returns two things: db object and err.

• So what we are doing here is, we are assigning the returned SQL db object to our globally defined DB.

• And then we check if there is any error occurred or not.

You will see this type of error handling thoughtout any Go project. This is a convention in Go. It might seems little overdoing but it is a good practice that we are handling error just after calling function.

Also note that there is no try catch or finally keywords in Go. You can read why they are using this type of error handling in this article : https://go.dev/doc/faq#exceptions

Verifying the DB Connection

• To verify whether the connection is successful or not we have Ping function that verifies if the database connection is still alive, and it also establishes the connection if necessary.

if err := DB.Ping(); err != nil {
    log.Fatalf("Error verifying database connection: %v", err)
}

• That’s pretty much it. We can log the message after this that the Database is connected successfully.

Final db.go code

package database

import (
    "database/sql"
    "fmt"
    "log"

    _ "github.com/go-sql-driver/mysql"
    "github.com/red-star25/crud-api/config"
)

var DB *sql.DB

func Connect() {
    dsn := fmt.Sprintf("%s:%s@tcp(%s)/%s",
        config.GetEnvValue("DB_USER", "root"),
        config.GetEnvValue("DB_PASSWORD", ""),
        config.GetEnvValue("DB_HOST", "localhost"),
        config.GetEnvValue("DB_NAME", "crud_api"),
    )

    var err error
    DB, err = sql.Open("mysql", dsn)
    if err != nil {
        log.Fatalf("Error connecting to database: %v", err)
    }

    if err := DB.Ping(); err != nil {
        log.Fatalf("Error verifying database connection: %v", err)
    }

    log.Println("Database connected successfully!")

}

• And now call this Connect function inside main.go

package main

import (
    "github.com/red-star25/crud-api/config"
    "github.com/red-star25/crud-api/database"
)

func main() {
    config.LoadEnv()

    database.Connect()

    // TODO: Setup router

    // TODO: Run server

}

• If everything goes right then you will see this message after you run the program.

  

  Make sure you have MySQL up and running on your computer.

Creating Book model

• We are creating CRUD API for managing Books. So in order to view data in a structured manner with all the required fields we need to create a Book model using struct

In Go, the term "model" refers to a structured representation of data used within an application, typically to map data between a program and an external data source, such as a database or an API. Models are often implemented as structs in Go, which are a way to define and organize related data fields.

• Let’s create our Book model

// internal/books/model.go
package books

type Book struct {
    ID     int
    Title  string
    Author string
    Year   int
}

• Here we have four fields inside our Book struct. ID, Title, Author, Year.

• That’s easy, right? Yes. But there is one more thing which we need to do. So as we know that when we deal with APIs, we use JSON to serialize and deserialize the data that we are getting from the user and data we are sending to the user.

• And generally, they are represented in lowercase like id, name, author, year like that. But if you see our Book struct we have the first letter capital for each field. And that is intentional. Because we need those struct’s fields outside to use. And to make it globally accessible we capitalize the first letter of any variable.

• So to resolve this issue, we use what we call in go “ struct tags “. In other programming languages, we refer to this as “ annotations “.

• This helps us convert these struct fields to proper naming conventions. Let’s see how we do that

package books

type Book struct {
    ID     int    `json:"id"`
    Title  string `json:"title"`
    Author string `json:"author"`
    Year   int    `json:"year"`
}

• So what we do is after giving the type of the struct field we use backticks ` `. And in there we write json: and then whatever name we want.

Implement Database Queries (CRUD)

• Now that our database is up and running let’s write database queries for Create, Read, Update, and Delete.

• First of all, let’s import two required packages

package books

import (
    "crud-api/internal/database"
    "errors"
)

• We are importing a database for accessing the DB object. Using this we can perform all the queries.

• And errors we use for handling errors.

The GetAllBooks Function

• Firstly we are creating a function to get all the books from the database

func GetAllBooks() ([]Book, error) {
    rows, err := database.DB.Query("SELECT * FROM books")
    if err != nil {
        return nil, err
    }
    defer rows.Close()

    ...
}

• Here we created a function called GetAllBooks which is responsible for getting all the books from the database.

• It returns two values:

  • [ ]Book: A slice of struct Book.

  • error: Go’s error object.

• Inside the function body, we are making a query to the SQL database using the DB variable’s Query function. It takes a string where we write a raw SQL query.

• In this function, we need all the books with data. And to do that we wrote this query “SELECT * FROM books“. Which will get all the books from the database.

• And then we checked if it threw any errors

• After that, we defer call the rows.Close() to close it once this function is finished.

• This rows object allows us to iterate over the results we get from the Query. And to iterate over the results we have rows.Next() function which goes over all the results.

• We can take advantage of this function and create a for loop inside which we can fill all the data inside a slice of the book

func GetAllBooks() ([]Book, error) {    
    ...
    var books []Book
    for rows.Next() {
        var book Book
        if err := rows.Scan(&book.ID, &book.Title, &book.Author, &book.Year); err != nil {
            return nil, err
        }
        books = append(books, book)
    }
    return books, nil
}

• Here we have created a temporary slice of the book variable. In this, we will fill in all the data.

• Then we looped through rows.Next() which will go over all the rows and inside the for the body we are getting all mapping the data in a row to the Book struct using rows.Scan

• And after that, we are appending that book to the books slice

• And once every row is scanned we return it from the function.

NOTE: Make sure you have the book table in MySQL database: If you don’t have one then create it and then run the project, otherwise it will throw an error

You can run this query inside your MySQL Workbench

CREATE TABLE books (
    id INT AUTO_INCREMENT PRIMARY KEY,
    title VARCHAR(255) NOT NULL,
    author VARCHAR(255) NOT NULL,
    year INT NOT NULL
);

• Let’s test GetAllBooks.

// main.go

func main() {
    config.LoadEnv()

    database.Connect()

    data, err := books.GetAllBooks()
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println("Books:",data)
    // TODO: Setup router

    // TODO: Run server

}

Output:

• It ran!!!!

• This is great! Now let’s create a Handler function for this function. Because we need to send this data to the user too, right? Till now we just got the data from the database.

• To do that head over to handler.go file

Setting up HTTP Handlers

• Let’s first import the required packages

// handler.go

import (
    "encoding/json"
    "net/http"
)

• Here encoding/json is used. This is used for converting JSON data into Go structures and Go data structures into JSON.

• net/http is used for handling HTTP requests.

• I’ll first show all the code to you and then we will understand what it does

package books

import (
    "encoding/json"
    "net/http"
)

func GetAllBooksHandler(w http.ResponseWriter, r *http.Request) {
    books, err := GetAllBooks()
    if err != nil {
        http.Error(w, "Failed to fetch books", http.StatusInternalServerError)
        return
    }

    w.Header().Set("Content-Type", "application/json")
    json.NewEncoder(w).Encode(books)
}

• func GetAllBooksHandler(w http.ResponseWriter, r *http.Request) {

  • Here we have two parameters:

    • w http.ResponseWriter: This is used for sending data back to the client.

    • r *http.Request: This is used for getting data from the client.

Fetching the Books

• books, err := GetAllBooks()

  • Here, the function GetAllBooks is called to retrieve a list of books. As we saw, that function will get the data from the database and return all the books with an err object.

Error Handling

if err != nil {
    http.Error(w, "Failed to fetch books", http.StatusInternalServerError)
    return
}

• If that function returns any error, we check it inside this if block and send back the HTTP error of InternalServerError.

• And then exiting the function with return

• Here, The http.Error function simplifies sending error responses. This has all the different types of errors you can throw according to your functionalities.

Setting the Response Header

w.Header().Set("Content-Type", "application/json")

• This line sets the Content-Type header of the response to application/json. It informs the client that the response body contains JSON data.

Encoding and Sending the Response

json.NewEncoder(w).Encode(books)

• Now we need to encode the slice of a book in JSON format for the client. To do data we use json Encoder method.

• This sent the data as the HTTP response. And it ensures compatibility with clients expecting data in JSON format.

Configuring Routes

• Now, how these handlers are going to trigger? Of course when the user hits a certain URL, or to be specific an Endpoint from their end.

• And so we want to return data for that particular endpoint/route right? For that, we need to create a route called /book.

• And to create routes we are using the Chi package.

• Go to routes.go and paste the below code

package routes

import (
    "crud-api/internal/books"

    "github.com/go-chi/chi/v5"
)

func SetupRouter() *chi.Mux {
    r := chi.NewRouter()

    r.Get("/books", books.GetAllBooksHandler)

    return r
}

• Here the SetupRouter function will configure all the routes.

• First, we need to create an instance of chi router. This router will handle the incoming HTTP requests and route them to the appropriate handlers.

• r := chi.NewRouter() - This line does that

• And now using this variable r we can create a map of the routes to the handlers.

r.Get("/books", books.GetAllBooksHandler)

• For different HTTP request, we have different HTTP Methods in r. POST, UPDATE, DELETE,etc

• Here we need GET as we are getting the data from the database. Inside r.Get(), first, we need to give the route as /books. This means once this endpoint is hit by the client, the handler defined in the second parameter will be triggered.

• And at the end, we are returning the pointer to the chi.Mux because we need it for the server in the main.go. You will see why below.

Finalizing the main.go file

• Now that we have everything set up we can start the server and listen to the endpoints.

package main

import (
    "log"
    "net/http"

    "github.com/red-star25/crud-api/config"
    "github.com/red-star25/crud-api/database"
    "github.com/red-star25/crud-api/internal/routes"
)

func main() {
    config.LoadEnv()
    database.Connect()

    router := routes.SetupRouter()
    port := config.GetEnvValue("PORT", "8080")
    log.Printf("Server running on port %s", port)
    log.Fatal(http.ListenAndServe(":"+port, router))
}

• Inside the main function, we are loading the environment variables.

• Then we connect to the database.

• And then we call SetupRouter to configure all the routes.

• To run the server we have ListenAndServe function inside http package. Inside we have to first pass the address and then the router that we set up. And we also have to check for any errors. So we are wrapping this function inside log.Fatal

Running The Server

• Now let’s cross our fingers and check if everything is working. To check the functionality of our program. We are using the Postman application.

• So open it up and run it

• You will see an empty slice because we have not added any data yet. We will be implementing other HTTP methods in the next article.

Wrapping Up

• If you came this far to the end of this article, then congratulations for making it.

• In the next article, we will be implementing the remaining CRUD operations which are Update, Delete, and Create. It will be fun because now we have the base setup for us, so now we just need to work on the functionalities.

• Hope you learned something from this article. There are lots of blogs coming in the future with more amazing topics and projects and I am so excited to write about all of it.

• See you in the next blog, until then….

• Connect with me on Twitter, LinkedIn, and Github

• Welcome to Part 3 of the Learning Go with Me series. I hope you like it so far and learned from it. Because that’s the most important thing. If you have any doubts or comments please feel free to comment it. I will try to help you out.

• In the last article, we learned lots of fundamental concepts of Go. And in this blog, I decided to just focus on the Pointers.

• Let’s start without further ado

What is a Pointer?

• A pointer is nothing but a variable that stores the “memory address” of another variable. What does that mean?

• So instead of holding a direct value which we typically do like age := 24, the pointer holds the address of where that value is stored in memory.

• You can think of pointers as post office boxes.

• Each box points to a certain address/ house number. Similarly, if you have defined a variable like this:

num := 42

• Then you can point to this variable using another variable with a pointer

ptr := &num

• You need to first understand what ‘ * ‘ and ‘ & ’ lexical elements mean.

1. Address Operator (&):

   • The & operator is used to get the memory address of a variable. If x is an integer variable then &x will give you pointers to x that is, a memory address where the integer x is stored.

2. Dereferencing Operator ( * ):

   • The * operator is used to get the value stored at a memory address. If p is a pointer to an integer, then *p will give you the integer that p points to.

Example:

func main(){
    num := 42
    ptr:= &num

    fmt.Println(ptr) // 0xc000104040 - Address where num is stored
    fmt.Println(*ptr) // 43 - Value at which ptr is pointing
}

• In this example, ptr is a pointer to num. You can get the value of num by dereferencing ptr with *ptr.

Passing Pointer to Functions

• Pointers also come into play when you're dealing with functions.

• If you pass a variable to a function in Go, the function gets a copy of the variable. Everything in go is a pass-by value, if the function changes the variable, it won't affect the original. But if you pass a pointer to a variable, the function can dereference the pointer to change the original variable.

• Let’s take an example

// Without Pointer - Pass by Value
func valueSemantic(num int){
    num = 43
}

func main(){
    num := 42

    fmt.Println("Before: ", num) // 42

    valueSemantic(num)

    fmt.Println("After: ", num) // 42
}

• This is because a copy of num is getting passed to the function valueSemantic and not the actual value. To update the value that we are passing from other functions we have to use a pointer.

// With pointer - Pass by Value
func valueSemantic(num *int){
    *num = 43
}

func main(){
    num := 42

    fmt.Println("Before: ", num) // 42

    valueSemantic(&num)

    fmt.Println("After: ", num) // 43
}

• As you can see now it got changed. Why?

• Because now we are not passing just the copy of the num but we are passing it an address where num is stored.

• And in the function parameter now that we are passing an address to it we need to accept that address using a pointer. Because remember? The pointer points to another variable’s address.

• Inside the function, we can’t just do this and change the variable:

num = 43

• Because num is a pointer. So first we need to get the value of that pointer by dereferencing it like this *num and then assign a new value to it.

*num = 43

Pointers with Structs

• Lastly, pointers are crucial when dealing with structs in Go. Since Go doesn’t have classes

  and objects, you can use structs to create complex types, and you can use pointers to structs to modify these structs or to avoid copying the structs around.

• Let’s take an example:

type Person struct {
    name string
    age  int
}

func updateName(p *Person, newName string) {
    p.name = newName // Modify the struct via pointer
}

func main() {
    p := Person{name: "Alice", age: 30}

    fmt.Println("Before:", p.name) // Prints: Alice
    updateName(&p, "Bob")
    fmt.Println("After:", p.name) // Prints: Bob
}

• We aren’t doing anything new in this. We created a struct of type Person. It has name and age fields.

• And inside main function we are passing the address of this person p in updateName function.

• updateName takes two parameters. Pointer to a struct person and new name.

• But inside the function, you see we didn’t use * to deference it. Why?

• This is because Go is automatically dereferencing that struct for us. So when you have a pointer to a struct in Go, the language provides a convenience called automatic dereferencing for accessing fields or calling methods. You don't need to explicitly dereference the pointer using * before accessing fields or methods.

For Primitive Types (e.g., int, float, etc.): You must use * to dereference the pointer and access the value.

For Structs: Go automatically dereferences pointers to structs when you access their fields or call methods.

• In my previous blog, I forgot two concepts to discuss which are defer and switch case statements. Let’s understand them one by one

defer

• In Go, the defer the keyword is used to ensure that a function call is executed later in a program’s execution, usually for the purposes of cleanup.

• defer is often used where ensure and finally would be used in other languages.

• When a function call is deferred, it's placed onto a stack. The function calls on the stack are executed when the surrounding function returns, not when the defer the statement is called.

• They're executed in Last In, First Out (LIFO) order, so if you have multiple defer statements, the most recently deferred function will be the first one to execute when the function returns.

func main() {
    fmt.Println("Start")

    defer fmt.Println("This will print last")

    fmt.Println("End")
}

Output:

Start
End
This will print last

Multiple defer Statements

func main() {
    defer fmt.Println("First")
    defer fmt.Println("Second")
    defer fmt.Println("Third")
}

Output

Third
Second
First

// LIFO order as explained above

Real-World Example: Closing a File

func main() {
    file, err := os.Open("example.txt") // Opening a file
    if err != nil {
        fmt.Println("Error opening file:", err)
        return
    }
    defer file.Close() // Ensure the file is closed before exiting the function

    // Perform file operations here...
    fmt.Println("File opened successfully")
}

• In real-world programming, we have many things open. We need to make sure we close those types of functions because otherwise, it will create memory leakage. Although Go has a Garbage collector that will dump those things down if not used, but it’s always a good practice to close the instance of it once the work is done.

• And that’s where defer can be used. We use defer right after we perform operations like opening a file to make sure we don’t forget.

Switch

• switch in Go is a powerful way to control the flow of execution by matching a value against multiple cases. It is simpler and cleaner than writing multiple if-else conditions.

How switch works

• switch evaluates a value and executes the first matching case.

• If no case matches, the default block if executed.

Example

func main() {
    day := "Monday"

    switch day {
    case "Monday":
        fmt.Println("Start of the workweek!")
    case "Friday":
        fmt.Println("Almost the weekend!")
    default:
        fmt.Println("Just another day.")
    }
}

• In the above example, the switch matches day to Monday and executes the corresponding block

• In Go we can use Switch without an Expression. So If we omit the expression, Go evaluates the first case that is true

func main() {
    x := 15

    switch {
    case x < 10:
        fmt.Println("Less than 10")
    case x < 20:
        fmt.Println("Between 10 and 20") // Prints this
    default:
        fmt.Println("20 or more")
    }
}

Fallthrough in switch

• The fallthrough keyword forces the execution of the next case, even if the condition is not true

func main() {
    num := 2

    switch num {
    case 1:
        fmt.Println("One")
    case 2:
        fmt.Println("Two")
        fallthrough
    case 3:
        fmt.Println("Three")
    default:
        fmt.Println("Other")
    }
}

/* 
Output: 
Two 
Three
*/

Wrapping Up

• In this article, we explored some of the fundamental concepts of Go, including pointers, defer statements, and switch statements.

• We learned that pointers store the memory address of variables, enabling functions to modify original data without creating copies.

• The defer statement schedules function calls to execute after the surrounding function completes.

• The switch statement offers a good control flow based on variable values.

• I think this is pretty much it for you to know to get started. So from the next article, we will start building out Book CRUD API. There are still a few things remaining to explain and I will be explaining on the go as it will make much more sense when we are building something.

• I hope you have learned something from this article. If you have any feedback or queries, feel free to ask them in the comments. I’ll be happy to answer all your questions

• See you in the next article, until then….

• Connect with me on Twitter, LinkedIn, and Github.

• Hey there! Welcome to Part 2 of the series of Learning Go with me. In Part 1 we set the stage by introducing the project, defining its goals, and setting up the basic folder structure for our Go project.

• Now it’s time to dive deeper into some fundamental concepts of Go programming.

• In this part, we’ll cover core topics like how Go organizes code using packages and imports, how exported names work, and explore basic Go constructs such as types, variables, constants, functions, slices, maps, loops, and conditionals. Then we’ll also understand how structs and pointers work.

• I will try my best to explain each concept in a beginner-friendly way and in a way that I understand because I am also learning with you. I will try to incorporate these concepts in a way that after understanding it we can directly apply it to our CRUD API project.

• By the end of this section, you’ll have a strong grasp of these concepts and be ready to use them to bring our API to life.

• Let’s jump right in it without wasting any more time

Packages

What is a Package in Go?

• In Go, packages are a way to organize and reuse code. What does that mean?

• Think of a Package like a Folder on your computer. Inside that folder, you have related files. Similarly in Go, the files inside a package folder work together to provide some functionality

• For example:

  • If you are familiar with math package in any other language then you must know that it contains tons of mathematical operations. So all these operations are inside this math package

• And when you want these functions which are defined somewhere else inside your program, you use “import“.

Types of Packages

1. Standard Library Packages:

   • These are built-in packages in Go. Packages like math for mathematical operations, net/http for handling HTTP requests, encoding/json for working with JSON, etc.

2. Third-Party Packages:

   • These are the additional packages that are created by people in the Go community. If you remember, in Part 1 we installed a few dependencies using go get the command. Those belong to this type.

3. Custom Packages:

   • You can create your own packages too which you can use to organize code. We will see this in action soon.

How Packages Work?

• In Go, every program belongs to a package. The main package is special because it’s where the program starts running. When we define the main package inside a file, Go will know from where it should start the program.

Using Packages in Our Project

• In Part 1 we created different folders and files. If you have noticed, every folder and files are in red color which shows that something is wrong. This is because we need to tell which file belongs to which package. Let’s resolve this.

• Go to each file and write the folder name as a package it belongs to.

// cmd/main.go
package main

// config/config.go
package config

// databse/db.go
package database

// internal/books/handler.go, model.go and service.go
package books

// routes/routes.go
package routes

//pkg/logger
package logger

• All the errors should be gone after giving the package name inside every file.

Imports

• To use any package inside a Go project, we need to import it using import a keyword.

import "fmt"

• If you have multiple packages then list them in parentheses

import (
    "fmt"    // for printing.
    "net/http"    // to handle HTTP requests and responses.    
    "encoding/json"    // to parse and generate JSON data
)

Exported Names

What Are Exported Names?

• So in Go, we don’t have keywords like private or public like other languages.

• If the name starts with an uppercase letter, then it is considered as exported, meaning it can be accessed from outside the package where they’re defined.

• Names starting with a lowercase letter are unexpected and private to the package.

Example

type Book struct {    // Exported struct
    Title string    // Exported field
    Author string    // Exported field
}

type car struct {    // Unexported struct
    brand string    // Unexported field
    isElectric bool    // Unexported field
}

• Above we created two structs (don’t worry if you don’t know about structs. We will see it in detail in this blog below).

• Here Book starts with an uppercase letter, meaning we can get this struct in other packages whereas car starts with a lowercase letter, meaning we can’t access this struct anywhere else.

Basic Go Concepts

Types and Variables

• Go is a statically typed language. This means every variable must have a type that’s known at compile time. Common types include:

  • int: Integer values

  • string: Text data

  • bool: Boolean values (true/false)

  • float: Decimal values

Declaring Variables

• There are two ways you can declare a variable:

1. Explicit Declaration

var age int = 24
var name string = "Dhruv"

2. Implicit Declaration

age := 24
name := "Dhruv"

• You can also declare with no value like this:

var age int    // Defaut: 0
var name string    // Default: ""

Constants

• Constants are immutable values, which means we can’t change them once defined.

• Use const keyword to declare constants

• Why Use Constants?

  • We use it to prevent accidental changes to critical values.

  • To improve the readability and maintainability of code

const apiVersion = "v1"
const defaultPort = 8080

• You can use parentheses for multiple declarations too

const (
    apiVersion = "v1"
    defaultPort = 8080
)

Functions

• We use func keyword to declare functions in Go.

func functionName(){
    // logic
}

• You can have parameters inside parentheses

func functionName(parameter1 type, parameter2 type){
    // logic
}

• You can return from a function by defining a return type like this

// Single value return
func functionName() int {
    return 0   
}

// For multiple value return
func functionName() int, string {
    return 0, "Hello"
}

Conditionals

• The if statement is used for decision-making

marks := 85
if marks >= 9- {
    fmt.Println("Grade: A")
} else if marks >= 75 {
    fmt.Println("Grade: B")
} else {
    fmt.Println("Grade: F")
}

• Short Statement with if. In this, we are first declaring num , and right after that we are checking the condition for it.

if num := 10; num%2 == 0 {
    fmt.Println("The number is even.")
} else {
    fmt.Println("The number is odd.")
}

Loops

• In Go we have only for loop for looping. There are different ways you can use for loop. Let’s see it in action

// Basic `for` loop
for i:= 0; i<5; i++ {
    fmt.Println("Value of i: ,", i)
}

// Infinite Loop
count := 0
for {
    fmt.Println("Looping forever")
    count++
    if count == 3 {
        break // Exists the loop
    }
}

// Condition Only Loop
count:= 0
for count < 3 {
    fmt.Println("Count: ", count)
    count++
}

Struct

• If you come from programming languages like Javascript, Dart, or Python, you must be familiar with class and objects.

• In Go we don’t have that. We use struct to group data together.

• Let’s take an example of our CRUD API project. We need to represent a book with multiple details like a title, id, name, etc.

• So instead of creating separate variables for each piece of data, you can use a struct to combine all of those into one entity.

Defining a Struct

• To define a structure, you use the type keyword followed by the structure name and its field. Each field has a name and a type

type Book struct {
    ID string
    Title string
    Author string
    Year int
}

Creating and Using a Struct

• Once we have defined a structure, we can use it to create an object and assign values to its fields

var book1 Book    // Creates an instance of `Book` struct
book1.ID = "1"    // Assigns the value "1" to the ID field of struct
book1.Title = "Maths"
book1.Author = "Jane"
book1.Year = 2012

Initializing a Struct in Different Ways

1. Using Field Names

   • We can initialize a struct by specifying the field names and their values like below

    book1 := Book{
        ID: "1"
        Title: "Maths"
        Author: "Jane"
        Year: 2012
    }

• This looks clear, isn’t it?

2. Without Field Names

   • We can initialize it without field names too like this

    book1 := Book{
         "1"
         "Maths"
         "Jane"
         2012
    }

• This is less readable because it relies on the order of fields.

Structs with Methods

• A method is a function that is attached to a specific type. This type of method initialization was new and kinda weird for me at first. Because usually we are required to create methods inside a class in other languages right? but here we create it separately and then attach it to the structure using the receiver.

• Let me show you how

type Book struct {
    ID string
    Title string
    Author string
    Year int
}

func (b Book) SayBookName() {
    fmt.Println("This book name is: ", b.Title)
}

func main() {
    book1 := Book{
        ID: "1"
        Title: "Maths"
        Author: "Jane"
        Year: 2012
    }

    book1.SayBookName() // Call the SayBookName method
}

• As you can see to attach this SayBookName method to Book struct we provided receiver before the method name inside the parenthesis (b Book). So Inside this file wherever any method has this receiver it will automatically get attached to this struct.

• You can also nest struct inside a struct like this

type Book struct {
    ID string
    Title string
    Author Author // Nested struct
    Year int
}

type Author struct {
    FirstName string
    LastName string
}

func main() {
     book1 := Book{
        ID: "1"
        Title: "Maths"
        Author: Author{
            FirstName: "Jane"
            LastName: "Doe"
        }
        Year: 2012
    }
}

Slice

• A slice in Go is a data structure used to hold a collection of elements of the same type.

• Think of it as a more powerful and flexible version of an array.

• While arrays have fixed sizes, slices can grow and shrink dynamically.

• It’s really simple to use

Creating Slices

1. Using the make function

   • The make function creates a slice with a specific length and capacity.

slice := make([]int, 3, 5) // Create a slice of type int, of length 3 and capacity 5

• Here length represents the number of elements currently in the slice which is 3 here: [0,0,0]

• And Capacity represents the total space allocated for the slice which is (5). This means this slice can have max 5 elements in it.

2. Using a Slice Literal

   • We can define a slice directly with values using slice literal

slice := []int{1,2,3,4,5}

Adding Elements to a Slice

• We can add elements to a any slice using append function.

slice := []int{1,2,3}
slice = append(slice, 4,5) // adding 4 and 5 to the slice

• We need to pass the slice where we want to add elements as the first parameter and then we can pass as many elements as we want after that.

Iterating Over a Slice

• We use for loop to access elements in a slice

slice := []int[1,2,3,4,5]

for index, value := range slice {
    fmt.Println("Index: ", index, "Value: ",value)
}

for _, value := range slice {
    fmt.Println("Value: ",value)
}

Range

• Here range the keyword is used to iterate over elements in a variety of data structures. In the above example, we use range to print the index and value in a slice.

• We use underscore (_) to skip the variable if it’s not needed

Slicing a Slice

• We can create a new slice by slicing an existing one.

slice := []int{1,2,3,4,5}
newSlice := slice[1:3] // Slice from index 1 to 2 (3 is exclusive)

// newSlice: [2,3]

Length And Capacity

• Slices have both length and capacity as we saw above. To get the length and capacity of a slice we use len() and cap() functions

slice := []int{1,2,3,4,5}
fmt.Println("Length: ", len(slice)) // 5
fmt.Println("Capacity: ", cap(slice)) // 5

Appending Slices

• We can add a whole new slice to an existing slice using . . . (unpack operator)

    slice := []int{1,2,3,4,5}
    slice2 := []int {6,7,8}
  
    combined := append(slice, slice2...) // [1,2,3,4,5,6,7,8]
  

Map

• This is another very helpful data structure in Go which is built in inside it. It stores key , value . This is similar to dictionaries in Python or hash maps in other programming languages.

• Keys must be of a comparable type that is string, int and values can be of any type.

• This data structure is useful, especially for lookups when the key uniquely identifies the data. In our case, it is Book ID

Declaring and Initializing a Map

1. Using make

myMap := make(map[string]int)

myMap["Maths"] = 2
myMap["Biology"] = 3

• Here [string] represents key and int represents value

2. Using Map Literal

   • We can initialize and create a map in a single step like we did in struct

myMap := map[string]int{
    "Math": 2,
    "Biology": 3,
}

Accessing Values in a Map

• To get the value associated with a key, you have to use the map followed by the key in square brackets

myMap := map[string]int{
    "Math": 2,
    "Biology": 3,
}

fmt.Println(myMap["Maths"]) // 2

Checking if a Key Exists

• We use the comma ok idiom to check if a key exists or not. This is very useful when we need to check if certain keys exist in a map or not. We can do that in a single line of code

myMap := map[string]int{
    "Math": 2,
    "Biology": 3,
}

value, ok := myMap["Physics"]

if exists {
    fmt.Println("Value: ", value)
} else {
    fmt.Println("Key does not exist")
}

• value, exists := myMap["Physics"]: ok is true if the key is present and false otherwise.

Adding and Updating Elements

• It’s really easy, you just access the value using the key and add a new value or assign a different value to it

myMap := map[string]int{
    "Math": 2,
    "Biology": 3,
}

myMap["Physics"] = 4 // Adding new element

myMap["Math"] = 1 // Updating exisitng element

Deleting Elements

• We use delete() function to remove a key-value pair from the map

myMap := map[string]int{
    "Math": 2,
    "Biology": 3,
}

delete(myMap, "Math") // Remove the key "Math"

Iterating Over a Map

• We use again for loop with range to iterate over all key-value pairs.

myMap := map[string]int{
    "Math": 2,
    "Biology": 3,
}

for key, value:= range myMap {
    fmt.Println("Key: ", key, "Value: ", value)
}

Wrapping Up

• I think that’s pretty much all you need to know for now. We learned lots of fundamental stuff in this article. I tried to make it as concise as possible in order to quickly grasp the underlying concepts because going into too much detail with all the theory doesn’t make sense at this point.

• There is still one concept remaining which is Pointers. I wanted to cover it in this blog only but then thought it would become quite a big read so I will be explaining it in detail in the next blog.

• I hope after this article you are at least comfortable with the basic stuff and syntaxes. Just practice the syntax that you find difficult to grasp. It will all make sense and your hands will then flow naturally on the keyboard afterwards.

• See you in the next one, until then…

• Connect with me on Twitter, LinkedIn, and Github

• Hello, everyone. It’s been a while since I last published anything on my blog. A lot has happened during this time, including leaving my full-time job and moving to the United States for my master's degree.

• So, as my full-time job was more focused on the front-end side. I always wanted to learn a backend programming language. I researched what’s going on in the software industry. And I found GO programming language which is in the trend in recent years. From what I have felt so far by learning a few things in GO till now I absolutely agree with this trend.

• There are many things that makes Golang a good choice for companies to use as their backend language. And so I started learning it and as you might be knowing if are my reader that I share what I learn. Because I like to learn in public and share everything that I am learning. So here I am with a series on Learning GO.

• I personally like to learn by building something. So, you and me both will be learning and building a backend system together in this series. I am so excited and looking forward to this series. I hope you get something from this. So without further a do, let’s get started.

Why GO?

• Go, also known as Golang is a programming language by Google. The creators of the Go programming language are Robert Griesemer, Rob Pike, and Ken Thompson.

• What I like about Go is, it is very fast and it’s really easy to pick up if you know any other programming languages. It is designed to be simple, efficient, and highly scalable which makes it an excellent choice for building web servers, APIs, and large-scale systems.

• I am listing down few points for you to understand why Go has become so popular among developers in recent years.

  • Ease of Learning:

    • It’s easy to learn as its syntax is clean and beginner friendly. It is syntactically similar to C language but it also has memory safety, garbage collection, structural typing and concurrency.

  • Performance:

    • As I mentioned, it is really fast, it compiles the code in seconds. This makes it perfect for web applications.

  • Concurrency Built in:

    • Go has powerful tools for handling multiple tasks simultaneously (more about concurrency in future blogs).

  • Wide Adoption

    • Companies like Google, Uber, Dropbox and many big tech giants are using Go as their primary backend language which gives programmers a sense of trust that this is something on which they can make a career and earn money.

• I personally think that learning Go might seem a little tricky due to its syntax. But once you pick up and know syntax it’s easy to code. There are a few things that might surprise you because Go doesn’t have few things that other programming languages have. We will see it in upcoming blogs.

What is a CRUD API?

• If you are fresher and don’t know about CRUD then this section is for you. So CRUD basically stands for Create, Read, Update and Delete. These are you can say the base of any backend system. These operations allow users to interact with a database.

• For example, you use Instagram. There are certain things you do like, Creating new Posts (Create), Reading Messages (Read), Update your profile (Update), Delete your past posts (Delete).

• You see, all these are CRUD operations in action. In this series we will learn how to create these operations. We will be applying all these operations in our books example. I picked this project because this is perfect for beginners like you and me as it is straightforward yet comprehensive enough to cover key programming concepts.

What Will You Learn?

• In this series of blogs, We will learn basics and some advance stuff of Go while building a RESTful API. And by the end of this project, you’ll have practical knowledge of:

  • Structs and slices to organize and manipulate data.

  • How to use http package to setup a web server.

  • How to manage router.

  • Encoding and decoding JSON to handle API requests and responses.

  • Writing clean, reusable, and maintainable Go code

  • We will also be learning advance concepts like contexts, go routines, channels, etc.

  • and so on…

Why Hands-On Projects Work Best

• I personally think that a person cannot learn a programming language by just reading or watching videos online. It’s not just about memorizing rules and syntax of it. It’s more about building something real. And that’s why In my blog also I try to use some real world example to make it simple to understand.

• This type of learning reinforces concepts which one can immediately apply on what you’ve learned to see how it works

• Also by building something while learning gives a sense of confidence that “Yes, I can do this!“

• And last but not least, you can add this to your resume because that is what an employer look for in a candidate which is Practical skills.

So roll up your sleeves, fire up your favorite code editor, and let’s get started. Because this isn’t just about learning Go- It’s about empowering yourself to build things. Let’s make coding fun, practical, and rewarding.

Prerequisites

• So before we go ahead there are few things which you need to make sure you have setup on your side. Don’t worry if you are new to Go. We will go step by step. So here’s what you’ll need to get started:

1. Basic Knowledge:

   • To follow along with this series you don’t need to be an expert programmer. However, a basic understanding of concepts like variables, functions, and loops will be helpful. So if you are completely new to coding, you might want to spend a little time learning these fundamentals first.

2. Install Go on Your Machine:

   • As it is not that complicated to install Go in your machine. I would like you to install by your own.

   • You have to just visit the official Go download page. And grab the installer for your OS (Windows, Mac or Linux)

   • Then run the installer and follow the instructions. Once installed, Go should be added to your system’s PATH.

   • You can verify the installation by opening the terminal or command prompt and type:

   •         go version
     

     

   • If you see a message like go version go1.xx.x, you are good to go!

3. Set Up a Code Editor

• While there are many options available online for code editor, I personally recommend VS Code for its simplicity and its excellent Go support. If you don’t have it already.

  1. Download and Install VS Code.

  2. Install this Go extension.

Project Overview

• Now that we are setup up, let’s see what are we gonna building. So the goal of this project is to create a simple RESTful API using Go while learning Go concepts.

• This API will allow users to manage collection of books by performing CRUD operations: Create, Read, Update and Delete.

What is a RESTful API?

• If you are completely new to this terms REST, then this section is for you. Basically a RESTful API is a way for different softwares to communicate over the web. REST stands for (Representational State Transfer). It is a design principle that make sure that APIs are simple, scalable and easy to use.

• In our case, we will be building an API that uses HTTP methods like:

  • GET to retrieve data.

  • POST to create new data.

  • PUT to update existing data.

  • DELETE to remove data.

Project Features

• Our API will manage list of books. So we will be having these properties in books:

  • ID - a unique identifier for each book.

  • Title - the name of the book.

  • Author - the person who wrote the book.

  • Year - the year the book was published.

Here’s is a sneak peak at what our API will do:

1. Add a New Book: Accept data from the user and add book to the list.

2. View All Books: Return a list of all books.

3. Update a Book: Modify the details of an existing book.

4. Delete a Book: Remove a book from the list based on its ID.

Setting Up Your Go Project

• Now we are ready to dive into coding. The first step is to setup your Go project.

• We need to setup a proper project structure for our Go application. Following an industry standard folder structure ensures that our code is clean, modular and easy to maintain as the project grows. Here is how to do it step by step.

Create the Project Directory

1. Open your terminal and navigate to the folder where you want to store your project.

2. Create new directory for the project:

mkdir crud-api
cd crud-api

3. Initialize a new Go module:

go mod init github.com/red-star25/crud-api

Go Modules

• If you are familiar with Javascript, Node, Flutter, then you might be knowing the file called package.json or pubspec.yaml

• This file is responsible for maintaining the dependencies/packages. Whether it is built-in in the language or third-party, all the dependencies with their versions are mentioned in this file.

• This dependency management system is introduces in Go 1.11 and made the default in Go 1.13.

• So what this above go mod init command will do is it will create go.mod file in the root directory of your project.

• This file specifies few things:

  1. The module’s name, typically repository URL or a path that identifies the codebase.

  2. Dependencies required by the project and their versions.

  3. The Go language version used.

You might have noticed that we are using GitHub repository URL for module. It is because Git repo ensures that no conflicts is there when fetching dependencies.

Another reason is because of versioning. Repo often follow semantic versioning using Git tags (ex., v1.0.0) which Go modules use to resolve the correct dependency version.

Installing dependencies

• In this project we will be using MySql as our database and also Chi package for managing routing. We will also be using godotenv for environmental configuration. Don’t worry if you don’t have any idea about it just install these packages for now and we will see what it does and how it works in upcoming blogs.

• So to install packages in our go project we have go get <package-name> command.

go get -u github.com/go-chi/chi/v5
go get -u github.com/go-sql-driver/mysql
go get -u github.com/joho/godotenv

Folder Structure

• Go encourages simplicity, but for larger applications, it is good practice to follow a clear folder structure.

• There are many tools available like: Go Project Layout and Go Blueprint. Which helps you to create a base project for your Go application.

• So let’s create folder structure for our project.

crud-api/
│
├── cmd/
│   └── main/
│       └── main.go            # Main entry point for the application
│
├── config/
│   └── config.go              # Environment and configuration management
│
├── internal/
│   ├── routes/
│   │   └── routes.go          # Centralized route definitions
│   ├── books/                 # Business logic for managing books
│   │   ├── handler.go         # HTTP handlers for book operations
│   │   ├── model.go           # Book data structure and database queries
│   │   └── service.go         # Business logic for books
│   ├── database/              # Database utilities
│   │   └── db.go              # Database connection logic
│
├── pkg/                       # Shared utility packages
│   └── logger/                # Logging utilities
│       └── logger.go
│
├── .env                       # Environment variables
├── go.mod                     # Go module dependencies
└── go.sum                     # Dependency checksums
|__ .gitignore

• Let’s understand this structure first before we go ahead.

Top Level Files

• .env

  • This contains environment specific configuration variables (e.g., database credentials, API keys).

• go.mod

  • We already know what it is now

• go.sum

  • Contains the checksum of dependencies to ensure integrity and consistency across builds.

• .gitignore

  • Files to ignore in Git commit

To add .gitignore file for Go, I recommend using gitignore VS code extension by CodeZombie. It will pull up Go specific .gitignore file for you

After installing, press Cmd + Shift + P or Ctrl + Shift + P and type Add gitignore. It will ask for languages, so search for Go and hit Enter.

• Final Top Level Structure will look like this.

Folder and File Structure

• cmd/main.go:

  • So in go you must define an entry point for the program to start. And to define it we create main.go file inside cmd folder

  • main.go have main function inside which initializes the application. We setup routes, server and handle the configuration of the project here in this function.

• config/config.go:

  • In this file we load all the environment variables which is defined in .env .

  • Remember we installed godotenv package? This package help us reading these values. There are many other packages which does the same. I will leave it for you to search it.

  • So basically it defines structs which will hold configuration data for example, database credentials, server ports, etc. (more on structs later)

• internal/

  • This folder holds the core business logic and internal code.

  • Code written inside this will not be exposed as part of the public API.

1. routes/routes.go

   • In this we register all our routes for API like /books, /authors which will be mapped to it’s respective handlers/functions. Again don’t worry if you don’t understand any of these term, we will be seeing it in action soon for better understanding. Just get the basic understanding for now.

2. books/

   • This folder handles all the business logic related to the “Books“ resource.

   • handler.go:

     • In this we define HTTP handlers for CRUD operations.

     • Ex: CreateBook, GetBooks, UpdateBook, DeleteBook

     • Basically we have the logic of our CRUD operations for book in this file for handling HTTP requests and responses.

   • model.go:

     • In this we have Book structure with fields like ID, Title, Author.

     • Also we implement database queries here like FindAllBooks, SaveBook etc.

   • service.go:

     • This contains all the business rules and logic for books and it will act as a Bridge between the handler and model like validating data.

3. database/db.go:

   • As the name suggests it has the logic for handling database connection and utilities.

   • In this we initialize database connection.

4. pkg/:

   • This folder contains code which is shared across different parts of the application or even other projects.

   • logger/logger.go:

     • This file implements logging logic using packages like logrus or zap

     • It may include support for logging levels like INFO, DEBUG, ERROR.

How It All Fits Together

• Let’s understand how all these files are connected to each other.

  • cmd/main/main.go Loads configuration from config/

  • It connects to database via internal/database/

  • Sets up routes using internal/routes/

  • And finally starts the server.

• internal/books handles CRUD operations for book and pkg/logger provides a way to log information or errors from anywhere in the project.

• .env is loaded to proved sensitive or environment specific configurations

This organization ensure clean separation of concerns, making the codebase scalable, maintainable and testable.

• I know I spent a lot of time explaining folder structure but it’s a one time investment. Almost every go project will have similar kind of structure, so in future if you encounter this you will have an idea of what each part is used for.

Configure Environment Variables

• Go to .env file in the root directory and let’s add configuration values:

DB_USER=root
DB_PASSWORD=crudAPI@1234
DB_HOST=127.0.0.1:3306
DB_NAME=crud_api
PORT=8080

• As we will be using MySQL database we need to configure variables like username, password and host value. So we mention everything inside .env file as we don’t want to expose these sensitive values publicly.

Wrapping Up

• I will end this blog here. We learned many things about Go:

  • Why Go?

  • What is CRUD operations?

  • What is RESTful APIs?

  • Go Modules

  • How to install packages in Go?

  • Folder Structure of Go project

  • Environmental variables file.

• Before we dive into coding. We first need to understand few concepts of Go. Which I will be explaining in the next blog.

• I hope you like the blog and learned something from it. If you have any questions or suggestions please leave them in the comment section. I would love to answer and discuss them.

• See you in the next article. Until then….

• Connect with me on Twitter, LinkedIn, and Github.

• Heleww there 👋🏻, welcome again or welcome if you are coming here for the first time. Thanks for clicking on this blog and taking the time to read it.

• I'm writing a series on Clean Architecture in Flutter, and this is the last article in that series.

• In Part 1, we covered the basics of Clean Architecture, what it is, and why it's important for Flutter developers or any developer to use this architecture when coding.

• In Part 2, we looked at how to organize your Flutter project and build an application using Clean Architecture. We set up Data, Domain, and Presentation layers, along with Core and other necessary folders. We went through each folder, explaining its purpose and how to use it.

The Objective of Part 3

• In this blog, we are going to create two feature.

1. News:

   • Here, we will use a dummy news API to get News.

   • We will also cache all the news using Hive DB, allowing users to read and interact with the app even when they are offline.

2. Number/Count Info:

   • We'll use the Numbers API. Here, we'll send a random number to the API, and it will give us an interesting fact about that number.

   • We'll also use Hive to cache everything for this feature.

• This blog is going to be very interesting. You'll learn a lot, not just about Clean Architecture, but also how to organize code, name things properly, and much more.

Are you readyyy!!!?

• Yes!!!? Then let's get started without any delay.

There is one💡BONUS for you too. So stay tuned.

Setting up the environment and project structure

• To get started clone below repository in your local machine:

Feature 1: News

• Let's create a feature called news inside lib/features/

• Now, let's also create sub-folders for each of these directories:

  • data:

    • datasources

    • models

    • repositories

  • domain:

    • entities

    • repositories

    • usecases

  • presentation

    • cubit

    • pages

    • widgets

• So, we've set up the basic structure for the news feature. What's next? Let's begin with the presentation layer.

Setting up routes

• Let's create a page named news_page.dart in features/news/presentation/pages/ so we can add it to the route and get started.

import 'package:flutter/material.dart';

class NewsPage extends StatelessWidget {
  const NewsPage({Key? key}) : super(key: key);

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(
        title: const Text('News'),
      ),
    );
  }
}

• To add this route go to routes.dart and paths.dart file and add below code snippets:

// paths.dart
class Paths {
  static String newsPage = '/news';
}

// routes.dart
final router = GoRouter(
  initialLocation: Paths.newsPage,    // <---- Add initial route too
  routes: [
    GoRoute(
      path: Paths.newsPage,
      builder: (context, state) => const NewsPage(),
    ),
  ],
);

• We also have to update app.dart files' configurations.

// app.dart
import 'package:clean_architecture_bloc/routes/routes.dart';

class App extends StatelessWidget {
  const App({super.key});

  @override
  Widget build(BuildContext context) {
    return ScreenUtilInit(
      designSize: const Size(390, 844),
      builder: (_, __) => MaterialApp.router(    // <------
        // other configuration
        routerConfig: router,     // <---- Provide our `router` as routerConfig
      ),
    );
  }
}

Now, If we run the app you will see News text in AppBar

Implementing Domain Layer

• As we know that Domain layer is the heart of the Clean Architecture. Let's start with this layer.

News Entity

• Let's start with the entity. First we need to see what the data looks like so click on this link and analyse the data : News API

• Here, we see that the main data we need is a list of articles, right? So, this becomes our entity because, remember, what is an Entity?

  They are like the main characters in a story. They are real-world things or concepts important to your app. For example, in an e-commerce app, entities could be Product, Customer, and Order.

• So let's create news_entity inside features/news/domain/entities folder.

class NewsEntity extends Equatable {
  @HiveField(0)
  final String? status;
  @HiveField(1)
  final int? totalResults;
  @HiveField(2)
  final List<ArticlesEntity>? articles;

  const NewsEntity({this.status, this.totalResults, this.articles});

  @override
  List<Object?> get props => [status, totalResults, articles];
}

class ArticlesEntity extends Equatable {
  @HiveField(1)
  final SourceEntity? source;
  @HiveField(2)
  final String? author;
  @HiveField(3)
  final String? title;
  @HiveField(4)
  final String? description;
  @HiveField(5)
  final String? url;
  @HiveField(6)
  final String? urlToImage;
  @HiveField(7)
  final String? publishedAt;
  @HiveField(8)
  final String? content;

  const ArticlesEntity(
      {this.source,
      this.author,
      this.title,
      this.description,
      this.url,
      this.urlToImage,
      this.publishedAt,
      this.content});

  @override
  List<Object?> get props => [
        source,
        author,
        title,
        description,
        url,
        urlToImage,
        publishedAt,
        content
      ];
}

class SourceEntity extends Equatable {
  @HiveField(0)
  final String? id;
  @HiveField(1)
  final String? name;

  const SourceEntity({this.id, this.name});

  @override
  List<Object?> get props => [id, name];
}

• Here were also annotating every field of our Entity with HiveField() because we want our news to be stored locally. If you don't have the functionality of storing data locally then you can exclude that step.

GetNews UseCase

• What is the main goal of this feature? To get News from the API, right?

• So that becomes our Usecase to GetNews from the API.

Recap: Use cases represent specific actions that users can do to accomplish something in your app. They act as the bridge between what the user does and the core logic covered by entities.

• Let's create a file get_news.dart inside features/news/domain/usecases and create GetNews use case

class GetNews {
  final NewsRepository newsRepository;

  GetNews({required this.newsRepository});

  Future<Either<Failure, NewsEntity>> call() async {
    return await newsRepository.getNews();
  }
}

• Here, we have the NewsRepository, and we're using its getNews() method to get data from the Data layer.

• We're using dartz for functional programming. The return type is Either, meaning this function will return either Failure or NewsEntity. For more info, check out the dartz package.

• You might see some errors because we haven't created NewsRepository yet. Let's create it now.

News Repository