🛒 Building an E-Commerce Website with a Microservices Architecture on Amazon EKS

Objective

This project demonstrates the implementation of a fully automated CI/CD pipeline for a microservices-based architecture deployed on AWS EKS using Jenkins. The application comprises 11 microservices, each independently developed, built, and deployed, ensuring scalability, flexibility, and seamless updates.

Key Accomplishments:

Infrastructure Automation: Automated creation of AWS resources using Terraform, including reusable EC2 instance modules for easy provisioning.
Pipeline Automation: Developed Jenkins pipelines triggered by GitHub webhooks for fully automated build, deployment, and validation of all microservices.
Containerized Deployments: Leveraged Docker and Kubernetes for seamless deployment and management of microservices on EKS.
Load Balancer Verification: Validated deployments through a load balancer, ensuring live application availability and functionality.
Security Integration: Configured Kubernetes service accounts, roles, and secrets for secure token management and smooth deployments.

This project highlights expertise in creating scalable, automated CI/CD pipelines tailored for microservices architectures, enabling efficient workflows and high application availability.

Prerequisites :

Before proceeding with the project, ensure you have the following:

AWS Account with permissions for EKS, EC2, and IAM.
AWS CLI and eksctl installed and configured.
kubectl to manage Kubernetes clusters.
Jenkins set up for CI/CD pipelines.
GitHub repository for microservices and deployment configurations.
SSH Key Pair for EC2 access.
IAM Roles for EKS and EC2 management.
Terraform (optional) for infrastructure provisioning.
Basic knowledge of Kubernetes and Microservices architecture.

Github project link: Click here

Project Highlights

Microservices Architecture

The project follows a microservices design, where each component performs a specific functionality, such as Email Service, Cart Service, Shipping Address Service, and Frontend UI.
Microservices are developed, tested, and deployed independently, ensuring flexibility and rapid updates.

Independent Development & Deployment

Each microservice resides in its own dedicated GitHub branch with source code specific to that service.
Modifications to one service can be made and deployed without affecting other services, enabling agile and seamless updates.

Jenkins Multi-Branch Pipeline

A Jenkinsfile is included in each branch for multi-branch pipeline configuration.
The pipeline is triggered automatically by GitHub webhooks when changes are pushed to the corresponding branch, eliminating manual intervention and accelerating deployment.

AWS EKS (Elastic Kubernetes Service)

The application is deployed on AWS EKS for scalable and robust orchestration of microservices.
Two additional branches are included:
- Infra Setup Branch: Contains resources and configurations for provisioning the EKS cluster.
- Main Branch: Includes YAML files for deploying the microservices to the EKS cluster.

Real-Time CI/CD Automation

Every change made to a microservice triggers the pipeline to automatically:
1. Build the microservice
2. Run tests
3. Deploy to the EKS cluster
This automation ensures continuous integration and delivery, reducing downtime and improving efficiency.

Repository Structure

13 GitHub branches:
- 11 branches for individual microservices
- 1 branch for infrastructure setup
- 1 branch for deployment configurations

Why Microservices?

Microservices offer several advantages:

Scalability: Each service can be independently scaled based on demand.
Flexibility: Services can be developed, tested, and deployed independently, allowing faster iterations.
Resilience: Failure in one microservice does not affect others, improving overall system stability

Task 1: Automating Infrastructure with Terraform

Steps for Infrastructure Setup

Set Up EC2 Instance: I created an Ubuntu EC2 instance using Terraform.
Security Group Configuration: Configured the security group to allow necessary ports like SSH (22), HTTP (80), and other relevant ports.
Storage: Added 25GB of storage to the instance for sufficient space.
Module Creation: Instead of rewriting the EC2 setup code, I created a reusable EC2 instance module and stored it on GitHub.
Root Configuration: In the main.tf file, I referenced the EC2 module from GitHub and passed the necessary variables (like instance type and security group).
Deployment: Ran the Terraform commands to successfully create the EC2 instance.

WHY MODULES?

I created a reusable EC2 instance module and stored it on GitHub. In my main main.tf file, I used the module block to refer to the EC2 instance module (which I stored on GitHub) and passed the required values like instance type, key name, and others. This way, I don't have to rewrite the same code every time I need an EC2 instance. It makes the process simpler and more organized!

Benefits:

Benefits of Modules:

Reusability: The EC2 module can be reused across different projects.
Easy Configuration: Simply provide the required values in main.tf and terraform.tfvars.

Terraform File Structure:

root/
│
├── main.tf                # Root module calling the EC2 instance module from GitHub
├── terraform.tfvars       # Variable values for the root module
├── variables.tf           # Declare variables for the root module
│
└── modules/
    └── ec2_instance/      # Child module stored in the GitHub repository
        ├── main.tf        # Define EC2 instance and security group
        ├── variables.tf   # Declare module-specific variables
        └── outputs.tf     # Output the EC2 instance details

Root Module (main.tf)

provider "aws" {
  region = "ap-south-1"
}

# Security Group Resource
resource "aws_security_group" "Web-SG" {
  name        = "Web-SG"
  vpc_id      = var.vpc_id

  ingress {
    from_port = 22
    to_port = 22
    protocol = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    from_port = 443
    to_port = 443
    protocol = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    from_port = 80
    to_port = 80
    protocol = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    from_port = 3000
    to_port = 10000
    protocol = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    from_port = 6443
    to_port = 6443
    protocol = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    from_port = 25
    to_port = 25
    protocol = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    from_port = 465
    to_port = 465
    protocol = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    from_port = 27017
    to_port = 27017
    protocol = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    from_port = 30000
    to_port = 32767
    protocol = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port = 0
    to_port = 0
    protocol = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name = "web-sg"
  }
}

module "ec2_instance" {
  source              = "git::https://github.com/Sandhyagito/terraform-modules.git//ec2_instance"  # GitHub source path
  ami_value           = var.ami_value
  instance_type_value = var.instance_type_value
  key_name            = var.key_name
  subnet_id           = var.subnet_id
  vpc_id              = var.vpc_id
  aws_security_group  = aws_security_group.Web-SG.id  # Pass the Security Group ID here
  volume_size         = var.volume_size
}

output "instance_public_ip" {
  value = module.ec2_instance.instance_public_ip
}

output "instance_id" {
  value = module.ec2_instance.instance_id
}

variables.tf

variable "ami_value" {
  description = "AMI ID for the EC2 instance"
}

variable "instance_type_value" {
  description = "Instance type for the EC2 instance"
}

variable "key_name" {
  description = "Key pair name to access EC2 instance"
}

variable "subnet_id" {
  description = "Subnet ID for the EC2 instance"
}

variable "vpc_id" {
  description = "VPC ID where the EC2 instance will be launched"
}

variable "aws_security_group" {
  description = "The security group to assign to the EC2 instance"
}

variable "volume_size" {
  description = "The size of the root volume in GB."
  type        = number
  default     = 25  # Default value; can be overridden when calling the root module
}

terraform.tfvars

ami_value = "ami-053b12d3152c0cc71"
instance_type_value = "t3.medium"
key_name = "DevOps-keypair"
subnet_id = "subnet-0f1ef5333d021abb3"
vpc_id = "vpc-0a948a81bf52eb8ee"

Instead of hard-coding values into the main.tf root, I have specified them in terraform.tfvars.

Child module

➡️Github repo: terraform-modules

Run Terraform Commands

Terraform init

Terraform Validate

Terraform plan

Terraform apply

I encountered a few errors while setting up my infrastructure in Terraform. Here they are:

1️⃣Unsupported Instance Type:
Issue: Terraform failed to create the EC2 instance because the requested instance type was not supported in the specified availability zone.

Fix: I have opted for t3.medium

2️⃣Unsupported Argument Error:

Root Cause: Misconfigured resource definition. The vpc_id argument was mistakenly added to the aws_instance resource, which doesn't require it.
Type of Issue: Misconfiguration due to adding an unnecessary or invalid argument.

3️⃣Volume Size Argument Error:

Root Cause: The volume_size argument wasn't defined or handled properly in the EC2 module.
Type of Issue: Misconfiguration and missing functionality in the module, as the module code did not include support for the argument.

Task 2: Setting Up AWS IAM User and Policies for EKS Management

Created a dedicated IAM user for managing EKS resources, with the following permissions:

AmazonEC2FullAccess: Manage EC2 instances, including EKS worker nodes.
AmazonEKS_CNI_Policy: Manage EKS networking via CNI plugins.
AmazonEKSClusterPolicy: Full control over EKS cluster operations.
AmazonEKSWorkerNodePolicy: Allow worker nodes to join and operate in the cluster.
AWSCloudFormationFullAccess: Deploy infrastructure like EKS clusters using CloudFormation.
IAMFullAccess: Configure roles and policies for EKS.

Additionally, a custom policy was created to grant full control over EKS (eks:* on all resources). This ensures the user has all necessary permissions to set up and manage Kubernetes clusters securely and effectively.

Why This Approach?

Ensures least privilege access while also meeting the specific requirements for managing EKS and EC2 resources.
Helps isolate EKS-related management tasks from other AWS operations.
Facilitates secure and efficient workflows for provisioning, scaling, and managing Kubernetes clusters.

➡️Navigate to credentials and create Access Key and Secret Key for the eks-user:

These keys enable secure access to AWS services, such as creating and managing EKS clusters using Terraform or AWS CLI.

Task 3: EKS Cluster Configuration

Configured the EKS cluster using the following commands:

Sudo apt update

AWSCLI

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
sudo apt install unzip
unzip awscliv2.zip
sudo ./aws/install
aws configure  # provide access key & secret access key created for eks-user

KUBECTL

curl -o kubectl https://amazon-eks.s3.us-west-2.amazonaws.com/1.19.6/2021-01-05/bin/linux/amd64/kubectl
chmod +x ./kubectl
sudo mv ./kubectl /usr/local/bin
kubectl version --short --client

EKSCTL

curl --silent --location "https://github.com/weaveworks/eksctl/releases/latest/download/eksctl_$(uname -s)_amd64.tar.gz" | tar xz -C /tmp
sudo mv /tmp/eksctl /usr/local/bin
eksctl version

Create EKS CLUSTER

eksctl create cluster --name=My-EKS \
                      --region=ap-south-1 \
                      --zones=ap-south-1a,ap-south-1b \
                      --without-nodegroup

When creating an EKS cluster with eksctl*, it uses AWS CloudFormation to set up all the required resources automatically. This simplifies the process by handling the infrastructure provisioning behind the scenes.*

Associate IAM OIDC Provider:

eksctl utils associate-iam-oidc-provider \
    --region ap-south-1 \
    --cluster My-EKS \
    --approve

OIDC (OpenID Connect) is needed for EKS to securely manage access between your Kubernetes workloads and AWS resources. By associating an OIDC provider, you can allow your pods to assume IAM roles without using access keys or secrets, simplifying security.

Create Nodegroup:

eksctl create nodegroup --cluster=My-EKS \
                       --region=ap-south-1 \
                       --name=node2 \
                       --node-type=t3.medium \
                       --nodes=3 \
                       --nodes-min=2 \
                       --nodes-max=4 \
                       --node-volume-size=20 \
                       --ssh-access \
                       --ssh-public-key=DevOps-keypair\
                       --managed \
                       --asg-access \
                       --external-dns-access \
                       --full-ecr-access \
                       --appmesh-access \
                       --alb-ingress-access

🚀Security Groups created for EKS

eks-cluster-sg-My-EKS-1977367316:

Purpose: This security group is typically the main security group assigned to the EKS cluster itself.
Role: It manages the inbound and outbound traffic rules for the entire cluster, including control plane communication and worker nodes.
- Control Plane Traffic: Allows communication between the control plane and worker nodes, ensuring Kubernetes API server access and other control-plane services.
- Cluster-Wide Security: Acts as the default security group for overall cluster security settings.

eksctl-My-EKS-cluster/ClusterSharedNodeSecurityGroup:

Purpose: This is a shared security group for the worker nodes in your EKS cluster.
Role: It defines the security settings for communication between all worker nodes within the cluster and shared resources.
- Internal Communication: Allows worker nodes within the same node group or across node groups to communicate securely (for inter-node communication).
- Networking and Pods: Manages traffic between nodes for services, pods, and other resources, ensuring proper Kubernetes networking functions (e.g., pod-to-pod communication)

eksctl-My-EKS-cluster/ControlPlaneSecurityGroup:

Purpose: This security group is responsible for controlling traffic to and from the EKS control plane (master nodes).
Role: It manages the security of the Kubernetes API server and other control-plane components.
- Control Plane Access: It ensures secure communication between the control plane and the worker nodes in the cluster.
- Management and Monitoring: Allows management traffic from AWS services and EKS controllers to access the control plane for cluster management and orchestration.

eksctl-My-EKS-nodegroup-node2/SSH:

Purpose: This security group provides SSH access to the worker nodes in node group 2 (node2).
Role: It is specifically used to allow SSH access for administrative and management tasks (e.g., troubleshooting, updates, etc.) on the worker nodes in node group 2.
- SSH Management: Ensures secure remote access to the nodes for DevOps engineers or admins when they need to perform tasks directly on the node instances.
- Limited Access: Typically, only specific IPs or ranges are allowed for SSH access, ensuring security for node management.

# Install Java
sudo apt install openjdk-17-jre-headless -y

# Install Jenkins
sudo wget -O /usr/share/keyrings/jenkins-keyring.asc \
  https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key
echo "deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc]" \
  https://pkg.jenkins.io/debian-stable binary/ | sudo tee \
  /etc/apt/sources.list.d/jenkins.list > /dev/null
sudo apt-get update
sudo apt-get install jenkins

Start Jenkins

You can enable the Jenkins service to start at boot with the command:

sudo systemctl enable jenkins

You can start the Jenkins service with the command:

sudo systemctl start jenkins

You can check the status of the Jenkins service using the command:

sudo systemctl status jenkins

# Docker install
sudo apt  install docker.io -y

# Adds the ubuntu user to the docker group, granting it permission to access the Docker daemon
sudo usermod -aG docker ubuntu

# Adds the jenkins user to the docker group, allowing Jenkins (used in pipelines) to execute Docker commands.
sudo usermod -aG docker jenkins

Refreshes the user's group membership without requiring a logout/login
newgrp docker

**This step is necessary because Docker commands typically require root privileges. Adding users to the docker group eliminates the need for sudo while ensuring smooth execution of Docker commands in automation tasks like Jenkins pipelines.**

Now you are ready to access Jenkins interface, follow these steps:

Open a browser and navigate to http://<ec2_public_ip>:8080.

Run, sudo cat /var/lib/jenkins/secrets/initialAdminPassword, retrieve the initial admin password for Jenkins. This password is needed to unlock the Jenkins web interface for the first time.

On the Jenkins setup screen, paste the 53bc8af4b72541039cc6be25fef50676 password from the terminal into the field provided.

Click Install Suggested plugins

Continue with the Jenkins setup process to create an admin user and configure other settings.

Click Available plugins→ Install the below plugins;

Docker, Docker Pipeline, Kubernetes, Kubernetes CLI

Install Docker on Jenkins [ Manage Jenkins→Tools ]

Add Docker Hub and GitHub credentials so that during pipeline execution, Jenkins can securely connect to these tools using the credentials.

Click Add Credentials→Enter username, password, and ID and create it.

Create a new Pipeline

Add the project repository URL and select the Git credentials from the dropdown menu.

Install Multibranch Scan Webhook Trigger Plugin on Jenkins

Steps to Configure "Scan by Webhook"

Enable Webhook Scan:
- Once the plugin is installed, navigate to the section that says Scan Multibranch Pipeline Triggers.
- Look for an option labeled "Scan by webhook" (or similar) and check the box to enable it.
Access the Trigger Token:
- Next to the checkbox, click on the ? icon (help or settings) to reveal the trigger token options.
- Enter a token name (a custom name you define to identify this token, e.g., WebhookScanToken).
Copy the Generated Token:
- After entering the token name, a trigger token (a long alphanumeric string) will be generated.
- Copy this token immediately and save it securely in a safe place (e.g., a password manager or your configuration file).
- JENKINS_URL/multibranch-webhook-trigger/invoke?token=[Trigger token]

Replace the JENKINS_URL with http://13.235.66.235:8080 and replace [Trigger token] it with the custom name you gave earlier.

http://13.235.66.235:8080/multibranch-webhook-trigger/invoke?token=sandhya

So now save the pipeline configuration by saving it.

Steps to Create a Webhook in GitHub:

GitHub repo→settings→Webhooks→add webhook

Payload URL: Paste the URL where you want the webhook to send the data (JENKINS_URL/multibranch-webhook-trigger/invoke?token=[Trigger token]).
Content type: Select application/json.
Events: Just the push event and Click 🖱️Add Webhook

Once you add the webhook, you will see that all the Jenkinsfiles in all the branches will automatically trigger the pipeline (ignore main for now)

Initially the pipelines failed due to the below errors

1️⃣Issue: Jenkins lacked permission to access Docker's Unix socket.

Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock.

Cause

The jenkins user did not have the necessary permissions to access the Docker socket (/var/run/docker.sock). This socket is owned by the root user and the docker group. By default:

Only users in the docker group can access the Docker daemon.
The jenkins user was not part of the docker group.

Fix

Add Jenkins User to the Docker Group
Run the following command to add the jenkins user to the docker group:
```
 sudo usermod -aG docker jenkins
```
Restart Jenkins
After adding the user to the group, restart Jenkins to apply the changes:
```
 sudo systemctl restart jenkins
```
Verify Docker Access for Jenkins
Test if the jenkins user can access the Docker daemon:
```
 sudo -u jenkins docker ps
```
- If this lists the running containers, the permissions are correctly configured.
Set Proper Permissions on Docker Socket (Optional)
Ensure the Docker socket has proper permissions for security:
```
 sudo chmod 660 /var/run/docker.sock
```
This ensures that only root and docker group members can access the Docker socket.
Retry the Jenkins Pipeline
Trigger the Jenkins pipeline again. The docker login and subsequent stages should now succeed.

Create Service Account, Role & Assign that role, And create a secret for Service Account and generate a Token

🚀 Kubernetes RBAC Configuration for Jenkins Service Account

RBAC (Role-Based Access Control) in Kubernetes is a security system that controls which users or service accounts can perform specific actions in the cluster. It ensures that only authorized accounts have the necessary permissions, reducing the risk of unauthorized access or changes.

Service Account: Creates a Jenkins service account in the webapps namespace.
Role: Defines a role named app-role that allows Jenkins to perform actions on resources like pods, deployments, and services in the webapps namespace.
RoleBinding: Binds the app-role to the jenkins service account, giving it the necessary permissions in the webapps namespace.
ClusterRole: Creates a cluster-wide role named jenkins-cluster-role that allows Jenkins to manage persistent volumes.
ClusterRoleBinding: Binds the jenkins-cluster-role to the jenkins service account, granting it cluster-wide permissions.
Token Generation: Generates an authentication token for the jenkins service account.

⚙️ Steps for Configuring RBAC and Secrets in the EKS Cluster:

To access the EKS cluster using the Jenkins pipeline, we need Jenkins to have access to EKS. For that, we will be creating the Service Account in the webapps namespace, which Jenkins will use to access the EKS resources.

1️⃣Creating a namespace and a Service Account

kubectl create ns webapps


vi svc.yml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins
  namespace: webapps

2️⃣Create an Apply Role

vi role.yml

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: app-role
  namespace: webapps
rules:
  - apiGroups:
        - ""
        - apps
        - autoscaling
        - batch
        - extensions
        - policy
        - rbac.authorization.k8s.io
    resources:
      - pods
      - componentstatuses
      - configmaps
      - daemonsets
      - deployments
      - events
      - endpoints
      - horizontalpodautoscalers
      - ingress
      - jobs
      - limitranges
      - namespaces
      - nodes
      - pods
      - persistentvolumes
      - persistentvolumeclaims
      - resourcequotas
      - replicasets
      - replicationcontrollers
      - serviceaccounts
      - services
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

3️⃣Bind the role to the service account

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: app-rolebinding
  namespace: webapps 
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: app-role 
subjects:
- namespace: webapps 
  kind: ServiceAccount
  name: jenkins

4️⃣Create a secret

vi secret.yml

Click Create Token and replace the myserviceaccount(replace your serviceaccount: jenkins)

To store the token as a secret text in Jenkins, follow these steps:

Run kubectl describe secret mysecretname -n webapps and copy the token.

A long JWT token is used to authenticate the service account. This token is used by the jenkins service account for API access within the webapps namespace.

Copy the token and save it in a safe place

Log in to Jenkins: Go to http://<jenkins_public_ip>:8080 and log in.
Add Secret Text:
- Navigate to Manage Jenkins > Manage Credentials.
- Select (global) and click Add Credentials.
- In the Kind dropdown, choose Secret text and paste the token.
- Add an ID and Description, then Save

Task 4: Create a CD pipeline for deploying the app to Kubernetes & verify the deployment

I created a dummy pipeline for the Jenkinsfile (to deploy to k8s and verify deployment) using Groovy syntax and copied the Jenkinsfile to my GitHub main branch. Once saved, you can see the main file is triggered in Jenkins. Once done with copying the Jenkisnfile to GitHub, delete the dummy pipeline.

To view the Jenkins file & deployment-service.yml for CD - https://github.com/Sandhyagito/11-Microservices-CICD-Pipeline-DevOps-Project-/blob/main/Jenkinsfile

🚀The console output of the main Pipeline

🚀Verify the Kubernetes Deployment

Once your Jenkins pipeline is triggered, it should deploy microservices to Kubernetes. Check the services created for internal and external communication.

ClusterIP Services
These services (adservice, cartservice, checkoutservice, etc.) are created to allow internal communication between pods in the cluster. They are only accessible using their Cluster-IP (10.100.x.x) and are not exposed to the outside world.
NodePort Service
The frontend service exposes the application on a specific port on each Kubernetes node. You can access it externally using <NodeIP>:31505.
LoadBalancer Service
The frontend-external service is set to expose the application via a cloud provider's load balancer. The EXTERNAL-IP will provide a public-facing URL for the application.

Access the frontend application via the external IP (Load Balancer DNS name).

Conclusion:

In this project, I successfully set up a CI/CD pipeline using Jenkins, Docker, Kubernetes, and GitHub, focusing on automating the deployment of a microservices-based application to a Kubernetes cluster. I integrated webhooks to trigger Jenkins builds automatically on code changes, ensuring continuous integration and delivery.

I encountered and resolved challenges such as setting the correct permissions for Docker and ensuring Jenkins had the necessary access to Docker and Kubernetes resources. By implementing RBAC and creating a service account with the right roles and permissions in Kubernetes, I ensured that Jenkins could seamlessly interact with the cluster for deployments.

The deployment included both ClusterIP, NodePort, and LoadBalancer services, which enabled me to manage internal, external, and public access to the microservices deployed in the Kubernetes cluster.

Overall, the project demonstrated my ability to integrate various DevOps tools and practices, such as Docker for containerization, Kubernetes for orchestration, and Jenkins for automation, while addressing real-world challenges such as securing Jenkins' access and managing complex infrastructure setups.

Command Palette