Prometheus and Grafana On EKS Cluster Using

Important Links -

1) To setup Prometheus - Deploying Prometheus

2) Managing the Amazon EBS CSI driver as an Amazon EKS add-on

3) Creating an IAM OIDC provider for your cluster

4) Setup Grafana

-> Here we are going to setup Prometheus and Grafana on eks cluster using helm.

Pre-requisites:

1) EKS Cluster needs to be up and running. Click here to learn how to setup EKS

cluster in AWS cloud using eksctl

2) Install Helm3

Setup Prometheus

Steps to setup Prometheus on eks cluster.

1) Create a namespace

kubectl create namespace prometheus

2) Add the prometheus-community chart repository.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

3) Update the repo

helm repo update

4) Deploy Prometheus.

helm upgrade -i prometheus prometheus-community/prometheus \
    --namespace prometheus \
    --set alertmanager.persistentVolume.storageClass="gp2",server.persistentVolume.storageClass="gp2"

-> When we run above command on the eks cluster then we get the below output as-

-> Here we can see that some pods are in pending state. Pending pods are alertmanagar and prometheus-server.

-> When we run below command then we can see the issues due to which pods are not coming up-

 kubectl describe pods prometheus-server-7b498b9998-t55cl -n prometheus

-> Here we can see that the pod is not able to schedule due to volume binding.

-> These pods are not able to bind with PV and PVC.

-> To resolve this issue we have to run the following commands

1) Create an IAM OIDC identity provider for your cluster with eksctl.

i) Determine whether you have an existing IAM OIDC provider for your cluster.

-> Replace cluster_name with your cluster name

export cluster_name=mycluster
oidc_id=$(aws eks describe-cluster --name $cluster_name --query "cluster.identity.oidc.issuer" --output text | cut -d '/' -f 5)

ii) Determine whether an IAM OIDC provider with your cluster's ID is already in your account.

aws iam list-open-id-connect-providers | grep $oidc_id | cut -d "/" -f4

-> If output is returned, then you already have an IAM OIDC provider for your cluster and you can skip the next step. If no output is returned, then you must create an IAM OIDC provider for your cluster.

iii) Create an IAM OIDC identity provider for your cluster with the following command.

eksctl utils associate-iam-oidc-provider --cluster $cluster_name --approve

2) Create your Amazon EBS CSI plugin IAM role with eksctl

i) At the place of --cluster, enter your cluster name

eksctl create iamserviceaccount \
    --name ebs-csi-controller-sa \
    --namespace kube-system \
    --cluster mycluster \
    --role-name AmazonEKS_EBS_CSI_DriverRole \
    --role-only \
    --attach-policy-arn arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy \
    --approve

3) Adding the Amazon EBS CSI driver add-on

-> Here you have to replace your cluster name and your aws account id

eksctl create addon --name aws-ebs-csi-driver --cluster mycluster --service-account-role-arn arn:aws:iam::046376948803:role/AmazonEKS_EBS_CSI_DriverRole --force

-> Now when we again run the below command we will get the output as -

aws iam list-open-id-connect-providers | grep $oidc_id | cut -d "/" -f4

-> Now we can check that pods are coming up or not

kubectl get pods -n prometheus

-> Now our pods are in running state.

-> Now for accessing the prometheus server, we need to expose the service to access this prometheus server.

-> We can check it by running the command as

kubectl get svc -n prometheus

-> Here we can see that prometheus-server is exposed via ClusterIP.

-> If we want to access this server outside the cluster then, we have to expose it either using NodePort or LoadBalancer.

-> Here we will expose it via LoadBalancer.

-> Here we need to edit the prometheus-server service and instead of ClusterIP, we will use LoadBalancer.

Run the command as -

kubectl edit svc prometheus-server -n prometheus

-> Change this ClusterIP to LoadBalancer

-> Now if we see the service, then we can see that it is exposed using LoadBalancer.

-> Now we can access the prometheus-server using this loadbalancer url on the browser.

-> We can see the prometheus-server in the browser as -

Setup Grafana

To setup the Grafana using Helm, we need to do the following steps -

1) Add Grafana Helm Repo

helm repo add grafana https://grafana.github.io/helm-charts

2) Update the repo

helm repo update

3) Create a file as grafana.yaml

vi grafana.yaml

4) Paste this content in grafana.yaml file

datasources:
  datasources.yaml:
    apiVersion: 1
    datasources:
    - name: Prometheus
      type: prometheus
      url: http://prometheus-server.prometheus.svc.cluster.local
      access: proxy
      isDefault: true

5) Create a namespace called grafana

kubectl create namespace grafana

6) Install grafana using helm

-> At the --values, we have to give the path where we have createa grafana.yaml file.

-> By default username is admin and here password is EKS!sAWSome

helm install grafana grafana/grafana \
    --namespace grafana \
    --set persistence.storageClassName="gp2" \
    --set persistence.enabled=true \
    --set adminPassword='EKS!sAWSome' \
    --values ./grafana.yaml \
    --set service.type=LoadBalancer

7) Run the following command to check if Grafana is deployed properly:

kubectl get all -n grafana

-> Here we can see that our grafana pod is deployed and running. It is exposed via LoadBalancer Service.

-> Using this LoadBalancer url we can access the Grafana Dashboard as-

-> Now we have logged in to the Grafana as -

-> Once we logged In then we have to add the Data Source.

-> Now click on the above marked area to add the Data Source

-> We have multiple Data Sources, But here we are using Prometheus.

-> So click on Prometheus

-> Here we have to add the prometheus server url where it is running.

-> After adding prometheus-server url, click on Save & test. And if url is correct then you will get a message that Successfully queried the Prometheus API.

-> Now click on Home and then Click on Dashboard