Monitoring Redis with Prometheus and Grafana
I'm having the opportunity to work more and more with kubernetes at home and at my job. Recently, I was challenged to deploy Prometheus+Grafana stack for a Redis application. All of them running on top of a kubernetes cluster. Or like my manager says, I'm going thru my "Kubernetes Dojo".
As per google definition:
It is an honored place where students and masters come together for deliberate practice to develop their > > skills. In organizations, a Dojo is where teams learn complex skills, practices, and tools that help them > develop better products faster. Dojos use immersive learning.
Besides that, I also liked the idea of having someone caring for my learning and bringing me challenges that are aligned on what I want to learn and the company needs.
The Dojo approach is more principle-driven than process-driven. It is a methodology that grows and > evolves based on experience and experimentation. There are a few principles common to most Dojos: core > tenets that define the Dojo as a concept.
- Prioritize learning over delivery.
- Align learning to the team’s actual work.
- Focus on safety, not on failure.
- Work small and exploit feedback.
- Make everything visible.
- Use the buddy system.
- Success depends on commitment.
Use Case
Image you have the following cenario: A Redis cluster that is used from application to subscribe and publish messages into a some kind of Queue. Multiple applications will be reading and writing data to the Redis Cluster.
At some point, you need to scrape your Redis Cluster metrics to your Prometheus+Grafana stack.
All of these applications are currently packed all together and can be easily installed Helm charts. However, new users would prefer to learn how to create those manifests manually and deploy them into a Kubernetes cluster.
Important
-
This post and github repository are used for learning purposes and probably not following yet the best practices.
-
The whole starting point and reference is the original creator of rq-exporter tool. At the original github repository, you'll find a docker-compose.yml file. That file was my starting point using with kompose convert to generate kubernetes manifests. From that point, I started my changes.
Python RQ Prometheus Exporter tool: https://github.com/mdawar/rq-exporter
Pierre Mdawar repo: https://github.com/mdawar
If you just want a excelent starting point with a good cenario for testing, I'd recommend go with the Docker Compose instructions from Pierre's page.
K8s Manifests
The whole idea was learn how to write kubernetes manifests. From there, I knew the Kompose tool, which uses a docker-compose.yml file to generate k8s manifests. It's not perfect, but a very good starting point. From there, just google the missing parts or whatever you want to modify or add.
Applications
-
Prometheus
For example, the kompose convert only generate the Persistent Volume Claims, I was trying to convert it to ConfigMap properties to make easier to recreate the PODs.
-
Grafana
I'm still working to convert Grafana configuration files to a ConfigMap setup, for now it's still using the PVCs created by kompose convert.
-
Redis
Redis is a standalong execution and it's using standard 123456 password. Soon I should improve the manifests with secrets, just lazy..
-
rq-exporter
Exporter for Redis can be found here: https://github.com/mdawar/rq-exporter. As mentioned earlier, the whole idea of these manifest files are based on rq-exporter docker-compose file.
-
rq-dashboard
This tool allows visualization of simple metrics generated by Redis server. It's based on Dockerhub image eoranged/rq-dashboard:legacy.
- rq-enqueue and rq-worker
From rq-exporter docker compose example, it starts Enqueue and Worker pods to generate data into the Redis Cluster. They keep running on background.
Important to mention here that I generated a docker image (from original Dockerfile) and included the remaining python files. In that way, my image dszortyka/rq-exporter:latest contains all the python files needed for the example cenario.
You can find the manifests resulted from this exercise on: https://github.com/dszortyka/k8s-redis-exporter-graf-prom
cd k8s/
kubectl apply -f redis/k8s \
-f rq-exporter/k8s \
-f grafana/k8s \
-f prometheus/k8s \
-f rq-enqueue/k8s \
-f rq-worker/k8s \
-f rq-dashboard/k8s
~/DEV/k8s-redis-exporter-graf-prom/k8s main ❯
kubectl apply -f redis/k8s \
-f rq-exporter/k8s \
-f grafana/k8s \
-f prometheus/k8s \
-f rq-enqueue/k8s \
-f rq-worker/k8s \
-f rq-dashboard/k8s
namespace/redis created
deployment.apps/redis created
service/redis created
namespace/rq-exporter created
deployment.apps/rq-exporter created
service/rq-exporter created
namespace/grafana created
configmap/grafana-server-conf created
persistentvolumeclaim/grafana-claim0 created
persistentvolumeclaim/grafana-claim1 created
persistentvolumeclaim/grafana-claim2 created
persistentvolumeclaim/grafana-data created
deployment.apps/grafana created
service/grafana created
namespace/prometheus created
configmap/prometheus-server-conf created
service/prometheus created
deployment.apps/prometheus created
persistentvolumeclaim/prometheus-claim0 created
namespace/enqueue created
persistentvolumeclaim/enqueue-claim0 created
deployment.apps/enqueue created
namespace/rq-worker created
deployment.apps/rq-worker created
namespace/rq-dashboard created
deployment.apps/rq-dashboard created
service/rq-dashboard created
tree k8s/
k8s
├── grafana
│ └── k8s
│ ├── 000-grafana-namespace.yaml
│ ├── 010-grafana-configmap.yaml
│ ├── files
│ │ ├── grafana-dashboards.yml
│ │ ├── grafana-datasources.yml
│ │ └── rq-dashboard.json
│ ├── grafana-claim0-persistentvolumeclaim.yaml
│ ├── grafana-claim1-persistentvolumeclaim.yaml
│ ├── grafana-claim2-persistentvolumeclaim.yaml
│ ├── grafana-data-persistentvolumeclaim.yaml
│ ├── grafana-deployment.yaml
│ └── grafana-service.yaml
├── prometheus
│ └── k8s
│ ├── 000-prometheus-namespace.yaml
│ ├── 010-prometheus-configmap.yaml
│ ├── 020-prometheus-service.yaml
│ ├── 040-prometheus-deployment.yaml
│ └── prometheus-claim0-persistentvolumeclaim.yaml
├── redis
│ └── k8s
│ ├── 000-redis-namespace.yaml
│ ├── redis-deployment.yaml
│ └── redis-service.yaml
├── rq-dashboard
│ └── k8s
│ ├── 000-rq-dashboard-namespace.yaml
│ ├── rq-dashboard-deployment.yaml
│ └── rq-dashboard-service.yaml
├── rq-enqueue
│ └── k8s
│ ├── 000-enqueue-namespace.yaml
│ ├── 010-enqueue-pvc-claim0.yaml
│ └── enqueue-deployment.yaml
├── rq-exporter
│ ├── docker-compose.yml
│ ├── image
│ │ ├── Dockerfile
│ │ ├── docker-compose
│ │ │ └── project
│ │ │ ├── custom.py
│ │ │ ├── enqueue.py
│ │ │ ├── jobs.py
│ │ │ └── queues.py
│ │ ├── requirements.txt
│ │ ├── rq_exporter
│ │ │ ├── __init__.py
│ │ │ ├── __main__.py
│ │ │ ├── __version__.py
│ │ │ ├── collector.py
│ │ │ ├── config.py
│ │ │ ├── exporter.py
│ │ │ └── utils.py
│ │ └── setup.py
│ └── k8s
│ ├── 000-rq-exporter-namespace.yaml
│ ├── rq-exporter-deployment.yaml
│ └── rq-exporter-service.yaml
└── rq-worker
└── k8s
├── 000-rq-worker-namespace.yaml
└── rq-worker-deployment.yaml
NAMESPACE NAME READY STATUS RESTARTS AGE
default pod/busybox 1/1 Running 90 (16m ago) 3d18h
enqueue pod/enqueue-84658d757b-2mqkt 1/1 Running 3 (31s ago) 61s
grafana pod/grafana-54fc684bbc-pqpwp 1/1 Running 0 62s
kube-system pod/calico-kube-controllers-c44b4545-chf7s 1/1 Running 2 (2d17h ago) 3d18h
kube-system pod/calico-node-vhnfl 1/1 Running 4 (2d17h ago) 3d18h
kube-system pod/coredns-6d4b75cb6d-r8m2k 1/1 Running 2 (2d17h ago) 3d18h
kube-system pod/etcd-minikube 1/1 Running 3 (2d17h ago) 3d18h
kube-system pod/kube-apiserver-minikube 1/1 Running 4 (2d17h ago) 3d18h
kube-system pod/kube-controller-manager-minikube 1/1 Running 4 (2d17h ago) 3d18h
kube-system pod/kube-proxy-svfbt 1/1 Running 3 (2d17h ago) 3d18h
kube-system pod/kube-scheduler-minikube 1/1 Running 3 (2d17h ago) 3d18h
kube-system pod/storage-provisioner 1/1 Running 17 (11h ago) 3d18h
prometheus pod/prometheus-597b4967c-9vlvk 1/1 Running 0 62s
redis pod/redis-75c4f49968-75824 1/1 Running 0 63s
rq-dashboard pod/rq-dashboard-f95dcdbf-q9lvk 1/1 Running 0 61s
rq-exporter pod/rq-exporter-5b9cf74dc4-s6kb2 1/1 Running 3 (29s ago) 62s
rq-worker pod/rq-worker-56d8cbb45f-5l59b 1/1 Running 2 (40s ago) 61s
rq-worker pod/rq-worker-56d8cbb45f-vrxcv 1/1 Running 2 (42s ago) 61s
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
bitbucket service/bitbucket NodePort 10.105.94.217 <none> 7990:30990/TCP,7999:30999/TCP 3d17h
bitbucket service/postgresql ClusterIP None <none> 5432/TCP 3d18h
default service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 3d18h
grafana service/grafana ClusterIP 10.102.22.56 <none> 3000/TCP 62s
kube-system service/kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 3d18h
prometheus service/prometheus ClusterIP 10.97.158.165 <none> 9090/TCP 62s
redis service/redis ClusterIP None <none> 6379/TCP 63s
rq-dashboard service/rq-dashboard ClusterIP 10.104.21.144 <none> 9181/TCP 61s
rq-exporter service/rq-exporter ClusterIP 10.101.203.160 <none> 9726/TCP 62s
NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
kube-system daemonset.apps/calico-node 1 1 1 1 1 kubernetes.io/os=linux 3d18h
kube-system daemonset.apps/kube-proxy 1 1 1 1 1 kubernetes.io/os=linux 3d18h
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
enqueue deployment.apps/enqueue 1/1 1 1 61s
grafana deployment.apps/grafana 1/1 1 1 62s
kube-system deployment.apps/calico-kube-controllers 1/1 1 1 3d18h
kube-system deployment.apps/coredns 1/1 1 1 3d18h
prometheus deployment.apps/prometheus 1/1 1 1 62s
redis deployment.apps/redis 1/1 1 1 63s
rq-dashboard deployment.apps/rq-dashboard 1/1 1 1 61s
rq-exporter deployment.apps/rq-exporter 1/1 1 1 63s
rq-worker deployment.apps/rq-worker 2/2 2 2 61s
NAMESPACE NAME DESIRED CURRENT READY AGE
enqueue replicaset.apps/enqueue-84658d757b 1 1 1 61s
grafana replicaset.apps/grafana-54fc684bbc 1 1 1 62s
kube-system replicaset.apps/calico-kube-controllers-c44b4545 1 1 1 3d18h
kube-system replicaset.apps/coredns-6d4b75cb6d 1 1 1 3d18h
prometheus replicaset.apps/prometheus-597b4967c 1 1 1 62s
redis replicaset.apps/redis-75c4f49968 1 1 1 63s
rq-dashboard replicaset.apps/rq-dashboard-f95dcdbf 1 1 1 61s
rq-exporter replicaset.apps/rq-exporter-5b9cf74dc4 1 1 1 62s
rq-worker replicaset.apps/rq-worker-56d8cbb45f 2 2 2 61s