K3S: Monitoring & Logging

Setting Up Prometheus for Cluster Monitoring#

You wouldn’t drive a car without a dashboard, so why would you run a Kubernetes cluster without monitoring? Enter Prometheus—your go-to tool for scraping metrics and making sure your cluster isn’t burning down.

What is Prometheus, and why use it for Kubernetes monitoring?#

Prometheus is an open-source monitoring system that collects metrics in real-time. It’s like having a personal assistant that tells you when your cluster is on fire—before it actually explodes.

Deploying Prometheus in a K3s cluster#

kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/main/bundle.yaml

Verifying Prometheus components#

kubectl get pods -n monitoring

Configuring Prometheus to collect Kubernetes metrics#

Once installed, Prometheus will start scraping data from your cluster. You can tweak the config to make sure it collects the right metrics.

Viewing cluster metrics in the Prometheus UI#

To access the Prometheus dashboard:

kubectl port-forward svc/prometheus 9090:9090 -n monitoring

Now, open http://localhost:9090 and enjoy your beautiful, raw metrics.

Configuring Grafana Dashboards#

Prometheus is great, but staring at raw metrics is a skill reserved for robots. That’s where Grafana comes in—it makes your data look good while saving your sanity.

Deploying Grafana on K3s#

kubectl apply -f https://raw.githubusercontent.com/grafana/grafana/main/deploy/kubernetes/grafana-deployment.yaml

Accessing the Grafana dashboard#

kubectl port-forward svc/grafana 3000:3000 -n monitoring

Head over to http://localhost:3000, log in (default user: admin, password: admin), and pretend you know what you’re doing.

Importing Kubernetes Monitoring Dashboards#

You can import prebuilt dashboards from Grafana’s website. Search for Kubernetes Monitoring and plug it into your setup.

Connecting Prometheus as a data source in Grafana#

In Grafana, go to Settings → Data Sources → Add Data Source → Prometheus, then enter your Prometheus URL (http://prometheus.monitoring.svc.cluster.local:9090).

Managing Logs using Fluentd or Loki#

Logs are your best friend when debugging—unless you ignore them, in which case they become your worst nightmare.

What is Fluentd, and how it helps in log collection#

Fluentd collects logs from multiple sources and ships them wherever you want. Think of it as Kubernetes’ gossip system but for useful information.

Deploying Fluentd as a logging agent#

kubectl apply -f fluentd-daemonset.yaml

Setting up Loki for log aggregation#

Unlike Prometheus, which handles metrics, Loki stores logs and works hand-in-hand with Grafana.

kubectl apply -f https://raw.githubusercontent.com/grafana/loki/main/deploy/kubernetes/loki.yaml

Configuring Grafana to visualize logs from Loki#

In Grafana, add Loki as a data source and create some queries to filter logs. Congratulations, you now have a live log dashboard!

Troubleshooting Common Issues in K3s#

Because Kubernetes loves throwing cryptic error messages, here’s how to make sense of them.

Debugging Kubernetes workloads#

kubectl logs <pod-name>
kubectl describe pod <pod-name>

If your Pod is misbehaving, check the logs. If the logs aren’t helpful, kubectl describe might give you a clue.

Monitoring resource usage with kubectl top#

kubectl top nodes
kubectl top pods

If your cluster is slow, check if your nodes are gasping for resources.

Checking K3s system logs#

journalctl -u k3s -f

Useful when your cluster decides to act up and you need a deeper look.

Common issues and fixes#

• Pods stuck in Pending state → Check node resources, your scheduler might be ignoring them.
• CrashLoopBackOff errors → Something is crashing in a loop. Check the logs and fix the root cause.
• Service not reachable → Double-check networking and Ingress settings before throwing your laptop out the window.

Hands-On Exercise#

Enough theory—time for action:

• Deploy Prometheus and Grafana in K3s to monitor your cluster.
• Configure Fluentd or Loki for centralized logging to track what’s happening.
• Create and visualize a custom Grafana dashboard for K3s metrics.
• Troubleshoot a failing application using kubectl logs and kubectl describe.

Once you master this, you’ll be the person everyone calls when things go wrong. Whether that’s a good or bad thing is up to you. 🚀