Search This Blog

Thursday 25 February 2021

Elastic Cloud with Observability 7.11 Using Rancher k3d for K8s

Started working locally with rancher's k3d to spin up K8s clusters so I thought why not use Elastic Cloud with Elastic Observability to monitor this local K8s cluster and even run container workloads that use Elastic APM to monitor the application through tracing.

A successful Kubernetes monitoring solution has a few requirements:
  • Monitors all layers of your technology stack, including:
    • The host systems where Kubernetes is running.
    • Kubernetes core components, nodes, pods, and containers running within the cluster.
    • All of the applications and services running in Kubernetes containers.
  • Automatically detects and monitors services as they appear dynamically.
  • Provides a way to correlate related data so that you can group and explore related metrics, logs, and other observability data.

Some basic steps to get this running as per below. This was tested with Elastic Stack 7.11 using the Elastic Cloud Service

Pre Steps

1. I have my Elastic stack running using Elastic Cloud as per the screen shot below

2. I have k3d installed on my Mac as follows

$ k3d --version
k3d version v4.0.0
k3s version latest (default)

3. You will need the kubectl CLI as well

Steps

1. First let's start a K8s cluster as follows. I have created a 5 node K8s cluster with 1 master node but you can reduce the number of worker nodes if you don't have the memory to support this

$ k3d cluster create elastic-k3d --servers 1 --agents 5

2. Start your k3d cluster as follows if not already started

$ k3d cluster start elastic-k3d
INFO[0000] Starting cluster 'elastic-k3d'
INFO[0000] Starting Node 'k3d-elastic-k3d-agent-4'
INFO[0000] Starting Node 'k3d-elastic-k3d-agent-3'
INFO[0000] Starting Node 'k3d-elastic-k3d-agent-2'
INFO[0001] Starting Node 'k3d-elastic-k3d-agent-1'
INFO[0001] Starting Node 'k3d-elastic-k3d-agent-0'
INFO[0001] Starting Node 'k3d-elastic-k3d-server-0'
INFO[0002] Starting Node 'k3d-elastic-k3d-serverlb'

3. Let's list out our nodes just to verify what you created

$ k get nodes
NAME                       STATUS   ROLES                  AGE   VERSION
k3d-elastic-k3d-server-0   Ready    control-plane,master   34h   v1.20.2+k3s1
k3d-elastic-k3d-agent-4    Ready    <none>                 34h   v1.20.2+k3s1
k3d-elastic-k3d-agent-1    Ready    <none>                 34h   v1.20.2+k3s1
k3d-elastic-k3d-agent-3    Ready    <none>                 34h   v1.20.2+k3s1
k3d-elastic-k3d-agent-2    Ready    <none>                 34h   v1.20.2+k3s1
k3d-elastic-k3d-agent-0    Ready    <none>                 34h   v1.20.2+k3s1

Now it's time to "Monitor Kubernetes: Observe the health and performance of your Kubernetes deployments" To do that we could just follow this tutorial as shown below

https://www.elastic.co/guide/en/observability/7.11/monitor-kubernetes.html

This tutorial will walk you through how to do the following

Deploy filebeat into your k3d cluster
Deploy metricbeat into your k3d cluster
Deploy an application which is instrumented using the Elastic APM agent 

4. When installing filebeat and metricbeat make sure you add your Elastic Cloud credentials which can be obtained from the Elastic Cloud deployments page and would have also been provided as a XLS file to download once you create your deployment

filebeat-kubernetes.yaml

    processors:
      - add_cloud_metadata:
      - add_host_metadata:

    cloud.id: pas-K8s:{PASSWORD}
    cloud.auth: elastic:{PASSWORD}

metricbeat-kubernetes.yaml (Notice how I have added a kubernetes_metadata processor)

    processors:
      - add_host_metadata:
      - add_kubernetes_metadata:

    cloud.id: pas-K8s:{PASSWORD}
    cloud.auth: elastic:{PASSWORD}

5. Once installed filebeat and metricbeat PODS should be running as per the output below

Filebeat

$ kubectl get pods -n kube-system -l k8s-app=filebeat
NAME             READY   STATUS    RESTARTS   AGE
filebeat-m8s6s   1/1     Running   2          32h
filebeat-g86vf   1/1     Running   2          32h
filebeat-bj548   1/1     Running   2          32h
filebeat-5cpcn   1/1     Running   2          32h
filebeat-nwj2h   1/1     Running   2          32h
filebeat-4hs8j   1/1     Running   2          32h

Metricbeat

$ kubectl get pods -n kube-system -l k8s-app=metricbeat
NAME               READY   STATUS    RESTARTS   AGE
metricbeat-pbfvs   1/1     Running   2          32h
metricbeat-v5n8l   1/1     Running   2          32h
metricbeat-cdfz9   1/1     Running   2          32h
metricbeat-z85g8   1/1     Running   2          32h
metricbeat-4fxhh   1/1     Running   2          32h
metricbeat-g25lh   1/1     Running   2          32h

6. At this point it's worth heading to Kibana Obserability page and from there you will see we have logs and metrics from our local k3d K8s cluster as shown below


7. Now click on Metrics view and you can clearly see each of your k3d K8s nodes and by clicking on any node you get a very convenient view of metrics, process, metadata and more all from a single page. This was one of the new Elastic 7.11 features for Observability



8. Finally to deploy a application to this K8s cluster and have Elastic APM instrument it's activity you can either use the example in the tutorial link provided above OR just follow this very simple example to get this done

https://github.com/papicella/elastic-customer-api-rest



More Information

k3d Home Page
k3d 

Elastic Cloud Service
Elastic Cloud 

No comments: