VMware Tanzu Kubernetes Grid Integrated Edition (formerly known as
VMware Enterprise PKS) is a Kubernetes-based container solution with
advanced networking, a private container registry, and life cycle
management.
In this post I show how to get
Elastic Cloud on Kubernetes (ECK) up and running on VMware Tanzu Kubernetes Grid Integrated Edition and how to access it using a Spring Boot Application using Spring Data Elasticsearch.
With
ECK, users now have a seamless way of deploying, managing, and operating the Elastic Stack on Kubernetes.
If you have a K8s cluster that's all you need to follow along.
Steps
1. Let's install ECK on our cluster we do that as follows
Note: There is a 1.1 version as the latest BUT I installing a slightly older one here
$ kubectl apply -f https://download.elastic.co/downloads/eck/1.0.1/all-in-one.yaml
2. Make sure the operator is up and running as shown below
$ kubectl get all -n elastic-system
NAME READY STATUS RESTARTS AGE
pod/elastic-operator-0 1/1 Running 0 26d
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/elastic-webhook-server ClusterIP 10.100.200.55 <none> 443/TCP 26d
NAME READY AGE
statefulset.apps/elastic-operator 1/1 26d
3. We can also see a CRD for Elasticsearch as shown below.
elasticsearches.elasticsearch.k8s.elastic.co
$ kubectl get crd
NAME CREATED AT
apmservers.apm.k8s.elastic.co 2020-06-17T00:37:32Z
clusterlogsinks.pksapi.io 2020-06-16T23:04:43Z
clustermetricsinks.pksapi.io 2020-06-16T23:04:44Z
elasticsearches.elasticsearch.k8s.elastic.co 2020-06-17T00:37:33Z
kibanas.kibana.k8s.elastic.co 2020-06-17T00:37:34Z
loadbalancers.vmware.com 2020-06-16T22:51:52Z
logsinks.pksapi.io 2020-06-16T23:04:43Z
metricsinks.pksapi.io 2020-06-16T23:04:44Z
nsxerrors.nsx.vmware.com 2020-06-16T22:51:52Z
nsxlbmonitors.vmware.com 2020-06-16T22:51:52Z
nsxlocks.nsx.vmware.com 2020-06-16T22:51:51Z
4. We are now ready to create our first Elasticsearch cluster. To do that create a file YML file as shown below
create-elastic-cluster-from-operator.yaml
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: quickstart
spec:
version: 7.7.0
http:
service:
spec:
type: LoadBalancer # default is ClusterIP
tls:
selfSignedCertificate:
disabled: true
nodeSets:
- name: default
count: 2
volumeClaimTemplates:
- metadata:
name: elasticsearch-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
config:
node.master: true
node.data: true
node.ingest: true
node.store.allow_mmap: false
From the YML a few things to note:
- We are creating two pods for our Elasticsearch cluster
- We are using a K8s LoadBalancer to expose access to the cluster through HTTP
- We are using version 7.7.0 but this is not the latest Elasticsearch version
- We have disabled the use of TLS given this is just a demo
5. Apply that as shown below.
$ kubectl apply -f create-elastic-cluster-from-operator.yaml
6. After about a minute we should have our Elasticsearch cluster running. The following commands show that
$ kubectl get elasticsearch
NAME HEALTH NODES VERSION PHASE AGE
quickstart green 2 7.7.0 Ready 47h
$ kubectl get all -n default
NAME READY STATUS RESTARTS AGE
pod/quickstart-es-default-0 1/1 Running 0 47h
pod/quickstart-es-default-1 1/1 Running 0 47h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.100.200.1 <none> 443/TCP 27d
service/quickstart-es-default ClusterIP None <none> <none> 47h
service/quickstart-es-http LoadBalancer 10.100.200.92 10.195.93.137 9200:30590/TCP 47h
NAME READY AGE
statefulset.apps/quickstart-es-default 2/2 47h
7. Let's deploy a Kibana instance. To do that create a YML as shown below
create-kibana.yaml
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
name: kibana-sample
spec:
version: 7.7.0
count: 1
elasticsearchRef:
name: quickstart
namespace: default
http:
service:
spec:
type: LoadBalancer # default is ClusterIP
8. Apply that as shown below.
$ kubectl apply -f create-kibana.yaml
9. To verify everything is up and running we can run a command as follows
$ kubectl get all
NAME READY STATUS RESTARTS AGE
pod/kibana-sample-kb-f8fcb88d5-jdzh5 1/1 Running 0 2d
pod/quickstart-es-default-0 1/1 Running 0 2d
pod/quickstart-es-default-1 1/1 Running 0 2d
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kibana-sample-kb-http LoadBalancer 10.100.200.46 10.195.93.174 5601:32459/TCP 2d
service/kubernetes ClusterIP 10.100.200.1 <none> 443/TCP 27d
service/quickstart-es-default ClusterIP None <none> <none> 2d
service/quickstart-es-http LoadBalancer 10.100.200.92 10.195.93.137 9200:30590/TCP 2d
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/kibana-sample-kb 1/1 1 1 2d
NAME DESIRED CURRENT READY AGE
replicaset.apps/kibana-sample-kb-f8fcb88d5 1 1 1 2d
NAME READY AGE
statefulset.apps/quickstart-es-default 2/2 2d
10. So to access out cluster we will need to obtain the following which we can do using a script as follows. This was tested on Mac OSX
What do we need?
- Elasticsearch password
- IP address of the LoadBalancer service we created
access.sh
export PASSWORD=`kubectl get secret quickstart-es-elastic-user -o go-template='{{.data.elastic | base64decode}}'`
export IP=`kubectl get svc quickstart-es-http -o jsonpath='{.status.loadBalancer.ingress[0].ip}'`
echo ""
echo $IP
echo ""
curl -u "elastic:$PASSWORD" "http://$IP:9200"
echo ""
curl -u "elastic:$PASSWORD" "http://$IP:9200/_cat/health?v"
Output:
10.195.93.137
{
"name" : "quickstart-es-default-1",
"cluster_name" : "quickstart",
"cluster_uuid" : "Bbpb7Pu7SmaQaCmEY2Er8g",
"version" : {
"number" : "7.7.0",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "81a1e9eda8e6183f5237786246f6dced26a10eaf",
"build_date" : "2020-05-12T02:01:37.602180Z",
"build_snapshot" : false,
"lucene_version" : "8.5.1",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
.....
11. Ideally I would load some data into the Elasticsearch cluster BUT let's do that as part of a sample application using "
Spring Data Elasticsearch". Clone the demo project as shown below.
$ git clone https://github.com/papicella/boot-elastic-demo.git
Cloning into 'boot-elastic-demo'...
remote: Enumerating objects: 36, done.
remote: Counting objects: 100% (36/36), done.
remote: Compressing objects: 100% (26/26), done.
remote: Total 36 (delta 1), reused 36 (delta 1), pack-reused 0
Unpacking objects: 100% (36/36), done.
12. Edit "
./src/main/resources/application.yml" with your details for the Elasticsearch cluster above.
spring:
elasticsearch:
rest:
username: elastic
password: {PASSWORD}
uris: http://{IP}:9200
13. Package as follows
$ ./mvnw -DskipTests package
14. Run as follows
$ ./mvnw spring-boot:run
....
2020-07-14 11:10:11.947 INFO 76260 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path ''
2020-07-14 11:10:11.954 INFO 76260 --- [ main] c.e.e.demo.BootElasticDemoApplication : Started BootElasticDemoApplication in 2.495 seconds (JVM running for 2.778)
....
15. Access application using "
http://localhost:8080/"
16. If we look at our code we will see the data was loaded into the Elasticsearch cluster using a java class called "
LoadData.java". Ideally data should already exist in the cluster but for demo purposes we load some data as part of the Spring Boot Application and clear the data prior to each application run given it's just a demo.
2020-07-14 11:12:33.109 INFO 76277 --- [ main] com.example.elastic.demo.LoadData : Pre loading Car{id='OjThSnMBLjyTRl7lZsDL', make='holden', model='commodore', bodystyles=[BodyStyle{type='2-door'}, BodyStyle{type='4-door'}, BodyStyle{type='5-door'}]}
2020-07-14 11:12:33.584 INFO 76277 --- [ main] com.example.elastic.demo.LoadData : Pre loading Car{id='OzThSnMBLjyTRl7laMCo', make='holden', model='astra', bodystyles=[BodyStyle{type='2-door'}, BodyStyle{type='4-door'}]}
2020-07-14 11:12:34.189 INFO 76277 --- [ main] com.example.elastic.demo.LoadData : Pre loading Car{id='PDThSnMBLjyTRl7lasCC', make='nissan', model='skyline', bodystyles=[BodyStyle{type='4-door'}]}
2020-07-14 11:12:34.744 INFO 76277 --- [ main] com.example.elastic.demo.LoadData : Pre loading Car{id='PTThSnMBLjyTRl7lbMDe', make='nissan', model='pathfinder', bodystyles=[BodyStyle{type='5-door'}]}
2020-07-14 11:12:35.227 INFO 76277 --- [ main] com.example.elastic.demo.LoadData : Pre loading Car{id='PjThSnMBLjyTRl7lb8AL', make='ford', model='falcon', bodystyles=[BodyStyle{type='4-door'}, BodyStyle{type='5-door'}]}
2020-07-14 11:12:36.737 INFO 76277 --- [ main] com.example.elastic.demo.LoadData : Pre loading Car{id='QDThSnMBLjyTRl7lcMDu', make='ford', model='territory', bodystyles=[BodyStyle{type='5-door'}]}
2020-07-14 11:12:37.266 INFO 76277 --- [ main] com.example.elastic.demo.LoadData : Pre loading Car{id='QTThSnMBLjyTRl7ldsDU', make='toyota', model='camry', bodystyles=[BodyStyle{type='4-door'}, BodyStyle{type='5-door'}]}
2020-07-14 11:12:37.777 INFO 76277 --- [ main] com.example.elastic.demo.LoadData : Pre loading Car{id='QjThSnMBLjyTRl7leMDk', make='toyota', model='corolla', bodystyles=[BodyStyle{type='2-door'}, BodyStyle{type='5-door'}]}
2020-07-14 11:12:38.285 INFO 76277 --- [ main] com.example.elastic.demo.LoadData : Pre loading Car{id='QzThSnMBLjyTRl7lesDj', make='kia', model='sorento', bodystyles=[BodyStyle{type='5-door'}]}
2020-07-14 11:12:38.800 INFO 76277 --- [ main] com.example.elastic.demo.LoadData : Pre loading Car{id='RDThSnMBLjyTRl7lfMDg', make='kia', model='sportage', bodystyles=[BodyStyle{type='4-door'}]}
LoadData.java
package com.example.elastic.demo;
import com.example.elastic.demo.indices.BodyStyle;
import com.example.elastic.demo.indices.Car;
import com.example.elastic.demo.repo.CarRepository;
import org.springframework.boot.CommandLineRunner;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import lombok.extern.slf4j.Slf4j;
import static java.util.Arrays.asList;
@Configuration
@Slf4j
public class LoadData {
@Bean
public CommandLineRunner initElasticsearchData(CarRepository carRepository) {
return args -> {
carRepository.deleteAll();
log.info("Pre loading " + carRepository.save(new Car("holden", "commodore", asList(new BodyStyle("2-door"), new BodyStyle("4-door"), new BodyStyle("5-door")))));
log.info("Pre loading " + carRepository.save(new Car("holden", "astra", asList(new BodyStyle("2-door"), new BodyStyle("4-door")))));
log.info("Pre loading " + carRepository.save(new Car("nissan", "skyline", asList(new BodyStyle("4-door")))));
log.info("Pre loading " + carRepository.save(new Car("nissan", "pathfinder", asList(new BodyStyle("5-door")))));
log.info("Pre loading " + carRepository.save(new Car("ford", "falcon", asList(new BodyStyle("4-door"), new BodyStyle("5-door")))));
log.info("Pre loading " + carRepository.save(new Car("ford", "territory", asList(new BodyStyle("5-door")))));
log.info("Pre loading " + carRepository.save(new Car("toyota", "camry", asList(new BodyStyle("4-door"), new BodyStyle("5-door")))));
log.info("Pre loading " + carRepository.save(new Car("toyota", "corolla", asList(new BodyStyle("2-door"), new BodyStyle("5-door")))));
log.info("Pre loading " + carRepository.save(new Car("kia", "sorento", asList(new BodyStyle("5-door")))));
log.info("Pre loading " + carRepository.save(new Car("kia", "sportage", asList(new BodyStyle("4-door")))));
};
}
}
17. Our CarRepository interface is defined as follows
CarRepository.java
package com.example.elastic.demo.repo;
import com.example.elastic.demo.indices.Car;
import org.springframework.data.domain.Page;
import org.springframework.data.domain.Pageable;
import org.springframework.data.elasticsearch.repository.ElasticsearchRepository;
import org.springframework.stereotype.Repository;
@Repository
public interface CarRepository extends ElasticsearchRepository <Car, String> {
Page<Car> findByMakeContaining(String make, Pageable page);
}
18. So let's also via this data using "
curl" and Kibana as shown below.
curl -X GET -u "elastic:{PASSWORD}" "http://{IP}:9200/vehicle/_search?pretty" -H 'Content-Type: application/json' -d'
{
"query": { "match_all": {} },
"sort": [
{ "_id": "asc" }
]
}
'
Output:
{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 10,
"relation" : "eq"
},
"max_score" : null,
"hits" : [
{
"_index" : "vehicle",
"_type" : "_doc",
"_id" : "OjThSnMBLjyTRl7lZsDL",
"_score" : null,
"_source" : {
"_class" : "com.example.elastic.demo.indices.Car",
"make" : "holden",
"model" : "commodore",
"bodystyles" : [
{
"type" : "2-door"
},
{
"type" : "4-door"
},
{
"type" : "5-door"
}
]
},
"sort" : [
"OjThSnMBLjyTRl7lZsDL"
]
},
{
"_index" : "vehicle",
"_type" : "_doc",
"_id" : "OzThSnMBLjyTRl7laMCo",
"_score" : null,
"_source" : {
"_class" : "com.example.elastic.demo.indices.Car",
"make" : "holden",
"model" : "astra",
"bodystyles" : [
{
"type" : "2-door"
},
{
"type" : "4-door"
}
]
},
"sort" : [
"OzThSnMBLjyTRl7laMCo"
]
},
{
"_index" : "vehicle",
"_type" : "_doc",
"_id" : "PDThSnMBLjyTRl7lasCC",
"_score" : null,
"_source" : {
"_class" : "com.example.elastic.demo.indices.Car",
"make" : "nissan",
"model" : "skyline",
"bodystyles" : [
{
"type" : "4-door"
}
]
},
"sort" : [
"PDThSnMBLjyTRl7lasCC"
]
},
{
"_index" : "vehicle",
"_type" : "_doc",
"_id" : "PTThSnMBLjyTRl7lbMDe",
"_score" : null,
"_source" : {
"_class" : "com.example.elastic.demo.indices.Car",
"make" : "nissan",
"model" : "pathfinder",
"bodystyles" : [
{
"type" : "5-door"
}
]
},
"sort" : [
"PTThSnMBLjyTRl7lbMDe"
]
},
{
"_index" : "vehicle",
"_type" : "_doc",
"_id" : "PjThSnMBLjyTRl7lb8AL",
"_score" : null,
"_source" : {
"_class" : "com.example.elastic.demo.indices.Car",
"make" : "ford",
"model" : "falcon",
"bodystyles" : [
{
"type" : "4-door"
},
{
"type" : "5-door"
}
]
},
"sort" : [
"PjThSnMBLjyTRl7lb8AL"
]
},
{
"_index" : "vehicle",
"_type" : "_doc",
"_id" : "QDThSnMBLjyTRl7lcMDu",
"_score" : null,
"_source" : {
"_class" : "com.example.elastic.demo.indices.Car",
"make" : "ford",
"model" : "territory",
"bodystyles" : [
{
"type" : "5-door"
}
]
},
"sort" : [
"QDThSnMBLjyTRl7lcMDu"
]
},
{
"_index" : "vehicle",
"_type" : "_doc",
"_id" : "QTThSnMBLjyTRl7ldsDU",
"_score" : null,
"_source" : {
"_class" : "com.example.elastic.demo.indices.Car",
"make" : "toyota",
"model" : "camry",
"bodystyles" : [
{
"type" : "4-door"
},
{
"type" : "5-door"
}
]
},
"sort" : [
"QTThSnMBLjyTRl7ldsDU"
]
},
{
"_index" : "vehicle",
"_type" : "_doc",
"_id" : "QjThSnMBLjyTRl7leMDk",
"_score" : null,
"_source" : {
"_class" : "com.example.elastic.demo.indices.Car",
"make" : "toyota",
"model" : "corolla",
"bodystyles" : [
{
"type" : "2-door"
},
{
"type" : "5-door"
}
]
},
"sort" : [
"QjThSnMBLjyTRl7leMDk"
]
},
{
"_index" : "vehicle",
"_type" : "_doc",
"_id" : "QzThSnMBLjyTRl7lesDj",
"_score" : null,
"_source" : {
"_class" : "com.example.elastic.demo.indices.Car",
"make" : "kia",
"model" : "sorento",
"bodystyles" : [
{
"type" : "5-door"
}
]
},
"sort" : [
"QzThSnMBLjyTRl7lesDj"
]
},
{
"_index" : "vehicle",
"_type" : "_doc",
"_id" : "RDThSnMBLjyTRl7lfMDg",
"_score" : null,
"_source" : {
"_class" : "com.example.elastic.demo.indices.Car",
"make" : "kia",
"model" : "sportage",
"bodystyles" : [
{
"type" : "4-door"
}
]
},
"sort" : [
"RDThSnMBLjyTRl7lfMDg"
]
}
]
}
}
Kibana
Obtain Kibana HTTP IP as shown below and login using username "
elastic" and password we obtained previously.
$ kubectl get svc kibana-sample-kb-http -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
10.195.93.174
Finally maybe you want to deploy the application to Kubernetes. To do that take a look at
Cloud Native Buildpacks CNCF project and/or
Tanzu Build Service to turn your code into a Container Image stored in a registry.
More Information
Spring Data Elasticsearch
https://spring.io/projects/spring-data-elasticsearch
VMware Tanzu Kubernetes Grid Integrated Edition Documentation
https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid-Integrated-Edition/index.html
