In this post I show how to get Elastic Cloud on Kubernetes (ECK) up and running on VMware Tanzu Kubernetes Grid Integrated Edition and how to access it using a Spring Boot Application using Spring Data Elasticsearch.
With ECK, users now have a seamless way of deploying, managing, and operating the Elastic Stack on Kubernetes.
If you have a K8s cluster that's all you need to follow along.
Steps
1. Let's install ECK on our cluster we do that as follows
Note: There is a 1.1 version as the latest BUT I installing a slightly older one here
$ kubectl apply -f https://download.elastic.co/downloads/eck/1.0.1/all-in-one.yaml
2. Make sure the operator is up and running as shown below
$ kubectl get all -n elastic-system NAME READY STATUS RESTARTS AGE pod/elastic-operator-0 1/1 Running 0 26d NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/elastic-webhook-server ClusterIP 10.100.200.55 <none> 443/TCP 26d NAME READY AGE statefulset.apps/elastic-operator 1/1 26d
3. We can also see a CRD for Elasticsearch as shown below.
elasticsearches.elasticsearch.k8s.elastic.co
$ kubectl get crd NAME CREATED AT apmservers.apm.k8s.elastic.co 2020-06-17T00:37:32Z clusterlogsinks.pksapi.io 2020-06-16T23:04:43Z clustermetricsinks.pksapi.io 2020-06-16T23:04:44Z elasticsearches.elasticsearch.k8s.elastic.co 2020-06-17T00:37:33Z kibanas.kibana.k8s.elastic.co 2020-06-17T00:37:34Z loadbalancers.vmware.com 2020-06-16T22:51:52Z logsinks.pksapi.io 2020-06-16T23:04:43Z metricsinks.pksapi.io 2020-06-16T23:04:44Z nsxerrors.nsx.vmware.com 2020-06-16T22:51:52Z nsxlbmonitors.vmware.com 2020-06-16T22:51:52Z nsxlocks.nsx.vmware.com 2020-06-16T22:51:51Z
4. We are now ready to create our first Elasticsearch cluster. To do that create a file YML file as shown below
create-elastic-cluster-from-operator.yaml
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: quickstart
spec:
version: 7.7.0
http:
service:
spec:
type: LoadBalancer # default is ClusterIP
tls:
selfSignedCertificate:
disabled: true
nodeSets:
- name: default
count: 2
volumeClaimTemplates:
- metadata:
name: elasticsearch-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
config:
node.master: true
node.data: true
node.ingest: true
node.store.allow_mmap: false
From the YML a few things to note:
- We are creating two pods for our Elasticsearch cluster
- We are using a K8s LoadBalancer to expose access to the cluster through HTTP
- We are using version 7.7.0 but this is not the latest Elasticsearch version
- We have disabled the use of TLS given this is just a demo
$ kubectl apply -f create-elastic-cluster-from-operator.yaml
6. After about a minute we should have our Elasticsearch cluster running. The following commands show that
$ kubectl get elasticsearch NAME HEALTH NODES VERSION PHASE AGE quickstart green 2 7.7.0 Ready 47h $ kubectl get all -n default NAME READY STATUS RESTARTS AGE pod/quickstart-es-default-0 1/1 Running 0 47h pod/quickstart-es-default-1 1/1 Running 0 47h NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kubernetes ClusterIP 10.100.200.1 <none> 443/TCP 27d service/quickstart-es-default ClusterIP None <none> <none> 47h service/quickstart-es-http LoadBalancer 10.100.200.92 10.195.93.137 9200:30590/TCP 47h NAME READY AGE statefulset.apps/quickstart-es-default 2/2 47h
7. Let's deploy a Kibana instance. To do that create a YML as shown below
create-kibana.yaml
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
name: kibana-sample
spec:
version: 7.7.0
count: 1
elasticsearchRef:
name: quickstart
namespace: default
http:
service:
spec:
type: LoadBalancer # default is ClusterIP
8. Apply that as shown below.
$ kubectl apply -f create-kibana.yaml
9. To verify everything is up and running we can run a command as follows
$ kubectl get all NAME READY STATUS RESTARTS AGE pod/kibana-sample-kb-f8fcb88d5-jdzh5 1/1 Running 0 2d pod/quickstart-es-default-0 1/1 Running 0 2d pod/quickstart-es-default-1 1/1 Running 0 2d NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kibana-sample-kb-http LoadBalancer 10.100.200.46 10.195.93.174 5601:32459/TCP 2d service/kubernetes ClusterIP 10.100.200.1 <none> 443/TCP 27d service/quickstart-es-default ClusterIP None <none> <none> 2d service/quickstart-es-http LoadBalancer 10.100.200.92 10.195.93.137 9200:30590/TCP 2d NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/kibana-sample-kb 1/1 1 1 2d NAME DESIRED CURRENT READY AGE replicaset.apps/kibana-sample-kb-f8fcb88d5 1 1 1 2d NAME READY AGE statefulset.apps/quickstart-es-default 2/2 2d
10. So to access out cluster we will need to obtain the following which we can do using a script as follows. This was tested on Mac OSX
What do we need?
- Elasticsearch password
- IP address of the LoadBalancer service we created
access.sh
export PASSWORD=`kubectl get secret quickstart-es-elastic-user -o go-template='{{.data.elastic | base64decode}}'`
export IP=`kubectl get svc quickstart-es-http -o jsonpath='{.status.loadBalancer.ingress[0].ip}'`
echo ""
echo $IP
echo ""
curl -u "elastic:$PASSWORD" "http://$IP:9200"
echo ""
curl -u "elastic:$PASSWORD" "http://$IP:9200/_cat/health?v"
Output:
10.195.93.137
{
"name" : "quickstart-es-default-1",
"cluster_name" : "quickstart",
"cluster_uuid" : "Bbpb7Pu7SmaQaCmEY2Er8g",
"version" : {
"number" : "7.7.0",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "81a1e9eda8e6183f5237786246f6dced26a10eaf",
"build_date" : "2020-05-12T02:01:37.602180Z",
"build_snapshot" : false,
"lucene_version" : "8.5.1",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
.....
11. Ideally I would load some data into the Elasticsearch cluster BUT let's do that as part of a sample application using "Spring Data Elasticsearch". Clone the demo project as shown below.
$ git clone https://github.com/papicella/boot-elastic-demo.git
Cloning into 'boot-elastic-demo'...
remote: Enumerating objects: 36, done.
remote: Counting objects: 100% (36/36), done.
remote: Compressing objects: 100% (26/26), done.
remote: Total 36 (delta 1), reused 36 (delta 1), pack-reused 0
Unpacking objects: 100% (36/36), done.
12. Edit "./src/main/resources/application.yml" with your details for the Elasticsearch cluster above.
spring:
elasticsearch:
rest:
username: elastic
password: {PASSWORD}
uris: http://{IP}:9200
13. Package as follows
$ ./mvnw -DskipTests package
14. Run as follows
$ ./mvnw spring-boot:run
....
2020-07-14 11:10:11.947 INFO 76260 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path ''
2020-07-14 11:10:11.954 INFO 76260 --- [ main] c.e.e.demo.BootElasticDemoApplication : Started BootElasticDemoApplication in 2.495 seconds (JVM running for 2.778)
....
15. Access application using "http://localhost:8080/"
16. If we look at our code we will see the data was loaded into the Elasticsearch cluster using a java class called "LoadData.java". Ideally data should already exist in the cluster but for demo purposes we load some data as part of the Spring Boot Application and clear the data prior to each application run given it's just a demo.
2020-07-14 11:12:33.109 INFO 76277 --- [ main] com.example.elastic.demo.LoadData : Pre loading Car{id='OjThSnMBLjyTRl7lZsDL', make='holden', model='commodore', bodystyles=[BodyStyle{type='2-door'}, BodyStyle{type='4-door'}, BodyStyle{type='5-door'}]}
2020-07-14 11:12:33.584 INFO 76277 --- [ main] com.example.elastic.demo.LoadData : Pre loading Car{id='OzThSnMBLjyTRl7laMCo', make='holden', model='astra', bodystyles=[BodyStyle{type='2-door'}, BodyStyle{type='4-door'}]}
2020-07-14 11:12:34.189 INFO 76277 --- [ main] com.example.elastic.demo.LoadData : Pre loading Car{id='PDThSnMBLjyTRl7lasCC', make='nissan', model='skyline', bodystyles=[BodyStyle{type='4-door'}]}
2020-07-14 11:12:34.744 INFO 76277 --- [ main] com.example.elastic.demo.LoadData : Pre loading Car{id='PTThSnMBLjyTRl7lbMDe', make='nissan', model='pathfinder', bodystyles=[BodyStyle{type='5-door'}]}
2020-07-14 11:12:35.227 INFO 76277 --- [ main] com.example.elastic.demo.LoadData : Pre loading Car{id='PjThSnMBLjyTRl7lb8AL', make='ford', model='falcon', bodystyles=[BodyStyle{type='4-door'}, BodyStyle{type='5-door'}]}
2020-07-14 11:12:36.737 INFO 76277 --- [ main] com.example.elastic.demo.LoadData : Pre loading Car{id='QDThSnMBLjyTRl7lcMDu', make='ford', model='territory', bodystyles=[BodyStyle{type='5-door'}]}
2020-07-14 11:12:37.266 INFO 76277 --- [ main] com.example.elastic.demo.LoadData : Pre loading Car{id='QTThSnMBLjyTRl7ldsDU', make='toyota', model='camry', bodystyles=[BodyStyle{type='4-door'}, BodyStyle{type='5-door'}]}
2020-07-14 11:12:37.777 INFO 76277 --- [ main] com.example.elastic.demo.LoadData : Pre loading Car{id='QjThSnMBLjyTRl7leMDk', make='toyota', model='corolla', bodystyles=[BodyStyle{type='2-door'}, BodyStyle{type='5-door'}]}
2020-07-14 11:12:38.285 INFO 76277 --- [ main] com.example.elastic.demo.LoadData : Pre loading Car{id='QzThSnMBLjyTRl7lesDj', make='kia', model='sorento', bodystyles=[BodyStyle{type='5-door'}]}
2020-07-14 11:12:38.800 INFO 76277 --- [ main] com.example.elastic.demo.LoadData : Pre loading Car{id='RDThSnMBLjyTRl7lfMDg', make='kia', model='sportage', bodystyles=[BodyStyle{type='4-door'}]}
LoadData.java
package com.example.elastic.demo;
import com.example.elastic.demo.indices.BodyStyle;
import com.example.elastic.demo.indices.Car;
import com.example.elastic.demo.repo.CarRepository;
import org.springframework.boot.CommandLineRunner;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import lombok.extern.slf4j.Slf4j;
import static java.util.Arrays.asList;
@Configuration
@Slf4j
public class LoadData {
@Bean
public CommandLineRunner initElasticsearchData(CarRepository carRepository) {
return args -> {
carRepository.deleteAll();
log.info("Pre loading " + carRepository.save(new Car("holden", "commodore", asList(new BodyStyle("2-door"), new BodyStyle("4-door"), new BodyStyle("5-door")))));
log.info("Pre loading " + carRepository.save(new Car("holden", "astra", asList(new BodyStyle("2-door"), new BodyStyle("4-door")))));
log.info("Pre loading " + carRepository.save(new Car("nissan", "skyline", asList(new BodyStyle("4-door")))));
log.info("Pre loading " + carRepository.save(new Car("nissan", "pathfinder", asList(new BodyStyle("5-door")))));
log.info("Pre loading " + carRepository.save(new Car("ford", "falcon", asList(new BodyStyle("4-door"), new BodyStyle("5-door")))));
log.info("Pre loading " + carRepository.save(new Car("ford", "territory", asList(new BodyStyle("5-door")))));
log.info("Pre loading " + carRepository.save(new Car("toyota", "camry", asList(new BodyStyle("4-door"), new BodyStyle("5-door")))));
log.info("Pre loading " + carRepository.save(new Car("toyota", "corolla", asList(new BodyStyle("2-door"), new BodyStyle("5-door")))));
log.info("Pre loading " + carRepository.save(new Car("kia", "sorento", asList(new BodyStyle("5-door")))));
log.info("Pre loading " + carRepository.save(new Car("kia", "sportage", asList(new BodyStyle("4-door")))));
};
}
}
17. Our CarRepository interface is defined as follows
CarRepository.java
package com.example.elastic.demo.repo;
import com.example.elastic.demo.indices.Car;
import org.springframework.data.domain.Page;
import org.springframework.data.domain.Pageable;
import org.springframework.data.elasticsearch.repository.ElasticsearchRepository;
import org.springframework.stereotype.Repository;
@Repository
public interface CarRepository extends ElasticsearchRepository <Car, String> {
Page<Car> findByMakeContaining(String make, Pageable page);
}
18. So let's also via this data using "curl" and Kibana as shown below.
curl -X GET -u "elastic:{PASSWORD}" "http://{IP}:9200/vehicle/_search?pretty" -H 'Content-Type: application/json' -d'
{
"query": { "match_all": {} },
"sort": [
{ "_id": "asc" }
]
}
'
Output:
{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 10,
"relation" : "eq"
},
"max_score" : null,
"hits" : [
{
"_index" : "vehicle",
"_type" : "_doc",
"_id" : "OjThSnMBLjyTRl7lZsDL",
"_score" : null,
"_source" : {
"_class" : "com.example.elastic.demo.indices.Car",
"make" : "holden",
"model" : "commodore",
"bodystyles" : [
{
"type" : "2-door"
},
{
"type" : "4-door"
},
{
"type" : "5-door"
}
]
},
"sort" : [
"OjThSnMBLjyTRl7lZsDL"
]
},
{
"_index" : "vehicle",
"_type" : "_doc",
"_id" : "OzThSnMBLjyTRl7laMCo",
"_score" : null,
"_source" : {
"_class" : "com.example.elastic.demo.indices.Car",
"make" : "holden",
"model" : "astra",
"bodystyles" : [
{
"type" : "2-door"
},
{
"type" : "4-door"
}
]
},
"sort" : [
"OzThSnMBLjyTRl7laMCo"
]
},
{
"_index" : "vehicle",
"_type" : "_doc",
"_id" : "PDThSnMBLjyTRl7lasCC",
"_score" : null,
"_source" : {
"_class" : "com.example.elastic.demo.indices.Car",
"make" : "nissan",
"model" : "skyline",
"bodystyles" : [
{
"type" : "4-door"
}
]
},
"sort" : [
"PDThSnMBLjyTRl7lasCC"
]
},
{
"_index" : "vehicle",
"_type" : "_doc",
"_id" : "PTThSnMBLjyTRl7lbMDe",
"_score" : null,
"_source" : {
"_class" : "com.example.elastic.demo.indices.Car",
"make" : "nissan",
"model" : "pathfinder",
"bodystyles" : [
{
"type" : "5-door"
}
]
},
"sort" : [
"PTThSnMBLjyTRl7lbMDe"
]
},
{
"_index" : "vehicle",
"_type" : "_doc",
"_id" : "PjThSnMBLjyTRl7lb8AL",
"_score" : null,
"_source" : {
"_class" : "com.example.elastic.demo.indices.Car",
"make" : "ford",
"model" : "falcon",
"bodystyles" : [
{
"type" : "4-door"
},
{
"type" : "5-door"
}
]
},
"sort" : [
"PjThSnMBLjyTRl7lb8AL"
]
},
{
"_index" : "vehicle",
"_type" : "_doc",
"_id" : "QDThSnMBLjyTRl7lcMDu",
"_score" : null,
"_source" : {
"_class" : "com.example.elastic.demo.indices.Car",
"make" : "ford",
"model" : "territory",
"bodystyles" : [
{
"type" : "5-door"
}
]
},
"sort" : [
"QDThSnMBLjyTRl7lcMDu"
]
},
{
"_index" : "vehicle",
"_type" : "_doc",
"_id" : "QTThSnMBLjyTRl7ldsDU",
"_score" : null,
"_source" : {
"_class" : "com.example.elastic.demo.indices.Car",
"make" : "toyota",
"model" : "camry",
"bodystyles" : [
{
"type" : "4-door"
},
{
"type" : "5-door"
}
]
},
"sort" : [
"QTThSnMBLjyTRl7ldsDU"
]
},
{
"_index" : "vehicle",
"_type" : "_doc",
"_id" : "QjThSnMBLjyTRl7leMDk",
"_score" : null,
"_source" : {
"_class" : "com.example.elastic.demo.indices.Car",
"make" : "toyota",
"model" : "corolla",
"bodystyles" : [
{
"type" : "2-door"
},
{
"type" : "5-door"
}
]
},
"sort" : [
"QjThSnMBLjyTRl7leMDk"
]
},
{
"_index" : "vehicle",
"_type" : "_doc",
"_id" : "QzThSnMBLjyTRl7lesDj",
"_score" : null,
"_source" : {
"_class" : "com.example.elastic.demo.indices.Car",
"make" : "kia",
"model" : "sorento",
"bodystyles" : [
{
"type" : "5-door"
}
]
},
"sort" : [
"QzThSnMBLjyTRl7lesDj"
]
},
{
"_index" : "vehicle",
"_type" : "_doc",
"_id" : "RDThSnMBLjyTRl7lfMDg",
"_score" : null,
"_source" : {
"_class" : "com.example.elastic.demo.indices.Car",
"make" : "kia",
"model" : "sportage",
"bodystyles" : [
{
"type" : "4-door"
}
]
},
"sort" : [
"RDThSnMBLjyTRl7lfMDg"
]
}
]
}
}
Kibana
Obtain Kibana HTTP IP as shown below and login using username "elastic" and password we obtained previously.
$ kubectl get svc kibana-sample-kb-http -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
10.195.93.174
Finally maybe you want to deploy the application to Kubernetes. To do that take a look at Cloud Native Buildpacks CNCF project and/or Tanzu Build Service to turn your code into a Container Image stored in a registry.
More Information
Spring Data Elasticsearch
https://spring.io/projects/spring-data-elasticsearch
VMware Tanzu Kubernetes Grid Integrated Edition Documentation
https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid-Integrated-Edition/index.html
No comments:
Post a Comment