tag:blogger.com,1999:blog-65276887434562052562024-02-21T14:49:06.216+11:00The Blas from PasInformation on Snyk : Develop Fast. Stay Secure.Pas Apicellahttp://www.blogger.com/profile/09389663166398991762noreply@blogger.comBlogger466125tag:blogger.com,1999:blog-6527688743456205256.post-67551253041446505642022-10-18T16:01:00.001+11:002022-10-18T16:01:20.027+11:00Secure your application from Argo CD to Kubernetes<p>GitOps is a popular framework for managing and securing the application development pipeline. For many who have embarked on a GitOps journey, a common question is: “how can I secure my pipeline when everything is automated?” </p><p>The GitOps framework is a concept where any code commits or changes are done through Git, which then triggers an automated pipeline that builds and deploys applications on Kubernetes. Because there are few touch points for development and security teams in the pipeline, its security needs to be mandated to ensure the deployed applications have as few vulnerabilities as possible. </p><p>This blog covers how Snyk can provide application security in GitOps, focusing on a popular tool, Argo CD. In this scenario, Snyk runs an IaC scan to ensure the to-be-deployed application is safe before deployment, and stops the build if it is not. Snyk also can monitor the deployed applications across different namespaces in Kubernetes in an automated fashion.</p><p><a href="https://snyk.io/blog/secure-apps-from-argocd-to-kubernetes/">https://snyk.io/blog/secure-apps-from-argocd-to-kubernetes/</a></p><p><br /></p><div class="blogger-post-footer">http://feeds.feedburner.com/TheBlasFromPas</div>Pas Apicellahttp://www.blogger.com/profile/09389663166398991762noreply@blogger.com0tag:blogger.com,1999:blog-6527688743456205256.post-89255513218696504552022-06-02T21:30:00.001+10:002022-06-02T21:30:10.808+10:00Note for self: Snyk scan from ArgoCD <p><b> Demo</b></p><p><a href="https://github.com/papicella/springbootemployee-api/tree/master/argocd">https://github.com/papicella/springbootemployee-api/tree/master/argocd</a></p><p><b>Demo Job on K8s to perform Snyk IaC Scan</b></p><p></p><pre class="brush: yml">
apiVersion: batch/v1
kind: Job
metadata:
name: snyk-iac-scan
annotations:
argocd.argoproj.io/hook: PreSync
spec:
ttlSecondsAfterFinished: 600
template:
spec:
containers:
- name: snyk-cli
image: snyk/snyk-cli:npm
command: ["/bin/sh","-c"]
args:
- git clone https://github.com/papicella/springbootemployee-api.git;
snyk auth $SNYK_TOKEN;
snyk iac test springbootemployee-api/argocd/employee-K8s.yaml || true;
env:
- name: SNYK_TOKEN
valueFrom:
secretKeyRef:
name: snyk-token
key: token
restartPolicy: Never
backoffLimit: 0
</pre> <p></p><p><br /></p><div style="text-align: left;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjCG-Eazg249jOqQe1_9MP9YhAlssklcnZiqJuB6HDYcXOLItgxxlZEvilUj3qvogqBy40oFhJfgKtRTeD_IePM0cRLWhYso56N8BulrO_P4t-VhWo2lFNCoNc9clCawL0jAZ-COvzAp8t4KPplSGARADvvytA6NGJdCJdRL2_R9KC7_OvMmABpafbsBw/s3836/Screen%20Shot%202022-06-02%20at%208.18.38%20pm.png" style="margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" data-original-height="2122" data-original-width="3836" height="177" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjCG-Eazg249jOqQe1_9MP9YhAlssklcnZiqJuB6HDYcXOLItgxxlZEvilUj3qvogqBy40oFhJfgKtRTeD_IePM0cRLWhYso56N8BulrO_P4t-VhWo2lFNCoNc9clCawL0jAZ-COvzAp8t4KPplSGARADvvytA6NGJdCJdRL2_R9KC7_OvMmABpafbsBw/s320/Screen%20Shot%202022-06-02%20at%208.18.38%20pm.png" width="320" /></a><table class="highlight tab-size js-file-line-container js-code-nav-container js-tagsearch-file" data-paste-markdown-skip="" data-tab-size="8" data-tagsearch-lang="YAML" data-tagsearch-path="goof/snyk-scan.yaml"><tbody></tbody></table></div><div class="blogger-post-footer">http://feeds.feedburner.com/TheBlasFromPas</div>Pas Apicellahttp://www.blogger.com/profile/09389663166398991762noreply@blogger.com0tag:blogger.com,1999:blog-6527688743456205256.post-44179343808985715012022-05-30T19:42:00.002+10:002022-05-30T19:42:11.517+10:00Using Pulumi to automate the Snyk Kubernetes integration for containers<p>Better late than never finally got around to posting this today.</p><p>In this blog post, we will walk through the process of using <a href="https://www.pulumi.com/" rel="noopener">Pulumi</a>,
a new open source tool that allows developers to build code in multiple
languages like JavaScript, Typescript, Python, and Go to create all
that is required to configure the <a href="https://docs.snyk.io/products/snyk-container/image-scanning-library/kubernetes-workload-and-image-scanning/kubernetes-integration-overview">Kubernetes integration in Snyk Container</a>.</p><div style="text-align: left;">Using Pulumi to automate the Snyk Kubernetes integration for containers<br /><a href="https://snyk.io/blog/automate-snyk-kubernetes-pulumi/">https://snyk.io/blog/automate-snyk-kubernetes-pulumi/</a></div><p><br /></p><div class="blogger-post-footer">http://feeds.feedburner.com/TheBlasFromPas</div>Pas Apicellahttp://www.blogger.com/profile/09389663166398991762noreply@blogger.com0tag:blogger.com,1999:blog-6527688743456205256.post-29290888902076532882021-12-23T16:41:00.000+11:002021-12-23T16:41:02.205+11:00Find and fix vulnerabilities in your CI/CD pipeline with Snyk and Harness<p>Integrating the Snyk developer-focused security platform into Harness’
unified delivery pipeline workflow ensures security and compliance
testing is part of every release. This allows you to prevent
applications with vulnerable dependencies and code from making their way
into production. With modern tooling like Snyk and Harness, you can
find, fix, and remediate through a <a href="https://snyk.io/learn/what-is-ci-cd-pipeline-and-tools-explained/">CI/CD pipeline</a> and mitigate the risk to the business without affecting your ability to release software quickly.</p><p>Created a new Snyk Blog with the harness team as per the link below.</p><p><a href="https://snyk.io/blog/find-fix-vulnerabilities-ci-cd-pipeline-snyk-harness/">https://snyk.io/blog/find-fix-vulnerabilities-ci-cd-pipeline-snyk-harness/</a></p><div class="blogger-post-footer">http://feeds.feedburner.com/TheBlasFromPas</div>Pas Apicellahttp://www.blogger.com/profile/09389663166398991762noreply@blogger.com0tag:blogger.com,1999:blog-6527688743456205256.post-33437221036650186432021-11-01T10:13:00.000+11:002021-11-01T10:13:27.451+11:00Using harness CI to run a series of Snyk Tests<p> <span style="font-family: Arial; font-size: 11pt; white-space: pre-wrap;">When DevOps emerged more than 10 years ago, the main focus was to bridge the gaps between Dev and Ops teams by introducing automation to the processes of building, testing and deployment of applications.</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">As development teams continue to deliver more rapidly and more frequently, security teams are finding it difficult to keep up and often end up being the bottleneck in the delivery pipeline. For this reason, bringing security early into the DevOps process from the outset – in other words, embracing a DevSecOps culture within a business – has become increasingly important.</span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Applications have changed as cloud-native technologies like Containers, Kubernetes, and the use of Infrastructure as Code technologies like Terraform, Cloudformation, ARM templates are now the norm. These elements are now built and customized by developers and live in their Source Code Management repositories.</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">In this demo below I show you how Harness CI along with Snyk can help setup a DevSecOps Pipeline well before we even think about deployment.</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></p><h2 style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: left;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Steps</span></h2><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="color: #ffa400; font-family: Arial;"><span style="font-size: 14.6667px; white-space: pre-wrap;"><i>Note: We are using the public GitHub repo below</i></span></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial;"><span style="font-size: 14.6667px; white-space: pre-wrap;"><br /></span></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial;"><span style="font-size: 14.6667px; white-space: pre-wrap;"><a href="https://github.com/papicella/springbootemployee-api">https://github.com/papicella/springbootemployee-api</a></span></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial;"><span style="font-size: 14.6667px; white-space: pre-wrap;"><br /></span></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial;"><span style="font-size: 14.6667px; white-space: pre-wrap;">1. First we need to identify our user token within Snyk. For enterprise customers you can setup service accounts and retrieve a token which we will require later. For non enterprise Snyk accounts you can just use the main User Token. Both methods of how to obtain this are defined below.</span></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;"><br /></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;">Service Accounts</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial;"><span style="font-size: 14.6667px; white-space: pre-wrap;"><a href="https://docs.snyk.io/features/integrations/managing-integrations/service-accounts">https://docs.snyk.io/features/integrations/managing-integrations/service-accounts</a></span></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial;"><span style="font-size: 14.6667px; white-space: pre-wrap;"><br /></span></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial;"><span style="font-size: 14.6667px; white-space: pre-wrap;">For those without access to service accounts you can obtain your Snyk user Token as follows</span></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial;"><span style="font-size: 14.6667px; white-space: pre-wrap;"><a href="https://docs.snyk.io/features/snyk-cli/install-the-snyk-cli/authenticate-the-cli-with-your-account">https://docs.snyk.io/features/snyk-cli/install-the-snyk-cli/authenticate-the-cli-with-your-account</a></span></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><br /></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;">2. In harness let's define a few secrets one being our Snyk Token we retrieved in step 1, also my GitHub token </span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;"><br /></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;"></span></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhJpZMH8JcWp2VPCIZQncI0oAboKsAD9L4iSaTynSIMgRA_v9NEeSrOFqG43DahpOlZPPeGL-9-Cc2kdsHNtcXkaagy6JoGMSDQUEhM8id18uCy9287FbkkoptBT4ZHmUHOJEKlahaLn9Dj/" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="1020" data-original-width="3084" height="106" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhJpZMH8JcWp2VPCIZQncI0oAboKsAD9L4iSaTynSIMgRA_v9NEeSrOFqG43DahpOlZPPeGL-9-Cc2kdsHNtcXkaagy6JoGMSDQUEhM8id18uCy9287FbkkoptBT4ZHmUHOJEKlahaLn9Dj/" width="320" /></a></div><br /><p></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;">3. Our builds need to run somewhere in this example we are using a connector to our K8s cluster which is defined at the "<b>Organization</b>" level within Harness</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;"><br /></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;"></span></p><div class="separator" style="clear: both; text-align: center;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjSbp9KGCJCldjGHCXjzT3Uejk8OT9Q9baU79HEYQJfL-siGqlEiLWkfDNvazvT9Sv_Y75u7hUAJ0PIMvnr6LGHNMdu8rBC4-weLu0keOtQTAJ3nDM7UbDPPsBPgVly0W7q-jT8GBKKnUH3/" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="1343" data-original-width="2048" height="210" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjSbp9KGCJCldjGHCXjzT3Uejk8OT9Q9baU79HEYQJfL-siGqlEiLWkfDNvazvT9Sv_Y75u7hUAJ0PIMvnr6LGHNMdu8rBC4-weLu0keOtQTAJ3nDM7UbDPPsBPgVly0W7q-jT8GBKKnUH3/" width="320" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhS7xUNEwQdx6KYWVdCJ0IDD6DE-xK3IlV42MyJTLiM4XjLp8R5RvuzDxnfpHxXCIrK2xGkHv42W3X2xebhNU8mxiLTRvtxHeMBXmNoTEUjveg8-mzdmzBXXXu7OoxDzfRJ3v-gHoEQ4kHs/" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="1090" data-original-width="1300" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhS7xUNEwQdx6KYWVdCJ0IDD6DE-xK3IlV42MyJTLiM4XjLp8R5RvuzDxnfpHxXCIrK2xGkHv42W3X2xebhNU8mxiLTRvtxHeMBXmNoTEUjveg8-mzdmzBXXXu7OoxDzfRJ3v-gHoEQ4kHs/" width="286" /></a></div><br /><br /></div><p></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial;"><span style="font-size: 14.6667px; white-space: pre-wrap;">4. Our first execution step is simply going to package the application. Our Spring Boot application is using maven and has a single pom.xml file. Notice below that we have to use a Docker connector for a "<b>Build-> Run</b>" step , here I am using an account level Docker connector</span></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial;"><span style="font-size: 14.6667px; white-space: pre-wrap;"><br /></span></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial;"><span style="font-size: 14.6667px; white-space: pre-wrap;"></span></span></p><div class="separator" style="clear: both; text-align: center;"><span style="font-family: Arial;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjcU2lsj27vpD__jQYdEzwpgCqEvWakpQaqKFc9Sx7o_okzsEvltF29XbaOMzpdBosLu-q1Ijvmps_D1zSgCRzYanp6JuT6XczbWmCK2vgiLEuaKkRL7RNOZM1qeq1vRJf81p0Q9nBde39w/" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="1758" data-original-width="1196" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjcU2lsj27vpD__jQYdEzwpgCqEvWakpQaqKFc9Sx7o_okzsEvltF29XbaOMzpdBosLu-q1Ijvmps_D1zSgCRzYanp6JuT6XczbWmCK2vgiLEuaKkRL7RNOZM1qeq1vRJf81p0Q9nBde39w/" width="163" /></a></span></div><span style="font-family: Arial;"><br /><br /></span><p></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial;"><span style="font-size: 14.6667px; white-space: pre-wrap;">5. Now we can run our first Snyk Test. In this simple example we don't fail the build at all and ensure we pass a SUCCESS exit code for our Snyk Open Source test which will pick up the one and only pom.xml file in the repo. I am also using Snyk Docker image that includes the Snyk CLI and Maven to perform this test.</span></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial;"><span style="font-size: 14.6667px; white-space: pre-wrap;"><br /></span></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial;"><span style="font-size: 14.6667px; white-space: pre-wrap;"></span></span></p><div class="separator" style="clear: both; text-align: center;"><span style="font-family: Arial;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiYPaPoih8kla5DqifVd4ICEbeRCY0ugS1LLnVEGsn3W6uyhmuCEjjzyy9QfXHyb2lnegreHOvJvjecmnP8-XHhKO1e-X1wS-eZ5Lv771k3oDZ0L5PBdvVe4-cFbHaRKu116K77QKOI9TCv/" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="1744" data-original-width="1240" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiYPaPoih8kla5DqifVd4ICEbeRCY0ugS1LLnVEGsn3W6uyhmuCEjjzyy9QfXHyb2lnegreHOvJvjecmnP8-XHhKO1e-X1wS-eZ5Lv771k3oDZ0L5PBdvVe4-cFbHaRKu116K77QKOI9TCv/" width="171" /></a></span></div><span style="font-family: Arial;"><br /><br /></span><p></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;"><br /></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;">6. So jumping right ahead let's quickly take a look at our other 3 tests.</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;"><br /></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;"><b>Snyk Code Test</b></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;"></span></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhq1Ryt2dg2K-EoaS_EKp1M_giwxvxGz3Veo1n2CQvYkXkzERnOipQD4XKBAazRWQ0J7U3bhK3WFmiPCT4vcTZjWMOrpMYU0NUIhQji4_z3CV2hheVQa15yCyygRb0Uni9dTXdbesXc_wne/" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="1802" data-original-width="1258" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhq1Ryt2dg2K-EoaS_EKp1M_giwxvxGz3Veo1n2CQvYkXkzERnOipQD4XKBAazRWQ0J7U3bhK3WFmiPCT4vcTZjWMOrpMYU0NUIhQji4_z3CV2hheVQa15yCyygRb0Uni9dTXdbesXc_wne/" width="168" /></a></div><br /><br /><p></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;"><b>Snyk IaC test</b></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;"><b><br /></b></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;"><b></b></span></p><div class="separator" style="clear: both; text-align: center;"><b><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhtnxHQITMZKxhh_pPxcA7O1kCg3kwtPTDcFzpHEgg0e7y8hHX4ORG5zb-8blGpKBt8Lqj_lAgpuepO_Y7Ds42_hrNGSlmCBzkGQ5p9GDekvikzTXzLgZJYr9xwPbcU1CSfsSMvIIXo6lUy/" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="1788" data-original-width="1268" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhtnxHQITMZKxhh_pPxcA7O1kCg3kwtPTDcFzpHEgg0e7y8hHX4ORG5zb-8blGpKBt8Lqj_lAgpuepO_Y7Ds42_hrNGSlmCBzkGQ5p9GDekvikzTXzLgZJYr9xwPbcU1CSfsSMvIIXo6lUy/" width="170" /></a></b></div><b><br /><br /></b><p></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;"><b>Snyk Container Test</b></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;"><b><br /></b></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;"><b></b></span></p><div class="separator" style="clear: both; text-align: center;"><b><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjZDE9nAsmUTqceBFR8rhi7bGGne058W4UeJICeUyfermLrQQzXOCzZDM0xOx_QnqxuhM5WyIwiLaj1aKC2-oGtsUZpstcZ4_xZQSvU5NNCx-mkxSnKgKscrrlzzLdDz30MRBZuJhOTtEYv/" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="1750" data-original-width="1262" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjZDE9nAsmUTqceBFR8rhi7bGGne058W4UeJICeUyfermLrQQzXOCzZDM0xOx_QnqxuhM5WyIwiLaj1aKC2-oGtsUZpstcZ4_xZQSvU5NNCx-mkxSnKgKscrrlzzLdDz30MRBZuJhOTtEYv/" width="173" /></a></b></div><b><br /><br /></b><p></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;">In all these Snyk Tests we ensure nothing fails instead we just report on </span><span style="font-family: Arial;"><span style="font-size: 14.6667px; white-space: pre-wrap;">vulnerabilities. WE could also set our </span><span style="font-size: 14.6667px; white-space: pre-wrap;">severity</span><span style="font-size: 14.6667px; white-space: pre-wrap;"> threshold as part of our tests as shown below.</span></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial;"><span style="font-size: 14.6667px; white-space: pre-wrap;"><br /></span></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial;"><span style="font-size: 14.6667px; white-space: pre-wrap;">--severity-threshold=medium</span></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial;"><span style="font-size: 14.6667px; white-space: pre-wrap;"><br /></span></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial;"><span style="font-size: 14.6667px; white-space: pre-wrap;"><a href="https://docs.snyk.io/features/snyk-cli/test-for-vulnerabilities/set-severity-thresholds-for-cli-tests">https://docs.snyk.io/features/snyk-cli/test-for-vulnerabilities/set-severity-thresholds-for-cli-tests</a></span></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial;"><span style="font-size: 14.6667px; white-space: pre-wrap;"><b><br /></b></span></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;">7. In order to run the container test we had to build our container image which was done using a "<b>Build and push image to Dockerhub</b>" step as shown below</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;"><br /></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;"></span></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhxJcphPPEpCRYKJ4PiN-yrz-U7io789XAfsO1_pOODfBZKYaIsk6Liq07tblOVxu5ce9wW8VTUuHawPf5UDA7edYvbM_WtwjvIDn8kcyvpqDNYxgV4q50EFLRoji6Gkwg0A1ZDoGMpZlnE/" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="2048" data-original-width="1130" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhxJcphPPEpCRYKJ4PiN-yrz-U7io789XAfsO1_pOODfBZKYaIsk6Liq07tblOVxu5ce9wW8VTUuHawPf5UDA7edYvbM_WtwjvIDn8kcyvpqDNYxgV4q50EFLRoji6Gkwg0A1ZDoGMpZlnE/" width="132" /></a></div><br /><p></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial;"><span style="font-size: 14.6667px; white-space: pre-wrap;">8. Now after running a few builds our Overview page on Harness UI nicely summarises what has passed versus what has failed for us.</span></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial;"><span style="font-size: 14.6667px; white-space: pre-wrap;"><br /></span></span><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;"></span></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgTBmbT3mcoG718PKSQorpxrmc8hu-0jtGbb8rm60pMguKSoydPABreMed1-P_jJ_0qEBcQtxO7N6SrA1z55JWp2mwtGaFZeh6aOveT1CqFhpTojKNGHvJkA1VYEy5WrV5cHQd6wPSfiKVC/" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="1239" data-original-width="2539" height="156" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgTBmbT3mcoG718PKSQorpxrmc8hu-0jtGbb8rm60pMguKSoydPABreMed1-P_jJ_0qEBcQtxO7N6SrA1z55JWp2mwtGaFZeh6aOveT1CqFhpTojKNGHvJkA1VYEy5WrV5cHQd6wPSfiKVC/" width="320" /></a></div><br /> <p></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;">9. Finally we can view each build as follows</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;"><br /></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;"></span></p><div class="separator" style="clear: both; text-align: center;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj35DCu02XBMpF8IF65ew09TWP0QG80w33zGLbvQqwxozGYoBcEFfyY9L88wyjXMDTVe5j8ERtSehCKax6KZKxUZdk9Du_FJyIvbteyee8QIjweiRN74dQI9MKAOLR3Je1D-FT4Qu07jqoh/" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="1299" data-original-width="2048" height="203" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj35DCu02XBMpF8IF65ew09TWP0QG80w33zGLbvQqwxozGYoBcEFfyY9L88wyjXMDTVe5j8ERtSehCKax6KZKxUZdk9Du_FJyIvbteyee8QIjweiRN74dQI9MKAOLR3Je1D-FT4Qu07jqoh/" width="320" /></a></div><br /><br /></div><br /><p></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial;"><span style="font-size: 14.6667px; white-space: pre-wrap;"><br /></span></span></p><h2 style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: left;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Sample Images</span></h2><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Overview Page</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"></span></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgdHaq94YJAgvbAYTeXkjDifXZ2eIdlNnmfkKANQJITM8zYbOWT0sN7-9_7A3R_Md8lFtHhbxvNRXSlKAFBWgY8Otb7fB_fsxrnmYAZ5hgSAWToyefK2tCCqa4l5Mu6eN3cRlRa_OmkE7h7/" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="1301" data-original-width="2048" height="203" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgdHaq94YJAgvbAYTeXkjDifXZ2eIdlNnmfkKANQJITM8zYbOWT0sN7-9_7A3R_Md8lFtHhbxvNRXSlKAFBWgY8Otb7fB_fsxrnmYAZ5hgSAWToyefK2tCCqa4l5Mu6eN3cRlRa_OmkE7h7/" width="320" /></a></div><br />Infrastructure Page - Here we define where are harness delegate running in a K8s cluster which is just one of the options here when it comes to a worker node for the pipeline steps we run<p></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"></span></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgJ3uL-_i6_OdevcIV_eT_L6IlBp58RmmMQxml5gsD3naVQz3vdggwS7REnR1KhrNH6RZxg_sPIzjfQ3LVHcNVe2xKs9aDmxYWep8s9oImcVyicA6tX0q2BAJvCbWoATGN3xaGf58ZZLqLJ/" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="1201" data-original-width="2619" height="147" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgJ3uL-_i6_OdevcIV_eT_L6IlBp58RmmMQxml5gsD3naVQz3vdggwS7REnR1KhrNH6RZxg_sPIzjfQ3LVHcNVe2xKs9aDmxYWep8s9oImcVyicA6tX0q2BAJvCbWoATGN3xaGf58ZZLqLJ/" width="320" /></a></div><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></p>Execution Page - here we define four seperate tests. <p></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"></p><ul style="text-align: left;"><li><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Snyk Open Source Test</span></li><li><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Snyk Code Test</span></li><li><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Snyk IaC Test</span></li><li><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Snyk Container Test</span></li></ul><p></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"></span></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhNwaFcKKp3heUNX4z4obORnPT1XkIKh5wwKnIo-GEMadCmiChTQug5C9KUHFbbRiZ2NPIlbpTIrdbwCj2t-YNKKt5_4kJ3Efxr0rslwRKXHv5OnbkzdoWbV97HRhk45NuARzyC_3pqqg3a/" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="1172" data-original-width="2683" height="140" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhNwaFcKKp3heUNX4z4obORnPT1XkIKh5wwKnIo-GEMadCmiChTQug5C9KUHFbbRiZ2NPIlbpTIrdbwCj2t-YNKKt5_4kJ3Efxr0rslwRKXHv5OnbkzdoWbV97HRhk45NuARzyC_3pqqg3a/" width="320" /></a></div><br />Pipeline Result Page<p></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"></span></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEik9PC4XkC55Ij2pfHh5pwwICNHk4uc2b0Rq3dkkioF6n8cHXWc9WZnWhO1smcSLNUd7Es_AWPJMFL-ONpWzJ90eb4pYioEvgZ6dANZA9abKk-Dwv9fRj-K8WCXw4QE7_vfIhCwHyw2Wsg2/" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="1226" data-original-width="2565" height="153" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEik9PC4XkC55Ij2pfHh5pwwICNHk4uc2b0Rq3dkkioF6n8cHXWc9WZnWhO1smcSLNUd7Es_AWPJMFL-ONpWzJ90eb4pYioEvgZ6dANZA9abKk-Dwv9fRj-K8WCXw4QE7_vfIhCwHyw2Wsg2/" width="320" /></a></div><br /><br /><p></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">For those that wish to see the whole pipeline YAML here it is below:</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 14.6667px; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Arial;"><span style="color: #3d85c6;">pipeline:
name: employee-api-pipeline
identifier: employeeapipipeline
projectIdentifier: Springboot_Employee_API
orgIdentifier: default
tags: {}
properties:
ci:
codebase:
connectorRef: pasgithub
repoName: springbootemployee-api
build: <+input>
stages:
- stage:
name: build employee api
identifier: build_employee_api
description: ""
type: CI
spec:
cloneCodebase: true
infrastructure:
type: KubernetesDirect
spec:
connectorRef: org.GKE
namespace: harness-builds
execution:
steps:
- step:
type: Run
name: Package Application
identifier: Package_Application
spec:
connectorRef: account.harnessImage
image: maven:3.8.3-openjdk-11-slim
command: mvn -DskipTests -Dsnyk.skip package
privileged: false
- step:
type: Run
name: snyk test
identifier: snyk_test
spec:
connectorRef: account.harnessImage
image: snyk/snyk-cli:1.745.0-maven-3.5.4
command: |-
SNYK_TOKEN=<+secrets.getValue("SNYK_TOKEN")>
snyk config set api=$SNYK_TOKEN
snyk test || true
privileged: false
failureStrategies: []
- step:
type: Run
name: snyk code test
identifier: snyk_code_test
spec:
connectorRef: account.harnessImage
image: snyk/snyk-cli:1.745.0-maven-3.5.4
command: |-
SNYK_TOKEN=<+secrets.getValue("SNYK_TOKEN")>
snyk config set api=$SNYK_TOKEN
snyk code test || true
privileged: false
when:
stageStatus: Success
failureStrategies: []
- step:
type: Run
name: snyk IaC test
identifier: snyk_IaC_test
spec:
connectorRef: account.harnessImage
image: snyk/snyk-cli:1.745.0-maven-3.5.4
command: |-
SNYK_TOKEN=<+secrets.getValue("SNYK_TOKEN")>
snyk config set api=$SNYK_TOKEN
snyk iac test ./employee-K8s.yaml || true
privileged: false
when:
stageStatus: Success
failureStrategies: []
- step:
type: BuildAndPushDockerRegistry
name: Build Container
identifier: Build_Container
spec:
connectorRef: pasdockerhub
repo: pasapples/springbootemployee
tags:
- harness
dockerfile: Dockerfile.harness
optimize: true
- step:
type: Run
name: snyk container test
identifier: snyk_container_test
spec:
connectorRef: pasdockerhub
image: snyk/snyk-cli:1.745.0-maven-3.5.4
command: |-
SNYK_TOKEN=<+secrets.getValue("SNYK_TOKEN")>
snyk config set api=$SNYK_TOKEN
snyk container test pasapples/springbootemployee:harness || true
privileged: false
resources:
limits:
memory: 2048Mi
variables: []</span>
</span></span></p><div><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></div><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></p><h2 style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: left;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">More Information</span></h2><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Snyk</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><a href="https://snyk.io">https://snyk.io</a></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><br /></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Harness</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 14.6667px; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><span style="font-family: Arial;"><a href="https://harness.io/">https://harness.io/</a></span></span></p><div class="blogger-post-footer">http://feeds.feedburner.com/TheBlasFromPas</div>Pas Apicellahttp://www.blogger.com/profile/09389663166398991762noreply@blogger.com0tag:blogger.com,1999:blog-6527688743456205256.post-80021061158751810442021-09-15T14:15:00.003+10:002021-09-15T15:19:51.129+10:00Using Buildkite to perform Snyk Open Source and Snyk Code (SAST) tests<p><a href="https://buildkite.com/" target="_blank">Buildkite</a> is a platform for running fast, secure, and scalable continuous integration pipelines on your own infrastructure. In the example below I will run my Buildkite pipeline on my Macbook to perform two Snyk Tests, one for Open-Source dependancies and the other a SAST test of the code itself.</p><p><a href="http://snyk.io" target="_blank">Snyk</a> is an open source security platform designed to help software-driven businesses enhance developer security.</p><p>You will need an account on <a href="https://app.snyk.io" target="_blank">Snyk</a> and <a href="https://buildkite.com/" target="_blank">Buildkite</a> to follow the steps below.</p><h2 style="text-align: left;">Steps</h2><p>1. First in Snyk let's create a Service Account which will be the Snyk token I will use to authenticate with. You can use the Snyk API Token but the service account is all you need to run "<b>Snyk Tests</b>" so makes sense to use that.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEha_sqU8Ykug6uT24FPrGrw6lIa52jfQU15kOXBz4TJBggmPKTnFa9Xuhoa76tMsgWlI721TbS3fQckTUwXO4TsS06FFqsiDD-w37a0XrPefkbbp326phjA-kjsG7MeqRP1-HQ91yKsPam6/s2048/Screen+Shot+2021-09-15+at+11.55.41+am.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1325" data-original-width="2048" height="207" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEha_sqU8Ykug6uT24FPrGrw6lIa52jfQU15kOXBz4TJBggmPKTnFa9Xuhoa76tMsgWlI721TbS3fQckTUwXO4TsS06FFqsiDD-w37a0XrPefkbbp326phjA-kjsG7MeqRP1-HQ91yKsPam6/s320/Screen+Shot+2021-09-15+at+11.55.41+am.png" width="320" /></a></div><br /><p>2. Next let's store that Service Account token somewhere where I can safely inject that into my pipeline at the appropriate step. In this example I am using "<b>Google Secret Manager</b>" but there are other choices of course.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgVVvr21hiaLSdbTWfZQBiH62IN8wdBIYKzVzM66nr-7lVH9rP_0i45radGWsOOs3roj7XcRtV3gTLqneKv7iKFcs2y4Hn8jFNgUZWLYo7ggfCUzWjrPq3F-h8FkeeZM2DEGAWOeI34djzj/s2822/Screen+Shot+2021-09-15+at+11.59.48+am.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1070" data-original-width="2822" height="121" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgVVvr21hiaLSdbTWfZQBiH62IN8wdBIYKzVzM66nr-7lVH9rP_0i45radGWsOOs3roj7XcRtV3gTLqneKv7iKFcs2y4Hn8jFNgUZWLYo7ggfCUzWjrPq3F-h8FkeeZM2DEGAWOeI34djzj/s320/Screen+Shot+2021-09-15+at+11.59.48+am.png" width="320" /></a></div><p><br /></p><p><span style="color: #ffa400;">Note: We will be using the secret NAME shortly "<b>PAS_BUILDKITE_SA_SNYK_TOKEN</b>"</span></p><p>3. You will need a Buildkite agent on your local Infrastructure in my case I using my Macbook so that's done as follows</p><p><a href="https://buildkite.com/docs/agent/v3/macos">https://buildkite.com/docs/agent/v3/macos</a> </p><div style="text-align: left;"><div><span style="color: #3d85c6;">pasapicella@192-168-1-113:~/demos/integrations/buildkite$ ./start-agent.sh</span></div><div><span style="color: #3d85c6;"><br /></span></div><div><span style="color: #3d85c6;"> _ _ _ _ _ _ _ _</span></div><div><span style="color: #3d85c6;"> | | (_) | | | | (_) | | |</span></div><div><span style="color: #3d85c6;"> | |__ _ _ _| | __| | | ___| |_ ___ __ _ __ _ ___ _ __ | |_</span></div><div><span style="color: #3d85c6;"> | '_ \| | | | | |/ _` | |/ / | __/ _ \ / _` |/ _` |/ _ \ '_ \| __|</span></div><div><span style="color: #3d85c6;"> | |_) | |_| | | | (_| | <| | || __/ | (_| | (_| | __/ | | | |_</span></div><div><span style="color: #3d85c6;"> |_.__/ \__,_|_|_|\__,_|_|\_\_|\__\___| \__,_|\__, |\___|_| |_|\__|</span></div><div><span style="color: #3d85c6;"> __/ |</span></div><div><span style="color: #3d85c6;"> https://buildkite.com/agent |___/</span></div><div><span style="color: #3d85c6;"><br /></span></div><div><span style="color: #3d85c6;">2021-09-15 11:09:33 NOTICE Starting buildkite-agent v3.32.3 with PID: 50130</span></div><div><span style="color: #3d85c6;">2021-09-15 11:09:33 NOTICE The agent source code can be found here: https://github.com/buildkite/agent</span></div><div><span style="color: #3d85c6;">2021-09-15 11:09:33 NOTICE For questions and support, email us at: hello@buildkite.com</span></div><div><span style="color: #3d85c6;">2021-09-15 11:09:33 INFO Configuration loaded path=/usr/local/etc/buildkite-agent/buildkite-agent.cfg</span></div><div><span style="color: #3d85c6;">2021-09-15 11:09:33 INFO Registering agent with Buildkite...</span></div><div><span style="color: #3d85c6;">2021-09-15 11:09:35 INFO Successfully registered agent "y.y.y.y.tpgi.com.au-1" with tags []</span></div><div><span style="color: #3d85c6;">2021-09-15 11:09:35 INFO Starting 1 Agent(s)</span></div><div><span style="color: #3d85c6;">2021-09-15 11:09:35 INFO You can press Ctrl-C to stop the agents</span></div></div><p>4. You're now ready to create a pipeline. A pipeline is a template of the steps you want to run. There are many types of steps, some run scripts, some define conditional logic, and others wait for user input. When you run a pipeline, a build is created. Each of the steps in the pipeline end up as jobs in the build, which then get distributed to available agents.</p><p>In the example below our pipeline is created from a GitHub repo and then select the default branch. At that point incoming webhooks are sent to Buildkite by source control providers (GitHub, GitLab, Bitbucket, etc.) to trigger builds, in this scenario we using GitHub</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEijNUTlNaCcaHOngO91Y4zGdehcQV4Me04_FkQjkfjZ0B-beAediNCU_6UjaKoh00uhsS-q9hXV-cEVgGgeMxtcX1xdZCupmHodgJmYhTtsjk6h4LFf94HmafNFI6Z3pj7avyUX6SEv9gtH/s2048/Screen+Shot+2021-09-15+at+1.36.13+pm.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1809" data-original-width="2048" height="283" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEijNUTlNaCcaHOngO91Y4zGdehcQV4Me04_FkQjkfjZ0B-beAediNCU_6UjaKoh00uhsS-q9hXV-cEVgGgeMxtcX1xdZCupmHodgJmYhTtsjk6h4LFf94HmafNFI6Z3pj7avyUX6SEv9gtH/s320/Screen+Shot+2021-09-15+at+1.36.13+pm.png" width="320" /></a></div><br /><p>5. Let's go ahead and actually just edit the build steps using YAML. My final YAML is as follows and I explain below why it looks this way but in short I just want to run two snyk tests rather then actually deploy anything for this demo.</p><div style="text-align: left;"><div><span style="color: #3d85c6;">steps:</span></div><div><span style="color: #3d85c6;"> - commands:</span></div><div><span style="color: #3d85c6;"> - "snyk config set api=$$SNYK_SA_TOKEN_VAR"</span></div><div><span style="color: #3d85c6;"> - "snyk test --severity-threshold=$$SEVERITY_THRESHOLD"</span></div><div><span style="color: #3d85c6;"> - "snyk code test --org=$$SNYK_ORG"</span></div><div><span style="color: #3d85c6;"> plugins:</span></div><div><span style="color: #3d85c6;"> - avaly/gcp-secret-manager#v1.0.0:</span></div><div><span style="color: #3d85c6;"> credentials_file: /Users/pasapicella/snyk/clouds/gcp/buildkite-secrets-gcp.json</span></div><div><span style="color: #3d85c6;"> env:</span></div><div><span style="color: #3d85c6;"> SNYK_SA_TOKEN_VAR: PAS_BUILDKITE_SA_SNYK_TOKEN</span></div><div><span style="color: #3d85c6;"> env:</span></div><div><span style="color: #3d85c6;"> SEVERITY_THRESHOLD: "critical"</span></div><div><span style="color: #3d85c6;"> SNYK_ORG: "pas.apicella-41p"</span></div><div><span style="color: #3d85c6;"> label: "Employee API Snyk Test"</span></div></div><p>Few things to note here:</p><div style="text-align: left;"><ul style="text-align: left;"><li>I am using a GCP secret manager plugin to retrieve my Snyk SA token with a name as follows "<b>PAS_BUILDKITE_SA_SNYK_TOKEN</b>"</li><li>I am using a Google Service Account JSON so I can authenticate with GCP and retrieve my secret "<b>SNYK_SA_TOKEN_VAR</b>", you will need to use a Service Account with privileges to at least READ from Google Secret Manager</li><li>I am using some local non sensitive ENV variables which get used at the appropriate time</li><li>I have three commands of which the first command sets my Snyk API token for the Snyk CLI</li><li>I have not installed the Snyk CLI because it already exists on my Macbook </li><li>I am only looking for my Snyk tests to fail if it finds any <b>CRITICAL</b> issues only</li><li>I should be running a "<b>mvn package</b>" here but I can still execute a "<b>snyk test</b>" without it for demo purposes as we have a <b>pom..xml</b></li><li>I could also build a container in the pipeline from the source code and then<b> </b>run a "<b>snyk container test</b>" as well, in fact I could even run "<b>snyk iac test</b>" against any IaC files in the repo as well</li><li>If a test fails we can easily run "<b>snyk monitor</b>" to load the results into the Snyk App but for this demo we don't do that</li></ul></div><p>6. Now we can manually run a build or wait for triggering event on our repo, here is some screen shots of what it looks like including some failures where we find vulnerabilities in a separate node.js repo</p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgPwoJ4ga7iBML29n5CWICWG_xf5kjoUhhNZyjhQUGmv8E2WgAdwe7LMjTHVRsPOpnrwnoZKbGqw1IKhy1IQdNYwFvZIAnXadlDORlThyffNvVzxfxPaZegjZTTuvIPxWEEJ0A8G4pDMKDO/s2682/Screen+Shot+2021-09-15+at+2.05.17+pm.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="686" data-original-width="2682" height="82" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgPwoJ4ga7iBML29n5CWICWG_xf5kjoUhhNZyjhQUGmv8E2WgAdwe7LMjTHVRsPOpnrwnoZKbGqw1IKhy1IQdNYwFvZIAnXadlDORlThyffNvVzxfxPaZegjZTTuvIPxWEEJ0A8G4pDMKDO/s320/Screen+Shot+2021-09-15+at+2.05.17+pm.png" width="320" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEht1C4zo24WfL4WN4oIUiP6GA05js9K3I5MirccQB0OC5pQa3pCiGljaUhPcUAeFwymo22xafzkxVkuVjtV0Rn5mnd65c71RAqdAUX72NBl7EcwLyNKeOel1Rdc2BOlQNkwIXHfzj0nFxXD/s2048/Screen+Shot+2021-09-15+at+2.05.31+pm.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1076" data-original-width="2048" height="168" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEht1C4zo24WfL4WN4oIUiP6GA05js9K3I5MirccQB0OC5pQa3pCiGljaUhPcUAeFwymo22xafzkxVkuVjtV0Rn5mnd65c71RAqdAUX72NBl7EcwLyNKeOel1Rdc2BOlQNkwIXHfzj0nFxXD/s320/Screen+Shot+2021-09-15+at+2.05.31+pm.png" width="320" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiuuiDzBBALjMGyXkkzxDp8Dp-xQFWoz93lTgMG_Fxr3E6yZCDzud3ct99eLUtTXvdI1NYoDAOyl6mZEg_Bg7vFUXPJUylCLFHSswFpu9-zxeFRLzDSs5T_eNTOGlF0TfuOzwwSivyVI4IS/s2048/Screen+Shot+2021-09-15+at+2.05.51+pm.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1790" data-original-width="2048" height="280" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiuuiDzBBALjMGyXkkzxDp8Dp-xQFWoz93lTgMG_Fxr3E6yZCDzud3ct99eLUtTXvdI1NYoDAOyl6mZEg_Bg7vFUXPJUylCLFHSswFpu9-zxeFRLzDSs5T_eNTOGlF0TfuOzwwSivyVI4IS/s320/Screen+Shot+2021-09-15+at+2.05.51+pm.png" width="320" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjU-ojuc7qXROAGpgKN7iwKVnfQeFBT_m6csKX9gN9LB0xIn3ivQmCOkdITdYSqnY2FS9bxyks5o7wo-YAiUSEvspmp_8wPdSpkSGgRI9W1xyRCOeJGz_qxrtoWGauM1k-18FcAz7Dt_dsj/s2048/Screen+Shot+2021-09-15+at+2.06.41+pm.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1698" data-original-width="2048" height="265" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjU-ojuc7qXROAGpgKN7iwKVnfQeFBT_m6csKX9gN9LB0xIn3ivQmCOkdITdYSqnY2FS9bxyks5o7wo-YAiUSEvspmp_8wPdSpkSGgRI9W1xyRCOeJGz_qxrtoWGauM1k-18FcAz7Dt_dsj/s320/Screen+Shot+2021-09-15+at+2.06.41+pm.png" width="320" /></a></div><br /> <p></p><p>It makes more sense to create a Buildkite plugin for Snyk rather than execute commands using a script and here is an example of one below. Having said that the commands you run to execute a "<b>snyk test</b>" are simple enough to include in the pipeline YML without the need for a plugin here especially if you have infrastructure already setup with the ability to run the "<b>snyk cli</b>". A plugin would be the right approach here though as per the example below.</p><p><a href="https://github.com/seek-oss/snyk-buildkite-plugin">https://github.com/seek-oss/snyk-buildkite-plugin</a></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgAVcuNkZN4cLbMio6rr32tCUI-iS8AmPBhKcuhKecWwMWJEvoOpHiu8FypYTe9e2X9psN5BDTtbmKpQRq1sJjlX3r-QbJXu2pNGvdd3SLvkJ79N2U28_xS3DQhpvkMZqfrpIw780aYGglX/s2048/Screen+Shot+2021-09-15+at+2.09.10+pm.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1421" data-original-width="2048" height="222" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgAVcuNkZN4cLbMio6rr32tCUI-iS8AmPBhKcuhKecWwMWJEvoOpHiu8FypYTe9e2X9psN5BDTtbmKpQRq1sJjlX3r-QbJXu2pNGvdd3SLvkJ79N2U28_xS3DQhpvkMZqfrpIw780aYGglX/s320/Screen+Shot+2021-09-15+at+2.09.10+pm.png" width="320" /></a></div><p><br /></p><p>Hopefully you have seen how easy it is to continuously avoid known vulnerabilities in your dependencies and code, by integrating Snyk into your continuous integration pipeline with Buildkite.</p><h2 style="text-align: left;">More Information</h2><div style="text-align: left;">Snyk</div><div style="text-align: left;"><a href="http://snyk.io">http://snyk.io</a></div><div style="text-align: left;"><br />Buildkite<br /><a href="https://buildkite.com">https://buildkite.com</a></div><div class="blogger-post-footer">http://feeds.feedburner.com/TheBlasFromPas</div>Pas Apicellahttp://www.blogger.com/profile/09389663166398991762noreply@blogger.com0tag:blogger.com,1999:blog-6527688743456205256.post-56582767755515183982021-08-30T15:47:00.006+10:002021-08-30T19:27:43.654+10:00Using the Elastic Snyk module to visualize Snyk imported project data with Elastic Kibana <p><a href="http://snyk.io" target="_blank">Snyk</a> is an open source security platform designed to help software-driven businesses enhance developer security</p><div style="text-align: left;">Elastic (ELK) stack is a distributed, RESTful search and analytics engine capable of addressing a growing number of use cases. </div><div style="text-align: left;"><br /></div><div style="text-align: left;">In this blog we will use Elastic to visualize our vulnerability data from Snyk using the <a href="https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-snyk.html" target="_blank">Elastic Snyk Module</a>. </div><div style="text-align: left;"><br /></div><div style="text-align: left;">This module is used for ingesting data from the different Snyk API Endpoints. Currently supports these filesets:</div><div style="text-align: left;"><ul style="text-align: left;"><li><b>vulnerabilities fileset</b>: Collects all found vulnerabilities for the related organizations and projects</li><li><b>audit fileset</b>: Collects audit logging from Snyk, this can be actions like users, permissions, groups, api access and more.</li></ul></div><div style="text-align: left;">When you run the module, it performs a few tasks under the hood:</div><div style="text-align: left;"><ul style="text-align: left;"><li>Sets the default paths to the log files (but don’t worry, you can override the defaults)</li><li>Makes sure each multiline log event gets sent as a single event</li><li>Uses ingest node to parse and process the log lines, shaping the data into a structure suitable for visualizing in Kibana</li></ul></div><p style="text-align: left;">Here is how to get started with this currently BETA module</p><h3 style="text-align: left;">Steps </h3><p>1. First you will need an account on Snyk App and have imported a few projects so you have some vulnerability data to get started with. Here is an example of some imported projects which already exist in my Snyk App Account.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi_k3AD9kx2xbcR2SRi2ll-_G9Fy7w4HBMUg02G7vkPToXHO9_jU6UW_FAqr8Lm3AqFSfU3wmlOWtwkOaMn-QDQ4USGVREFydC86MtCbfiKAtnRkMsZK8yCaVA1bbPpp2bE5yKot5nGg71Z/s2048/Screen+Shot+2021-08-30+at+2.40.45+pm.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1716" data-original-width="2048" height="268" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi_k3AD9kx2xbcR2SRi2ll-_G9Fy7w4HBMUg02G7vkPToXHO9_jU6UW_FAqr8Lm3AqFSfU3wmlOWtwkOaMn-QDQ4USGVREFydC86MtCbfiKAtnRkMsZK8yCaVA1bbPpp2bE5yKot5nGg71Z/s320/Screen+Shot+2021-08-30+at+2.40.45+pm.png" width="320" /></a></div><br /><p>2. You will need an Elastic Cluster best way to get one of those if you don't have one is to head to the <a href="https://www.elastic.co/cloud/" target="_blank">Elastic Cloud Service</a> for a free trial</p><p>3. Next we need to install Elastic Filebeat. The quick start guide here is the best way to do that</p><p><a href="https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-installation-configuration.html">https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-installation-configuration.html</a></p><p>Couple of things here</p><p></p><ul style="text-align: left;"><li>You need to run this command to enable the snyk module as shown below. In this example the module is already enabled so the output would differ the first time you run this.</li></ul><p></p><div style="text-align: left;"><span style="color: #3d85c6;">$ ./filebeat modules enable snyk<br />Module snyk is already enabled</span></div><p></p><ul style="text-align: left;"><li>Make sure you have configured connectivity to your Elastic Cluster as per the doc above. The example below is for the Elastic Cloud Service itself.</li></ul><p></p><div style="text-align: left;"><span style="color: #3d85c6;">cloud.id: "staging:dXMtZWFzdC0xLmF3cy5mb3VuZC5pbyRjZWM2ZjI2MWE3NGJmMjRjZTMzYmI4ODExYjg0Mjk0ZiRjNmMyY2E2ZDA0MjI0OWFmMGNjN2Q3YTllOTYyNTc0Mw=="<br />cloud.auth: "filebeat_setup:YOUR_PASSWORD"</span></div><p>4. With Elastic Filebeat installed and configured now we can setup the Snyk module to do that we perform the following.</p><p>Edit "<b>./modules.d/snyk.yml</b>" </p><p>The following link shows how to configure the <b>snyk.yml</b> file and what settings are available</p><p><a href="https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-snyk.html">https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-snyk.html</a></p><p><b>Example Config to get started with: </b></p><div style="text-align: left;"><div><span style="color: #3d85c6;">- module: snyk</span></div><div><span style="color: #3d85c6;"> audit:</span></div><div><span style="color: #3d85c6;"> enabled: true</span></div><div><span style="color: #3d85c6;"> var.input: httpjson</span></div><div><span style="color: #3d85c6;"> var.audit_type: organization</span></div><div><span style="color: #3d85c6;"> var.audit_id: SNYK_ORG_ID</span></div><div><span style="color: #3d85c6;"> var.interval: 1h</span></div><div><span style="color: #3d85c6;"> var.api_token: SNYK_API_TOKEN</span></div><div><span style="color: #3d85c6;"> var.first_interval: 30d</span></div><div><span style="color: #3d85c6;"><br /></span></div><div><div><span style="color: #3d85c6;"> vulnerabilities:</span></div><div><span style="color: #3d85c6;"> enabled: true</span></div><div><span style="color: #3d85c6;"> var.interval: 1h</span></div><div><span style="color: #3d85c6;"> var.first_interval: 30d</span></div><div><span style="color: #3d85c6;"> var.api_token: SNYK_API_TOKEN</span></div><div><span style="color: #3d85c6;"> var.orgs:</span></div><div><span style="color: #3d85c6;"> - SNYK_ORG_ID_1</span></div><div><span style="color: #3d85c6;"> - SNYK_ORG_ID_2</span></div></div><div><br /></div><div><span style="color: #e69138;">Note: In this example we are obtaining data from 2 organizations</span></div><div><br /></div><div>You obtain your ORG_ID's and SNYK_TOKEN_ID from Snyk App as shown below</div><div><br /></div><div><b>Account Settings -> General</b></div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj5PTBkzu4LogaAkczuXeDhcBNvbxV83Nib2jsr49c5agpto4s30CR6IeJ0Xt2XqmMP1MZqbHMRqu8UzTmRY6C7Ihb_Ry43g2Rhon9qDNsYsUQL0ZPyS-08x8XAyF8LNrJM42xxxHtBsYX-/s2690/Screen+Shot+2021-08-30+at+3.01.02+pm.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="880" data-original-width="2690" height="105" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj5PTBkzu4LogaAkczuXeDhcBNvbxV83Nib2jsr49c5agpto4s30CR6IeJ0Xt2XqmMP1MZqbHMRqu8UzTmRY6C7Ihb_Ry43g2Rhon9qDNsYsUQL0ZPyS-08x8XAyF8LNrJM42xxxHtBsYX-/s320/Screen+Shot+2021-08-30+at+3.01.02+pm.png" width="320" /></a></div><br /><div><br /></div><div><b>Settings -> General -> Organization ID (For each orgnization you wish to use)</b></div><div> </div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEirHy7s4P0gp5uFfuAAyH-Ll4iyyHWKEK59rF7gtqaLoaskRwdq9NaAKHlbn7FUvb2rpCnl4mKhBIpyyIkfLTFqyOOOC2Tl5afGiLyh8-AP-LAcnaCrlX-OcQFB3aHLqJsROvhOaTBASrUV/s2684/Screen+Shot+2021-08-30+at+3.07.15+pm.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1150" data-original-width="2684" height="137" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEirHy7s4P0gp5uFfuAAyH-Ll4iyyHWKEK59rF7gtqaLoaskRwdq9NaAKHlbn7FUvb2rpCnl4mKhBIpyyIkfLTFqyOOOC2Tl5afGiLyh8-AP-LAcnaCrlX-OcQFB3aHLqJsROvhOaTBASrUV/s320/Screen+Shot+2021-08-30+at+3.07.15+pm.png" width="320" /></a></div><br /><div><br /></div></div><p>5. At this point we can start Elastic Filebeat as shown below.</p><div style="text-align: left;"><span style="color: #3d85c6;">$ ./filebeat -e<br /></span></div><div style="text-align: left;"><span style="color: #3d85c6;">...</span></div><div style="text-align: left;"><div><span style="color: #3d85c6;">2021-08-30T14:23:48.034+1000<span style="white-space: pre;"> </span>INFO<span style="white-space: pre;"> </span>[esclientleg]<span style="white-space: pre;"> </span>eslegclient/connection.go:273<span style="white-space: pre;"> </span>Attempting to connect to Elasticsearch version 7.14.0</span></div><div><span style="color: #3d85c6;">2021-08-30T14:23:48.163+1000<span style="white-space: pre;"> </span>INFO<span style="white-space: pre;"> </span>cfgfile/reload.go:224<span style="white-space: pre;"> </span>Loading of config files completed.</span></div><div><span style="color: #3d85c6;">2021-08-30T14:23:48.163+1000<span style="white-space: pre;"> </span>INFO<span style="white-space: pre;"> </span>[input.httpjson-cursor]<span style="white-space: pre;"> </span>compat/compat.go:111<span style="white-space: pre;"> </span>Input httpjson-cursor starting<span style="white-space: pre;"> </span>{"id": "BC01B4DEC1514B32"}</span></div><div><span style="color: #3d85c6;">2021-08-30T14:23:48.163+1000<span style="white-space: pre;"> </span>INFO<span style="white-space: pre;"> </span>[input.httpjson-cursor]<span style="white-space: pre;"> </span>compat/compat.go:111<span style="white-space: pre;"> </span>Input httpjson-cursor starting<span style="white-space: pre;"> </span>{"id": "303DFE9AECEEEF55"}</span></div><div><span style="color: #3d85c6;">2021-08-30T14:23:48.164+1000<span style="white-space: pre;"> </span>INFO<span style="white-space: pre;"> </span>[input.httpjson-cursor]<span style="white-space: pre;"> </span>v2/input.go:112<span style="white-space: pre;"> </span>Process another repeated request.<span style="white-space: pre;"> </span>{"id": "303DFE9AECEEEF55", "input_source": "https://snyk.io/api/v1/reporting/issues/?page=1&perPage=10&sortBy=issueTitle&order=asc&groupBy=issue", "input_url": "https://snyk.io/api/v1/reporting/issues/?page=1&perPage=10&sortBy=issueTitle&order=asc&groupBy=issue"}</span></div><div><span style="color: #3d85c6;">2021-08-30T14:23:48.164+1000<span style="white-space: pre;"> </span>INFO<span style="white-space: pre;"> </span>[input.httpjson-cursor]<span style="white-space: pre;"> </span>v2/input.go:112<span style="white-space: pre;"> </span>Process another repeated request.<span style="white-space: pre;"> </span>{"id": "BC01B4DEC1514B32", "input_source": "https://snyk.io/api/v1/org/yyyy/audit?page=1&sortOrder=ASC", "input_url": "https://snyk.io/api/v1/org/yyyy/audit?page=1&sortOrder=ASC"}</span></div><div><span style="color: #3d85c6;">2021-08-30T14:23:49.656+1000<span style="white-space: pre;"> </span>INFO<span style="white-space: pre;"> </span>[input.httpjson-cursor]<span style="white-space: pre;"> </span>v2/request.go:210<span style="white-space: pre;"> </span>request finished: 0 events published<span style="white-space: pre;"> </span>{"id": "BC01B4DEC1514B32", "input_source": "https://snyk.io/api/v1/org/yyyy/audit?page=1&sortOrder=ASC", "input_url": "https://snyk.io/api/v1/org/yyyy/audit?page=1&sortOrder=ASC"}</span></div><div><span style="color: #3d85c6;"><br /></span></div><div><span style="color: #3d85c6;">...</span></div><div><br /></div></div><p>6. If we head to Kibana and go into the "<b>Discover</b>" Page we will see data flowing into the cluster by setting "<b>event.module = snyk</b>" on the "<b>filebeat-*</b>" index pattern as shown below.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhOVQeuxhvx8hbaac_OW71d9SpLRFfXy5lQvtCoV0AAeGjGWULufTb8K64Ma_xCE3DY6f94T0Nh91L7f4iG2t7DhLoEiqLvK3Oqs_-RFtM38iyd4nR8RPW4WY7LSB98q7ZPr-zCOde9upRG/s2048/Screen+Shot+2021-08-30+at+3.15.49+pm.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1176" data-original-width="2048" height="184" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhOVQeuxhvx8hbaac_OW71d9SpLRFfXy5lQvtCoV0AAeGjGWULufTb8K64Ma_xCE3DY6f94T0Nh91L7f4iG2t7DhLoEiqLvK3Oqs_-RFtM38iyd4nR8RPW4WY7LSB98q7ZPr-zCOde9upRG/s320/Screen+Shot+2021-08-30+at+3.15.49+pm.png" width="320" /></a></div><p>Or maybe we want to use Dev Tools itself</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgCOtbzSiXTcEP59W0PHjXgtueKnW3iNkC_SIFrRL44J_Q7D40ve0PJZuJs-VI8MN28VaCTZWz2-hQNnlzHJYq4IEj9e8WqRgR9Dt1vZZOSkWaYAZ-0QnibKBkOKVqPTA9YF_6LgxZ0LoG2/s2048/Screen+Shot+2021-08-30+at+3.42.59+pm.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1233" data-original-width="2048" height="193" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgCOtbzSiXTcEP59W0PHjXgtueKnW3iNkC_SIFrRL44J_Q7D40ve0PJZuJs-VI8MN28VaCTZWz2-hQNnlzHJYq4IEj9e8WqRgR9Dt1vZZOSkWaYAZ-0QnibKBkOKVqPTA9YF_6LgxZ0LoG2/s320/Screen+Shot+2021-08-30+at+3.42.59+pm.png" width="320" /></a></div><p><br /></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj1TspIqs2ST9ezBa72ST-X84D9kMltJyZJ4lFtbiZNRSQI0oG2K2AbUf-5mtJrN321X8g-5PqP2kiWPeTfGHNA_lrYTfJCE3e9a-kksUzXem448yBwlfkdFFJBe7-e1oeM0plKZoM0TcEn/s2048/Screen+Shot+2021-08-30+at+5.22.05+pm.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1179" data-original-width="2048" height="184" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj1TspIqs2ST9ezBa72ST-X84D9kMltJyZJ4lFtbiZNRSQI0oG2K2AbUf-5mtJrN321X8g-5PqP2kiWPeTfGHNA_lrYTfJCE3e9a-kksUzXem448yBwlfkdFFJBe7-e1oeM0plKZoM0TcEn/s320/Screen+Shot+2021-08-30+at+5.22.05+pm.png" width="320" /></a></div><p><br /></p><p>7. Finally a Dashboard of some graphs as we monitor the "<b>Snyk Data</b>" coming into the cluster </p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgYR_37e6gB0qOr_CoJKNHHRSwGFXiiuxdg3w2ik092X-BsgGW2xmxbC4Suo7Zo8fFcCI5bMbvdJOByNn32-Zs_MSdJ63T3vNy9yj8HieOgWJ0aUIsBxJZSIcUGF0ZLExripf77NBqCuI_7/s2048/Screen+Shot+2021-08-30+at+3.26.23+pm.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1247" data-original-width="2048" height="195" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgYR_37e6gB0qOr_CoJKNHHRSwGFXiiuxdg3w2ik092X-BsgGW2xmxbC4Suo7Zo8fFcCI5bMbvdJOByNn32-Zs_MSdJ63T3vNy9yj8HieOgWJ0aUIsBxJZSIcUGF0ZLExripf77NBqCuI_7/s320/Screen+Shot+2021-08-30+at+3.26.23+pm.png" width="320" /></a></div><p><br /></p><p><b>Next Steps?</b></p><p>Kibana gives you the freedom to select the way you give shape to your
data. With its interactive visualizations, start with one question and
see where it leads you!!!. If it was me I would go straight to Kibana Lens. Kibana Lens is an easy-to-use, intuitive UI that simplifies the process of data visualization through a drag-and-drop experience it allowed me to create the dashboard page above in under 3 minutes even without knowing where to start.</p><p><a href="https://www.elastic.co/kibana/kibana-lens">https://www.elastic.co/kibana/kibana-lens</a></p><p>Want all your vulnerability data form the <a href="https://app.snyk.io" target="_blank">Snyk Platform</a> within your Elastic Stack the "<a href="https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-snyk.html" target="_blank">Elastic Snyk Module</a>" will do that for you in a few easy steps.</p><h1 style="text-align: left;">More Information</h1><div style="text-align: left;">Snyk</div><div style="text-align: left;"><a href="http://snyk.io">http://snyk.io</a></div><div style="text-align: left;"><br /></div><div style="text-align: left;">Elastic Snyk Module</div><div style="text-align: left;"><a href="https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-snyk.html">https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-snyk.html</a></div><div class="blogger-post-footer">http://feeds.feedburner.com/TheBlasFromPas</div>Pas Apicellahttp://www.blogger.com/profile/09389663166398991762noreply@blogger.com0tag:blogger.com,1999:blog-6527688743456205256.post-52739111007615991992021-08-27T08:25:00.001+10:002021-08-27T08:25:23.181+10:00Snyk Container meets Cloud Native Buildpacks: Cloud Native Application Security the right way<p><span class="break-words"><span dir="ltr">Released my first Snyk
Blog post on how Snyk Container along with Cloud Native Buildpacks can
mitigate the security risks of containerized applications.</span></span></p><p><a href="https://snyk.io/blog/snyk-container-cloud-native-buildpacks-cloud-native-application-security/">https://snyk.io/blog/snyk-container-cloud-native-buildpacks-cloud-native-application-security/</a></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgw410wQf50j2BBvChPoQ53K1CxheCCxtE8uKjzLjP5i-e9LrZbEsLtqEU0sofRh2J0W_knnN-3qgI5RzMXE4iFj_2yPl9w9dSds4KYHa8rXoy5p5Zwid2iiBgoiVaVfL1R9nckuOgd75N4/s2048/Screen+Shot+2021-08-27+at+8.23.28+am.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1539" data-original-width="2048" height="240" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgw410wQf50j2BBvChPoQ53K1CxheCCxtE8uKjzLjP5i-e9LrZbEsLtqEU0sofRh2J0W_knnN-3qgI5RzMXE4iFj_2yPl9w9dSds4KYHa8rXoy5p5Zwid2iiBgoiVaVfL1R9nckuOgd75N4/s320/Screen+Shot+2021-08-27+at+8.23.28+am.png" width="320" /></a></div><br /><p><br /></p><div class="blogger-post-footer">http://feeds.feedburner.com/TheBlasFromPas</div>Pas Apicellahttp://www.blogger.com/profile/09389663166398991762noreply@blogger.com0tag:blogger.com,1999:blog-6527688743456205256.post-70464580684107718782021-08-20T11:45:00.000+10:002021-08-20T11:45:43.287+10:00Taking Snyk Code for Test Drive<p>Snyk Code is our newest addition to the Snyk platform. Snyk Code uses a revolutionary approach designed to be developer-first. Conventional Static Application Security Testing (SAST) tools are limited by lengthy scans times and poor accuracy, returning too many false positives, and eroding developer trust. Snyk Code makes developer efforts efficient and actionable.</p><p>In this short blog post let's take it for a test drive on a large Java Project.</p><h2 style="text-align: left;">Steps</h2><p>1. You will need an account on Snyk App sign up for free at <a href="http://snyk.io">snyk.io</a></p><p>2. Once you have an account and are logged in go ahead and enable "<b>Snyk Code</b>" as follows</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgiHVqY5qUYEzO51-VMRl8fuC2KW_qUzFKLR6-RpZ8B8c55gyLsTXvUfv1F8m17fbjgEr1KrOmLOCRdtTX8sfdDx3Hg7EucmWL1GFM4FRIIDsol8FqXadOnaFTkQyM4LpW0QeHNysjHfS6C/s2048/Screen+Shot+2021-08-20+at+11.23.44+am.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1193" data-original-width="2048" height="186" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgiHVqY5qUYEzO51-VMRl8fuC2KW_qUzFKLR6-RpZ8B8c55gyLsTXvUfv1F8m17fbjgEr1KrOmLOCRdtTX8sfdDx3Hg7EucmWL1GFM4FRIIDsol8FqXadOnaFTkQyM4LpW0QeHNysjHfS6C/s320/Screen+Shot+2021-08-20+at+11.23.44+am.png" width="320" /></a></div><p><br /></p><p>3. Clone the following GitHub repo as follows.</p><p><a href="https://github.com/papicella/CWE-Juliet-TestSuite-Java">https://github.com/papicella/CWE-Juliet-TestSuite-Java</a></p><div style="text-align: left;"><div><span style="color: #3d85c6;">$ git clone https://github.com/papicella/CWE-Juliet-TestSuite-Java</span></div><div><span style="color: #3d85c6;">Cloning into 'CWE-Juliet-TestSuite-Java'...</span></div><div><span style="color: #3d85c6;">remote: Enumerating objects: 12964, done.</span></div><div><span style="color: #3d85c6;">remote: Counting objects: 100% (12964/12964), done.</span></div><div><span style="color: #3d85c6;">remote: Compressing objects: 100% (969/969), done.</span></div><div><span style="color: #3d85c6;">remote: Total 12964 (delta 11931), reused 12964 (delta 11931), pack-reused 0</span></div><div><span style="color: #3d85c6;">Receiving objects: 100% (12964/12964), 6.44 MiB | 6.04 MiB/s, done.</span></div><div><span style="color: #3d85c6;">Resolving deltas: 100% (11931/11931), done.</span></div><div><span style="color: #3d85c6;">Updating files: 100% (13095/13095), done.</span></div></div><p>4. Authenticate with "<b>Snyk CLI</b>" as shown below</p><p><span style="color: #ffa400;">Note: The following link will get the CLI installed for you</span></p><p><a href="https://support.snyk.io/hc/en-us/articles/360003812538-Install-the-Snyk-CLI">https://support.snyk.io/hc/en-us/articles/360003812538-Install-the-Snyk-CLI</a></p><p><span style="color: #3d85c6;">$ snyk auth</span></p><p>5. Let's check the number of lines of code we scanning as follows. Here we are making sure we only scan Java Source files as that's all that exists in this repo in any case. Snyk Code supports other programming languages as well like Python, JavaScript etc.</p><div style="text-align: left;"><span style="color: #3d85c6;">$ cd CWE-Juliet-TestSuite-Java</span></div><div style="text-align: left;"><div><span style="color: #3d85c6;">$ find ./Java/src -name "*.java" -type f -exec wc -l {} \; | awk '{total += $1} END{print total}'</span></div><div><span style="color: #3d85c6;">2,479,301</span></div></div><p>So we have close 2.5 million lines of code here of course that counts lines of empty space such as carriage returns BUT still that's lots of code to scan.</p><p>6. Run your "<b>Snyk Code</b>" Test as follows to see two things , how long the scan took and the number of results returned. From this we can see the scan and results took less then 2 minutes!</p><p><span style="color: #3d85c6;">$ time snyk code test ./Java/src</span></p><p><span style="color: #3d85c6;">....</span></p><div style="text-align: left;"><div><span style="color: #3d85c6;"> ✗ [High] SQL Injection</span></div><div><span style="color: #3d85c6;"> Path: testcases/CWE89_SQL_Injection/s02/CWE89_SQL_Injection__getCookies_Servlet_execute_41.java, line 42</span></div><div><span style="color: #3d85c6;"> Info: Unsanitized input from cookies flows into execute, where it is used in an SQL query. This may result in an SQL Injection vulnerability.</span></div><div><span style="color: #3d85c6;"><br /></span></div><div><span style="color: #3d85c6;"> ✗ [High] SQL Injection</span></div><div><span style="color: #3d85c6;"> Path: testcases/CWE89_SQL_Injection/s02/CWE89_SQL_Injection__getCookies_Servlet_execute_14.java, line 62</span></div><div><span style="color: #3d85c6;"> Info: Unsanitized input from cookies flows into execute, where it is used in an SQL query. This may result in an SQL Injection vulnerability.</span></div><div><span style="color: #3d85c6;"><br /></span></div><div><span style="color: #3d85c6;"> ✗ [High] SQL Injection</span></div><div><span style="color: #3d85c6;"> Path: testcases/CWE89_SQL_Injection/s02/CWE89_SQL_Injection__getCookies_Servlet_execute_10.java, line 62</span></div><div><span style="color: #3d85c6;"> Info: Unsanitized input from cookies flows into execute, where it is used in an SQL query. This may result in an SQL Injection vulnerability.</span></div><div><span style="color: #3d85c6;"><br /></span></div><div><span style="color: #3d85c6;"><br /></span></div><div><span style="color: #3d85c6;">✔ Test completed</span></div><div><span style="color: #3d85c6;"><br /></span></div><div><span style="color: #3d85c6;">Organization: undefined</span></div><div><span style="color: #3d85c6;">Test type: Static code analysis</span></div><div><span style="color: #3d85c6;">Project path: ./Java/src</span></div><div><span style="color: #3d85c6;"><br /></span></div><div><span style="color: #3d85c6;">3086 Code issues found</span></div><div><span style="color: #3d85c6;">1491 [High] 1595 [Medium]</span></div><div><span style="color: #3d85c6;"><br /></span></div><div><span style="color: #3d85c6;"><br /></span></div><div><span style="color: #3d85c6;">real<span style="white-space: pre;"> </span>1m4.269s</span></div><div><span style="color: #3d85c6;">user<span style="white-space: pre;"> </span>0m45.630s</span></div><div><span style="color: #3d85c6;">sys<span style="white-space: pre;"> </span>0m7.182s</span></div><div><br /></div></div><p>The following post shows some comparison data against other SAST engines and code repositories.</p><p><a href="https://snyk.io/blog/sast-tools-speed-comparison-snyk-code-sonarqube-lgtm/">https://snyk.io/blog/sast-tools-speed-comparison-snyk-code-sonarqube-lgtm/</a></p><h2 style="text-align: left;">More Information</h2><div style="text-align: left;">Snyk Code</div><div style="text-align: left;"><a href="https://snyk.io/product/snyk-code/">https://snyk.io/product/snyk-code/</a></div><div style="text-align: left;"><br /></div><div style="text-align: left;">Snyk Code is now available for free</div><div style="text-align: left;"><a href="https://snyk.io/blog/snyk-code-now-available-free-sast/">https://snyk.io/blog/snyk-code-now-available-free-sast/</a></div><div class="blogger-post-footer">http://feeds.feedburner.com/TheBlasFromPas</div>Pas Apicellahttp://www.blogger.com/profile/09389663166398991762noreply@blogger.com0tag:blogger.com,1999:blog-6527688743456205256.post-2981516333947967982021-07-23T14:56:00.000+10:002021-07-23T14:56:03.546+10:00Snyk provides native integration for Atlassian Bitbucket Cloud security - Here is how!!!<p>The Snyk security integration is free and easy to set up with just a
few clicks inside the Bitbucket Cloud product. For the first time,
developers can consume information that was previously only available
inside Snyk now within Bitbucket Cloud. Snyk enables developers to see
new vulnerabilities as they emerge and implement fixes early and quickly
in the process. </p><p>In this post we show how easily you can integrate Snyk into Bitbucket Cloud</p><h2 style="text-align: left;">Steps</h2><div style="text-align: left;"><i><span style="color: #ffa400;">Note: For the following to work you have to integrate Bitbucket Cloud with the Snyk App as per this link </span></i></div><div style="text-align: left;"><br /></div><div style="text-align: left;"><a href="https://support.snyk.io/hc/en-us/articles/360004032097-Bitbucket-Cloud-how-it-works">https://support.snyk.io/hc/en-us/articles/360004032097-Bitbucket-Cloud-how-it-works</a></div><div style="text-align: left;"><br /></div><div style="text-align: left;">1. Once logged into Bitbucket Cloud navigate to your "<b>Workplace Settings</b>" and authenticate with Snyk as shown below.</div><div style="text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjGY3sGfXukYKvKaFu0f7B5GnmPPuXG3wQsOIgcNYiioq6TtbaNOtPXSrgi8P3A2nJCcOOmPANk8LeUrDpf-d7W6bmYDxOmYDv67J-KBSAm5t_gTGYu5fN9w0N0DiQl3gXTgz9BLmrPFsD2/s2048/Screen+Shot+2021-07-23+at+1.23.44+pm.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1662" data-original-width="2048" height="325" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjGY3sGfXukYKvKaFu0f7B5GnmPPuXG3wQsOIgcNYiioq6TtbaNOtPXSrgi8P3A2nJCcOOmPANk8LeUrDpf-d7W6bmYDxOmYDv67J-KBSAm5t_gTGYu5fN9w0N0DiQl3gXTgz9BLmrPFsD2/w400-h325/Screen+Shot+2021-07-23+at+1.23.44+pm.png" width="400" /></a></div><br /><div style="text-align: left;"><br /></div><div style="text-align: left;"><br /></div><div style="text-align: left;"><br /></div><div style="text-align: left;">2. Next select the repository you wish to use as shown below.</div><div style="text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiRnKAg8toTVB-7_sSvm63SgUMxUlE68xqsUIlJBTTn-5YT9tj3o0dpVRw5KNt9EQ_-cCfElPTFqhcD8lc67P2Je6i1W9go4qi2Np3kAbTlT_qcHVHFhClR1LXRLnlynj8uKcOuXA8xg170/s3212/Screen+Shot+2021-07-23+at+1.24.49+pm.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="979" data-original-width="3212" height="122" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiRnKAg8toTVB-7_sSvm63SgUMxUlE68xqsUIlJBTTn-5YT9tj3o0dpVRw5KNt9EQ_-cCfElPTFqhcD8lc67P2Je6i1W9go4qi2Np3kAbTlT_qcHVHFhClR1LXRLnlynj8uKcOuXA8xg170/w400-h122/Screen+Shot+2021-07-23+at+1.24.49+pm.png" width="400" /></a></div><br /><div style="text-align: left;"><br /></div><div style="text-align: left;"><br /></div><div style="text-align: left;"><br /></div><div style="text-align: left;">3. You should see a "<b>Snyk</b>" tab on the left hand side click on this and then click on "<b>Import this repository</b>" as shown below.</div><div style="text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgB6QC3b-g6rmGQqb9q5Uwe2-woyZmDoqXNrwHu-ZZ2ZWT8dXwe93RhW2e2zj278b_CTS84Bdr4abOrDTxNLaMwzhcGNOmY_rvYq_OPyXpyTl05ANCGtjDcKwK9wyNGzHc6VmyXPjCU2ZxP/s2635/Screen+Shot+2021-07-23+at+1.25.08+pm.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1193" data-original-width="2635" height="181" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgB6QC3b-g6rmGQqb9q5Uwe2-woyZmDoqXNrwHu-ZZ2ZWT8dXwe93RhW2e2zj278b_CTS84Bdr4abOrDTxNLaMwzhcGNOmY_rvYq_OPyXpyTl05ANCGtjDcKwK9wyNGzHc6VmyXPjCU2ZxP/w400-h181/Screen+Shot+2021-07-23+at+1.25.08+pm.png" width="400" /></a></div><br /><div style="text-align: left;"><br /></div><div style="text-align: left;"><br /></div><div style="text-align: left;">4. In a couple of minutes or less you should see a "<b>Snyk</b>" overview report as per below. </div><div style="text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhmqA7Oz5AcXJDhLl-kGq9u_Wex8JM1m_XSUesEGRJ198YDVwh_Tx-QuVLKZoRy5HMq9iGIkrZ-mA2mrCoawrbW7FOKwGUVWpC9f7qbQSh4lAMnjpCeevMXRWoExv2fq13ZMlOwEZT4seA7/s2763/Screen+Shot+2021-07-23+at+1.26.56+pm.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1138" data-original-width="2763" height="165" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhmqA7Oz5AcXJDhLl-kGq9u_Wex8JM1m_XSUesEGRJ198YDVwh_Tx-QuVLKZoRy5HMq9iGIkrZ-mA2mrCoawrbW7FOKwGUVWpC9f7qbQSh4lAMnjpCeevMXRWoExv2fq13ZMlOwEZT4seA7/w400-h165/Screen+Shot+2021-07-23+at+1.26.56+pm.png" width="400" /></a></div><br /><div style="text-align: left;"><br /></div><div style="text-align: left;"><br /></div><div style="text-align: left;">5. Click on "<b>pom.xml</b>" to get more information as shown below. Here we get a list of all Vulnerabilities as per a scan of the package manifest file "<b>pom.xml</b>" in this example.</div><div style="text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhuY5SXTyNaQHixJb6aDeObRXzUVfENnOUuTvLsOVjIrpWJ9FYq7qAvqUlwh5uMSVj6253_gZUD2NiQEwN8fFb2FYQ4u8WSJNizxl2WOZPuVFmp30yH0QrASQn81PvJteIPD5p-dhKBjqr6/s2048/Screen+Shot+2021-07-23+at+1.29.57+pm.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1807" data-original-width="2048" height="353" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhuY5SXTyNaQHixJb6aDeObRXzUVfENnOUuTvLsOVjIrpWJ9FYq7qAvqUlwh5uMSVj6253_gZUD2NiQEwN8fFb2FYQ4u8WSJNizxl2WOZPuVFmp30yH0QrASQn81PvJteIPD5p-dhKBjqr6/w400-h353/Screen+Shot+2021-07-23+at+1.29.57+pm.png" width="400" /></a></div><br /><div style="text-align: left;"><br /></div><div style="text-align: left;"><br /></div><div style="text-align: left;"><br /></div><div style="text-align: left;">6. On the top of this page you can directly go to the project page on "<b>Snyk App</b>" by clicking on "<b>Visit Snyk</b>" as shown below.</div><div style="text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiv69oQ6mK6yC3NEKmYZSMkdga6BRJDj8FBCndxhd5VflRuOKTiXB_hzMwiUnN1IDgxdN07L8gH7rW6YHiWYuIJB2d7vIZO8LN4HpOr-f5oaWCnxyLAN8-UgjHyuUgJKhGlAwxF0NWQgqFa/s2048/Screen+Shot+2021-07-23+at+1.30.56+pm.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1506" data-original-width="2048" height="294" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiv69oQ6mK6yC3NEKmYZSMkdga6BRJDj8FBCndxhd5VflRuOKTiXB_hzMwiUnN1IDgxdN07L8gH7rW6YHiWYuIJB2d7vIZO8LN4HpOr-f5oaWCnxyLAN8-UgjHyuUgJKhGlAwxF0NWQgqFa/w400-h294/Screen+Shot+2021-07-23+at+1.30.56+pm.png" width="400" /></a></div><br /><div style="text-align: left;"><br /></div><div style="text-align: left;"><br /></div><div style="text-align: left;">It's as simple as that!</div><div style="text-align: left;"> </div><h2 style="text-align: left;">More Information</h2><div style="text-align: left;">Demo Video</div><div style="text-align: left;"><a href="https://www.youtube.com/watch?v=IqRjH7zkxiM">https://www.youtube.com/watch?v=IqRjH7zkxiM</a></div><div style="text-align: left;"><br /></div><div style="text-align: left;">Snyk provides native integration for Atlassian Bitbucket Cloud security<br /><a href="https://snyk.io/blog/snyk-native-integration-atlassian-bitbucket-cloud-security/?utm_campaign=PR-LH-Bitbucket-2021&utm_medium=Social&utm_source=Linkedin-Organic">https://snyk.io/blog/snyk-native-integration-atlassian-bitbucket-cloud-security/?utm_campaign=PR-LH-Bitbucket-2021&utm_medium=Social&utm_source=Linkedin-Organic</a></div><div class="blogger-post-footer">http://feeds.feedburner.com/TheBlasFromPas</div>Pas Apicellahttp://www.blogger.com/profile/09389663166398991762noreply@blogger.com0tag:blogger.com,1999:blog-6527688743456205256.post-64222235604033642362021-07-02T11:43:00.001+10:002021-07-02T11:43:42.241+10:00Snyk Import Project API using Azure Repos<p>In this example below we show how you could import a Azure Repo directly from the <a href="https://snyk.docs.apiary.io/#" target="_blank">Snyk API</a>. The Snyk API is available to customers on paid plans and allows you to programatically integrate with Snyk.</p><p><b>API url</b></p><p>The base URL for all API endpoints is <a href="https://snyk.io/api/v1/">https://snyk.io/api/v1/</a></p><p><b>Authorization</b></p><p>To use this API, you must get your token from Snyk. It can be seen on <a href="https://snyk.io/account/">https://snyk.io/account/</a> after you register with Snyk and login.</p><p>The token should be supplied in an Authorization header with the token, preceded by token:</p><p><span style="color: #3d85c6;">Authorization: token API_KEY</span></p><h2 style="text-align: left;">Steps</h2><p><span style="color: #e69138;">Note: You would have configured an Azure Repos Integration as shown below prior to running these steps</span></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjBJPSNByjFaX5NTcMsvfB2aiitXgt9ZBMTjscKO8IgB9qUTLnSNVHgTgDhWmhNK4JVb8iiAk8GlMveV1YmfRbpqXvUTa6DBI25GTVxYUbnVRQcAAOkwQxQa3w9XDSBXUGtwk2fhgcViLCt/s380/Screen+Shot+2021-07-02+at+10.39.28+am.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="352" data-original-width="380" height="185" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjBJPSNByjFaX5NTcMsvfB2aiitXgt9ZBMTjscKO8IgB9qUTLnSNVHgTgDhWmhNK4JVb8iiAk8GlMveV1YmfRbpqXvUTa6DBI25GTVxYUbnVRQcAAOkwQxQa3w9XDSBXUGtwk2fhgcViLCt/w200-h185/Screen+Shot+2021-07-02+at+10.39.28+am.png" width="200" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgPpquA90Pd1EXv_Z0vUF77v4hEJa1VksET1T3A4e9gr_OGTZA-e3oXSmllM7nIU5OW9i3K3UhyrILawFRo4a-lkS9sPIGWarM4VhIQiyKiUyzfZ7J3tn4ymGhBRplk01ooMDl9el_m11Ss/s2607/Screen+Shot+2021-07-02+at+10.41.00+am.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1206" data-original-width="2607" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgPpquA90Pd1EXv_Z0vUF77v4hEJa1VksET1T3A4e9gr_OGTZA-e3oXSmllM7nIU5OW9i3K3UhyrILawFRo4a-lkS9sPIGWarM4VhIQiyKiUyzfZ7J3tn4ymGhBRplk01ooMDl9el_m11Ss/s320/Screen+Shot+2021-07-02+at+10.41.00+am.png" width="320" /></a></div><br /><p><br /></p><p>1. Authenticate with the CLI using your Snyk Token as follows</p><p><span style="color: #3d85c6;">$ snyk auth TOKEN</span></p><p><span style="color: #3d85c6;">Your account has been authenticated. Snyk is now ready to be used.</span></p><p>2. Log into your Azure DevOps account and verify which Project Repo you wish to import as per the image below</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj3AQazbyNDzdjeimYG1d8gF1Brp7v7s5g39Rmmlps4LFHz-fqcBB_Xv3CxFv1VYQvIbaTm2Xk3jzTrKcxVP2t_HModH31G2IPDF_ZRmSR8njjBtTuzLCWuF17757qKwUUS2LX98CLMt_Xe/s3162/Screen+Shot+2021-07-02+at+10.42.31+am.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="894" data-original-width="3162" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj3AQazbyNDzdjeimYG1d8gF1Brp7v7s5g39Rmmlps4LFHz-fqcBB_Xv3CxFv1VYQvIbaTm2Xk3jzTrKcxVP2t_HModH31G2IPDF_ZRmSR8njjBtTuzLCWuF17757qKwUUS2LX98CLMt_Xe/s320/Screen+Shot+2021-07-02+at+10.42.31+am.png" width="320" /></a></div><p><br /></p><p>3. With your project selected you will need the following details to perform the Snyk API import request</p><p><b>owner: Name of your project</b></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgzqa97WVINUfW_K26-edLucKaEp_O35diBgHbvtH9VEDHOkOdtbWhRdTQT_M8lgBQhpyIrxNA43IX408JNZDA2ffpliwvC_Zu5BiDjiqPK-djLsH6cS1EaVIY0MR66rqahC2M4sVqebflB/s2534/Screen+Shot+2021-07-02+at+10.46.50+am.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1241" data-original-width="2534" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgzqa97WVINUfW_K26-edLucKaEp_O35diBgHbvtH9VEDHOkOdtbWhRdTQT_M8lgBQhpyIrxNA43IX408JNZDA2ffpliwvC_Zu5BiDjiqPK-djLsH6cS1EaVIY0MR66rqahC2M4sVqebflB/s320/Screen+Shot+2021-07-02+at+10.46.50+am.png" width="320" /></a></div><br /><p>name: Name of your Repository</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiLsRUR2OL3uRjhvhLkSKga0_mx4tj6SfAbC_XLpQ74wwdpmhWCXFFNnXvuUEz2jlcj3LZl2trEsTCM-QoiYa48H5nUZH25mlLHt53qsbiFubkNpU4_y2GSPTCWCejdW19oX17hMz9XEny1/s2534/Screen+Shot+2021-07-02+at+10.46.50+am+2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1241" data-original-width="2534" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiLsRUR2OL3uRjhvhLkSKga0_mx4tj6SfAbC_XLpQ74wwdpmhWCXFFNnXvuUEz2jlcj3LZl2trEsTCM-QoiYa48H5nUZH25mlLHt53qsbiFubkNpU4_y2GSPTCWCejdW19oX17hMz9XEny1/s320/Screen+Shot+2021-07-02+at+10.46.50+am+2.png" width="320" /></a></div><p><br /></p><p>branch: Branch name you wish to import</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiBdopwDMg532KnpQ55Dm0ELYUehP04fyOwmoV8BqDwsuDcIRQMiBKd-2vlQw-9QvKYeQ_XhbFO3fqDMFMYYKxFhE40zcL3jxFlafZCKKxh1svXvqG25HLu7FRbmEeNp1YIQm7oPDvpARzq/s2534/Screen+Shot+2021-07-02+at+10.46.50+am+3.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1241" data-original-width="2534" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiBdopwDMg532KnpQ55Dm0ELYUehP04fyOwmoV8BqDwsuDcIRQMiBKd-2vlQw-9QvKYeQ_XhbFO3fqDMFMYYKxFhE40zcL3jxFlafZCKKxh1svXvqG25HLu7FRbmEeNp1YIQm7oPDvpARzq/s320/Screen+Shot+2021-07-02+at+10.46.50+am+3.png" width="320" /></a></div><br /><p><br /></p><p>4. Finally we will need our Organisation ID and Azure Repos ID which we retrieve from the Snyk UI as follows</p><p><b>organization ID</b></p><p>Setting > General -> Organization ID</p><p><b>Azure Repos ID</b></p><p>Setting > Integrations -> Azure Repos -> Edit Settings -> Scroll to bottom of page</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjyKBAuTvkZ1V2Aije-qtl3R8xJeGV2Nnq72qvFjY8suTLIiPeSRYS8o7e2jkbZ74ISSDsy-ctccXzs1E0h9tVgRHtCzaCx7baEkR9KuDOBG-tlSukXombAMF1093xHK3cuzLmjrhTPl4nw/s2620/Screen+Shot+2021-07-02+at+11.07.49+am.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="500" data-original-width="2620" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjyKBAuTvkZ1V2Aije-qtl3R8xJeGV2Nnq72qvFjY8suTLIiPeSRYS8o7e2jkbZ74ISSDsy-ctccXzs1E0h9tVgRHtCzaCx7baEkR9KuDOBG-tlSukXombAMF1093xHK3cuzLmjrhTPl4nw/s320/Screen+Shot+2021-07-02+at+11.07.49+am.png" width="320" /></a></div><br /><p><br /></p><p>Take of note of both ID's we will need them in the steps below</p><p>5. Now we are ready to import our Azure Repo using a command as follows. We are using a "curl" command to issue a POST request and this command is for a Mac OSX or linux system so it would need tweaking if running on Windows for example</p><p>Command</p><div style="text-align: left;"><div><span style="color: #3d85c6;">curl --include \</span></div><div><span style="color: #3d85c6;"> --request POST \</span></div><div><span style="color: #3d85c6;"> --header "Content-Type: application/json; charset=utf-8" \</span></div><div><span style="color: #3d85c6;"> --header "Authorization: token `snyk config get api`" \</span></div><div><span style="color: #3d85c6;"> --data-binary "{</span></div><div><span style="color: #3d85c6;"> \"target\": {</span></div><div><span style="color: #3d85c6;"> \"owner\": \"spring-book-service\",</span></div><div><span style="color: #3d85c6;"> \"name\": \"spring-book-service\",</span></div><div><span style="color: #3d85c6;"> \"branch\": \"master\"</span></div><div><span style="color: #3d85c6;"> }</span></div><div><span style="color: #3d85c6;">}" \</span></div><div><span style="color: #3d85c6;">'https://snyk.io/api/v1/org/$ORG_ID/integrations/$AZURE_REPO_ID/import'</span></div></div><p>6. Set the ENV variables for ORG_ID and AZURE_REPO_ID as follows</p><div style="text-align: left;"><span style="color: #3d85c6;">export ORG_ID=FROM_STEP-4<br />export AZURE_REPO_ID=FROM_STEP-4</span></div><p>7. Run command</p><div style="text-align: left;"><div><span style="color: #3d85c6;">$ curl --include \</span></div><div><span style="color: #3d85c6;">> --request POST \</span></div><div><span style="color: #3d85c6;">> --header "Content-Type: application/json; charset=utf-8" \</span></div><div><span style="color: #3d85c6;">> --header "Authorization: token `snyk config get api`" \</span></div><div><span style="color: #3d85c6;">> --data-binary "{</span></div><div><span style="color: #3d85c6;">> \"target\": {</span></div><div><span style="color: #3d85c6;">> \"owner\": \"spring-book-service\",</span></div><div><span style="color: #3d85c6;">> \"name\": \"spring-book-service\",</span></div><div><span style="color: #3d85c6;">> \"branch\": \"master\"</span></div><div><span style="color: #3d85c6;">> }</span></div><div><span style="color: #3d85c6;">> }" \</span></div><div><span style="color: #3d85c6;">> "https://snyk.io/api/v1/org/${ORG_ID}/integrations/${AZURE_REPO_ID}/import"</span></div><div><span style="color: #3d85c6;">HTTP/2 201</span></div><div><span style="color: #3d85c6;">content-security-policy: base-uri 'none'; script-src 'self' https: 'nonce-ENlk6rSQsdLgbcWNcCaA7A==' 'unsafe-inline' 'unsafe-eval' 'strict-dynamic' 'report-sample'; img-src https: data:; object-src 'none'; report-to csp-report-group; report-uri https://web-reports.snyk.io/csp?version=39e8721a74a6dffcb97fc790cfbf1fca91cefc03;</span></div><div><span style="color: #3d85c6;">report-to: {"group":"csp-report-group","max_age":1800,"endpoints":[{"url":"https://web-reports.snyk.io/csp?version=39e8721a74a6dffcb97fc790cfbf1fca91cefc03"}],"include_subdomains":true}</span></div><div><span style="color: #3d85c6;">x-snyk-version: undefined</span></div><div><span style="color: #3d85c6;">snyk-request-id: 1669e85e-abe5-401b-80bb-dae41829d6e1</span></div><div><span style="color: #3d85c6;">location: https://snyk.io/api/v1/org/.....</span></div><div><span style="color: #3d85c6;">content-type: application/json; charset=utf-8</span></div><div><span style="color: #3d85c6;">content-length: 2</span></div><div><span style="color: #3d85c6;">etag: W/"2-vyGp6PvFo4RvsFtPoIWeCReyIC8"</span></div><div><span style="color: #3d85c6;">date: Fri, 02 Jul 2021 01:39:45 GMT</span></div><div><span style="color: #3d85c6;">x-frame-options: deny</span></div><div><span style="color: #3d85c6;">x-content-type-options: nosniff</span></div><div><span style="color: #3d85c6;">x-xss-protection: 1; mode=block</span></div><div><span style="color: #3d85c6;">strict-transport-security: max-age=31536000; preload</span></div><div><span style="color: #3d85c6;"><br /></span></div><div><span style="color: #3d85c6;">{}</span></div></div><p>Finally switch back to the Snyk UI and verify you have imported the Azure repo as shown below</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgKhlroTJftf425o5PpDSHma2OrS1C9fYZNxhh_Q7Q57tfwALg8JhV0vKJbi_GSUZYjjeE8b9um3XKuFfuKNKUkddP3HTl7enB8g0dBiF5aAc1i9jFdBswmbdoPx__O7_ebULat8q9Ehgin/s2796/Screen+Shot+2021-07-02+at+11.41.15+am.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="918" data-original-width="2796" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgKhlroTJftf425o5PpDSHma2OrS1C9fYZNxhh_Q7Q57tfwALg8JhV0vKJbi_GSUZYjjeE8b9um3XKuFfuKNKUkddP3HTl7enB8g0dBiF5aAc1i9jFdBswmbdoPx__O7_ebULat8q9Ehgin/s320/Screen+Shot+2021-07-02+at+11.41.15+am.png" width="320" /></a></div><p><br /></p><h2 style="text-align: left;">More Information</h2><div style="text-align: left;">Import Projects API</div><div style="text-align: left;"><a href="https://snyk.docs.apiary.io/#reference/integrations/import-projects/import">https://snyk.docs.apiary.io/#reference/integrations/import-projects/import</a></div><div style="text-align: left;"><br /></div><p><br /></p><div class="blogger-post-footer">http://feeds.feedburner.com/TheBlasFromPas</div>Pas Apicellahttp://www.blogger.com/profile/09389663166398991762noreply@blogger.com0tag:blogger.com,1999:blog-6527688743456205256.post-70992317418418130032021-06-14T15:20:00.002+10:002021-06-14T15:20:41.994+10:00Basic Pipeline using Snyk Container, OCI Images, Azure DevOps all part of Cloud Native Application Security<p><span style="font-family: Arial; font-size: 11pt; white-space: pre-wrap;"><a href="https://snyk.io/product/container-vulnerability-management/" target="_blank">Snyk Container</a> will find vulnerabilities in containers and Kubernetes workloads throughout the SDLC by scanning any compliant OCI image which includes those created by <a href="https://buildpacks.io/" target="_blank">Cloud Native Buildpacks</a> or other build tools that create OCI images.</span></p><p><span style="font-family: Arial;"><span style="font-size: 14.6667px; white-space: pre-wrap;">So what could an Azure DevOps Pipeline look like that incorporates the following using Snyk?</span></span></p><h4 style="text-align: left;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;">Running a Snyk Scan against the project </span><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;">repository</span></h4><div style="text-align: left;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;">Here we run a "<b>snyk test</b>" from the root folder of the repository itself and that report is then</span></div><div style="text-align: left;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;"><br /><br /></span><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiQx4oavAysilE5hj7M_jAYCtIvLVbz5GhkMeIRfUFtu-u7kdT7GncjMyzmAUAg35BzhvpawPKXF1ChRS5rU2zW0Xk5VBB8rp3zihvLJRAsv1kacKcmlR6nhRVgPx_VUwRTpXxE7VzQjGnX/s2048/Screen+Shot+2021-06-14+at+3.05.24+pm.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1205" data-original-width="2048" height="376" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiQx4oavAysilE5hj7M_jAYCtIvLVbz5GhkMeIRfUFtu-u7kdT7GncjMyzmAUAg35BzhvpawPKXF1ChRS5rU2zW0Xk5VBB8rp3zihvLJRAsv1kacKcmlR6nhRVgPx_VUwRTpXxE7VzQjGnX/w640-h376/Screen+Shot+2021-06-14+at+3.05.24+pm.png" width="640" /></a></div><br /></div><h4 style="text-align: left;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;">Building your Artifact</span></h4><div style="text-align: left;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;">Here we use a Maven task which packages the application Artifact as a JAR file ready to run</span></div><div style="text-align: left;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;"><br /></span></div><div style="text-align: left;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj_JL9UR35NnEIM_enVrmjsJkw9UsCPtYbMMKBx5_WegiWO3uj5S7zKYq9kiwivWNJpZdgK87U4mj27dXbUNv9wF_lQY2ZxMsvcI_qUpQ1CCyFb90rV8y9ZYywlKDhLefXOFM_-QXXdZ5WN/" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="1198" data-original-width="2048" height="374" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj_JL9UR35NnEIM_enVrmjsJkw9UsCPtYbMMKBx5_WegiWO3uj5S7zKYq9kiwivWNJpZdgK87U4mj27dXbUNv9wF_lQY2ZxMsvcI_qUpQ1CCyFb90rV8y9ZYywlKDhLefXOFM_-QXXdZ5WN/w640-h374/Screen+Shot+2021-06-14+at+3.07.13+pm.png" width="640" /></a></div><br /><br /></span></div><div style="text-align: left;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;"><br /></span></div><h4 style="text-align: left;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;">Creating an OCI compliant container image from the Artifact itself</span></h4><div style="text-align: left;"><span style="font-family: Arial;"><span style="font-size: 14.6667px; white-space: pre-wrap;">There are various ways to create a OCI compliant image but by the far the simplest is using Cloud Native Buildpacks and for this we use the pack CLI which in turns using the Java Buildpack from our JAR file directly avoid a compilation step from the source code given we already did that on the step above.</span></span></div><div style="text-align: left;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;"><br /></span></div><div style="text-align: left;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhd74jWyi-UMBWHrqB3ji8YHUnNDcMTMjiR5xt6r30CkSXNeoTchpzBTS9_LgBu4DkHoH2r1Dy54VpFE4vqyztUhdFmyHZS-QQbH9H18qPd2FN2mpEasZIf4-EVOZKH9OHYBLfG7RUrHF1-/" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="202" data-original-width="2582" height="50" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhd74jWyi-UMBWHrqB3ji8YHUnNDcMTMjiR5xt6r30CkSXNeoTchpzBTS9_LgBu4DkHoH2r1Dy54VpFE4vqyztUhdFmyHZS-QQbH9H18qPd2FN2mpEasZIf4-EVOZKH9OHYBLfG7RUrHF1-/w640-h50/Screen+Shot+2021-06-14+at+3.10.31+pm.png" width="640" /></a></div><br /><br /></span></div><h4 style="text-align: left;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;">Running a Snyk Scan against the container image directly on the Container Registry</span></h4><div style="text-align: left;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;">With our container image now in our Container Registry we can use "</span><b style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;">snyk container</b><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;">" to check for issues directly from the registry and also check for application security issues from the open source </span><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;">dependancies</span><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;"> as well.</span></div><div style="text-align: left;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;"><br /></span></div><div style="text-align: left;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhjEIpi0coaaBlFods3tgRK971c03C0fES-XB1fm50HbXDnHskAHBa0jE3NwEcdX0YFFkV0ixQt8KjW8wtw9Xhc9fGZQQTcXf3Hp1WIGCo1L10qDLGMaPwj7aVEQej9gim61GbKiqo7jo4B/" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="1246" data-original-width="2048" height="390" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhjEIpi0coaaBlFods3tgRK971c03C0fES-XB1fm50HbXDnHskAHBa0jE3NwEcdX0YFFkV0ixQt8KjW8wtw9Xhc9fGZQQTcXf3Hp1WIGCo1L10qDLGMaPwj7aVEQej9gim61GbKiqo7jo4B/w640-h390/Screen+Shot+2021-06-14+at+3.11.40+pm.png" width="640" /></a></div><br /><br /></span></div><div style="text-align: left;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;"><br /></span></div><div style="text-align: left;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;"><br /></span></div><div style="text-align: left;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;">The finished Pipeline ...</span></div><div style="text-align: left;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;"><br /></span></div><div style="text-align: left;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;"><br /></span></div><div style="text-align: left;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhTLxn5lEe4ev0V_zgBYnpyamJF2hdUZPl4nUyh2VDPGM1GGsvpW2ZLrkE4-c4lPlobojfv8jITyQOp9NEO6XOPuDbStoxtQx2tYoTYMjt57i07du_zBLQUjIkQ7G5_jpoitAAnCs81ro5H/" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="1196" data-original-width="2048" height="374" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhTLxn5lEe4ev0V_zgBYnpyamJF2hdUZPl4nUyh2VDPGM1GGsvpW2ZLrkE4-c4lPlobojfv8jITyQOp9NEO6XOPuDbStoxtQx2tYoTYMjt57i07du_zBLQUjIkQ7G5_jpoitAAnCs81ro5H/w640-h374/Screen+Shot+2021-06-14+at+2.54.09+pm.png" width="640" /></a></div><br /><br /></span></div><div style="text-align: left;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;"><b><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhfr6dJxSW-24wLTVvbQSLZ1Q_59sizHCUtStDSf7BSKoBh61XsT8xcU4-QtKCCc7C6NGGbEijx3tU6cBSegN4kCDjcrns6JruD1P4NzMhYs3CQ9kXvCYqUbNb-Gqbc9c8uEwjhlFhBpFDc/" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="1167" data-original-width="2048" height="364" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhfr6dJxSW-24wLTVvbQSLZ1Q_59sizHCUtStDSf7BSKoBh61XsT8xcU4-QtKCCc7C6NGGbEijx3tU6cBSegN4kCDjcrns6JruD1P4NzMhYs3CQ9kXvCYqUbNb-Gqbc9c8uEwjhlFhBpFDc/w640-h364/Screen+Shot+2021-06-14+at+3.18.14+pm.png" width="640" /></a></div><br /><br /></b></span></div><div style="text-align: left;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;"><b><br /></b></span></div><div style="text-align: left;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;"><b>azure-pipeline.yml Pipeline used in Azure DevOps </b></span></div><div style="text-align: left;"><span style="font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;"><br /></span></div><div style="text-align: left;"><div style="background-color: #fffffe; font-family: Menlo, Monaco, "Courier New", monospace; font-size: 14px; line-height: 21px; white-space: pre;"><div><span style="color: #007200;"># Starter pipeline</span></div><div><span style="color: #007200;"># Start with a minimal pipeline that you can customize to build and deploy your code.</span></div><div><span style="color: #007200;"># Add steps that build, run tests, deploy, and more:</span></div><div><span style="color: #007200;"># https://aka.ms/yaml</span></div><br /><div><span style="color: teal;">trigger</span>:</div><div>- <span style="color: #0451a5;">master</span></div><br /><div><span style="color: teal;">pool</span>:</div><div> <span style="color: teal;">vmImage</span>: <span style="color: #0451a5;">ubuntu-latest</span></div><br /><div><span style="color: teal;">steps</span>:</div><br /><div>- <span style="color: teal;">task</span>: <span style="color: #0451a5;">Maven@3</span></div><div> <span style="color: teal;">inputs</span>:</div><div> <span style="color: teal;">mavenPomFile</span>: <span style="color: #0451a5;">'pom.xml'</span></div><div> <span style="color: teal;">mavenOptions</span>: <span style="color: #0451a5;">'-Xmx3072m'</span></div><div> <span style="color: teal;">javaHomeOption</span>: <span style="color: #0451a5;">'JDKVersion'</span></div><div> <span style="color: teal;">jdkVersionOption</span>: <span style="color: #0451a5;">'1.11'</span></div><div> <span style="color: teal;">jdkArchitectureOption</span>: <span style="color: #0451a5;">'x64'</span></div><div> <span style="color: teal;">publishJUnitResults</span>: <span style="color: blue;">true</span></div><div> <span style="color: teal;">options</span>: <span style="color: #0451a5;">"-Dskiptests -Dsnyk.skip"</span></div><div> <span style="color: teal;">goals</span>: <span style="color: #0451a5;">'package'</span></div><div> <span style="color: teal;">displayName</span>: <span style="color: #0451a5;">"Build artifact JAR"</span></div><div> </div><div>- <span style="color: teal;">task</span>: <span style="color: #0451a5;">SnykSecurityScan@0</span></div><div> <span style="color: teal;">inputs</span>:</div><div> <span style="color: teal;">serviceConnectionEndpoint</span>: <span style="color: #0451a5;">'snyk-token'</span></div><div> <span style="color: teal;">testType</span>: <span style="color: #0451a5;">'app'</span></div><div> <span style="color: teal;">monitorOnBuild</span>: <span style="color: blue;">false</span></div><div> <span style="color: teal;">failOnIssues</span>: <span style="color: blue;">false</span></div><div> <span style="color: teal;">displayName</span>: <span style="color: #0451a5;">"snyk test from source"</span></div><div>- <span style="color: teal;">task</span>: <span style="color: #0451a5;">Docker@2</span></div><div> <span style="color: teal;">inputs</span>:</div><div> <span style="color: teal;">containerRegistry</span>: <span style="color: #0451a5;">'docker-pasapples-connection'</span></div><div> <span style="color: teal;">command</span>: <span style="color: #0451a5;">'login'</span></div><div> <span style="color: teal;">displayName</span>: <span style="color: #0451a5;">"Login to DockerHub"</span></div><br /><div>- <span style="color: teal;">script</span>: |</div><div><span style="color: #0451a5;"> curl -sSL "https://github.com/buildpacks/pack/releases/download/v0.19.0/pack-v0.19.0-linux.tgz" | tar -C ./ --no-same-owner -xzv pack</span></div><div><span style="color: #0451a5;"> ./pack build pasapples/springbootemployee:cnb-paketo-base --builder paketobuildpacks/builder:base --publish --path ./target/springbootemployee-0.0.1-SNAPSHOT.jar</span></div><div> <span style="color: teal;">displayName</span>: <span style="color: #0451a5;">'Build Container with Pack'</span></div><br /><div>- <span style="color: teal;">task</span>: <span style="color: #0451a5;">SnykSecurityScan@0</span></div><div> <span style="color: teal;">inputs</span>:</div><div> <span style="color: teal;">serviceConnectionEndpoint</span>: <span style="color: #0451a5;">'snyk-token'</span></div><div> <span style="color: teal;">testType</span>: <span style="color: #0451a5;">'container'</span></div><div> <span style="color: teal;">dockerImageName</span>: <span style="color: #0451a5;">'pasapples/springbootemployee:cnb-paketo-base'</span></div><div> <span style="color: teal;">severityThreshold</span>: <span style="color: #0451a5;">'low'</span></div><div> <span style="color: teal;">monitorOnBuild</span>: <span style="color: blue;">false</span></div><div> <span style="color: teal;">failOnIssues</span>: <span style="color: blue;">false</span></div><div> <span style="color: teal;">additionalArguments</span>: <span style="color: #0451a5;">"--app-vulns"</span></div><div> <span style="color: teal;">displayName</span>: <span style="color: #0451a5;">"snyk container scan from image"</span> </div><br /></div></div><h4 style="text-align: left;"><span style="font-family: Arial; font-size: 11pt; white-space: pre-wrap;"><br /></span></h4><h2 style="text-align: left;"><span style="font-family: Arial; font-size: 11pt; white-space: pre-wrap;">More Information</span></h2><p dir="ltr" id="docs-internal-guid-ac01cef4-7fff-6902-f2da-cc247938f2ff" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">So, for Container and Kubernetes security, designed to help developers find and fix vulnerabilities in cloud native applications, click the links below to learn more and get started today.</span></p><p><a href="https://snyk.io/product/container-vulnerability-management/" style="text-decoration-line: none;"><span style="color: #1155cc; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">Snyk Container</span></a></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><a href="https://snyk.io" style="text-decoration-line: none;"><span style="color: #1155cc; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">Snyk Platform</span></a></p><div class="blogger-post-footer">http://feeds.feedburner.com/TheBlasFromPas</div>Pas Apicellahttp://www.blogger.com/profile/09389663166398991762noreply@blogger.com0tag:blogger.com,1999:blog-6527688743456205256.post-51859141128752935652021-06-03T15:16:00.003+10:002021-06-03T15:16:41.958+10:00Installing Snyk Controller into a k3d kubernetes cluster to enable runtime container scanning with the Snyk Platform<p>Snyk integrates with Kubernetes, enabling you to import and test your running workloads and identify vulnerabilities in their associated images and configurations that might make those workloads less secure. Once imported, Snyk continues to monitor those workloads, identifying additional security issues as new images are deployed and the workload configuration changes</p><p>In the example below we show you how easy it is to integrate the Snyk Platform with any K8s distribution in this case k3d running on my laptop.</p><h2 style="text-align: left;">Steps </h2><p>1. Install k3d using the instructions from the link below.</p><p><a href="https://k3d.io/">https://k3d.io/</a></p><p>2. Create a cluster as shown below.</p><div style="text-align: left;"><div><span style="color: #3d85c6;">pasapicella@192-168-1-113:~/snyk/demos/kubernetes/k3d$ k3d cluster create snyk-k3d --servers 1 --agents 2</span></div><div><span style="color: #3d85c6;">INFO[0000] Prep: Network</span></div><div><span style="color: #3d85c6;">INFO[0003] Created network 'k3d-snyk-k3d'</span></div><div><span style="color: #3d85c6;">INFO[0003] Created volume 'k3d-snyk-k3d-images'</span></div><div><span style="color: #3d85c6;">INFO[0004] Creating node 'k3d-snyk-k3d-server-0'</span></div><div><span style="color: #3d85c6;">INFO[0005] Creating node 'k3d-snyk-k3d-agent-0'</span></div><div><span style="color: #3d85c6;">INFO[0005] Creating node 'k3d-snyk-k3d-agent-1'</span></div><div><span style="color: #3d85c6;">INFO[0005] Creating LoadBalancer 'k3d-snyk-k3d-serverlb'</span></div><div><span style="color: #3d85c6;">INFO[0005] Starting cluster 'snyk-k3d'</span></div><div><span style="color: #3d85c6;">INFO[0005] Starting servers...</span></div><div><span style="color: #3d85c6;">INFO[0005] Starting Node 'k3d-snyk-k3d-server-0'</span></div><div><span style="color: #3d85c6;">INFO[0012] Starting agents...</span></div><div><span style="color: #3d85c6;">INFO[0012] Starting Node 'k3d-snyk-k3d-agent-0'</span></div><div><span style="color: #3d85c6;">INFO[0023] Starting Node 'k3d-snyk-k3d-agent-1'</span></div><div><span style="color: #3d85c6;">INFO[0031] Starting helpers...</span></div><div><span style="color: #3d85c6;">INFO[0031] Starting Node 'k3d-snyk-k3d-serverlb'</span></div><div><span style="color: #3d85c6;">INFO[0033] (Optional) Trying to get IP of the docker host and inject it into the cluster as 'host.k3d.internal' for easy access</span></div><div><span style="color: #3d85c6;">INFO[0036] Successfully added host record to /etc/hosts in 4/4 nodes and to the CoreDNS ConfigMap</span></div><div><span style="color: #3d85c6;">INFO[0036] Cluster 'snyk-k3d' created successfully!</span></div><div><span style="color: #3d85c6;">INFO[0036] --kubeconfig-update-default=false --> sets --kubeconfig-switch-context=false</span></div><div><span style="color: #3d85c6;">INFO[0036] You can now use it like this:</span></div><div><span style="color: #3d85c6;">kubectl config use-context k3d-snyk-k3d</span></div><div><span style="color: #3d85c6;">kubectl cluster-info</span></div></div><p>3. View the Kubernetes nodes.</p><div style="text-align: left;"><div><span style="color: #3d85c6;">$ kubectl get nodes</span></div><div><span style="color: #3d85c6;">NAME STATUS ROLES AGE VERSION</span></div><div><span style="color: #3d85c6;">k3d-snyk-k3d-server-0 Ready control-plane,master 21h v1.20.5+k3s1</span></div><div><span style="color: #3d85c6;">k3d-snyk-k3d-agent-0 Ready <none> 21h v1.20.5+k3s1</span></div><div><span style="color: #3d85c6;">k3d-snyk-k3d-agent-1 Ready <none> 21h v1.20.5+k3s1</span></div></div><p>4. Run the following command in order to add the Snyk Charts repository to Helm.</p><div style="text-align: left;"><span style="color: #3d85c6;">$ helm repo add snyk-charts https://snyk.github.io/kubernetes-monitor/<br />"snyk-charts" already exists with the same configuration, skipping</span></div><p>5. Once the repository is added, create a unique namespace for the Snyk controller:</p><p><span style="color: #3d85c6;">$ kubectl create namespace snyk-monitor</span></p><p>6. Now, log in to your Snyk account and navigate to Integrations. Search for and click Kubernetes. Click Connect from the page that loads, copy the Integration ID. The Snyk Integration ID is a UUID, similar to this format: abcd1234-abcd-1234-abcd-1234abcd1234. Save it for use from your Kubernetes environment in the next step</p><p>Instructions link : <a href="https://support.snyk.io/hc/en-us/articles/360006368657-Viewing-your-Kubernetes-integration-settings">https://support.snyk.io/hc/en-us/articles/360006368657-Viewing-your-Kubernetes-integration-settings</a></p><p>7. Snyk monitor runs by using your Snyk Integration ID, and using a dockercfg file. If you are not using any private registries which we are not in this demo, create a Kubernetes secret called snyk-monitor containing the Snyk Integration ID from the previous step and run the following command:</p><div style="text-align: left;"><span style="color: #3d85c6;">$ kubectl create secret generic snyk-monitor -n snyk-monitor \<br /> --from-literal=dockercfg.json={} \<br /> --from-literal=integrationId=INTEGRATION_TOKEN_FROM_STEP_6</span></div><div style="text-align: left;"><span style="color: #3d85c6;">secret/snyk-monitor created</span></div><p>8. Install the Snyk Helm chart as follows:</p><div style="text-align: left;"><span style="color: #3d85c6;">$ helm upgrade --install snyk-monitor snyk-charts/snyk-monitor \<br /> --namespace snyk-monitor \<br /> --set clusterName="k3d Dev cluster"<br />Release "snyk-monitor" does not exist. Installing it now.<br />NAME: snyk-monitor<br />LAST DEPLOYED: Wed Jun 2 17:47:13 2021<br />NAMESPACE: snyk-monitor<br />STATUS: deployed<br />REVISION: 1<br />TEST SUITE: None</span></div><p>9. Verify the Snyk Controller is running using either </p><div style="text-align: left;"><span style="color: #3d85c6;">$ kubectl get pods -n snyk-monitor<br />NAME READY STATUS RESTARTS AGE<br />snyk-monitor-64c94685b-fwpvx 1/1 Running 3 21h</span></div><div style="text-align: left;"><br /></div><div style="text-align: left;">10. At this point we can create some workloads as follows let's just add a single POD to the cluster for a basic Spring Boot application.</div><div style="text-align: left;"><br /></div><div style="text-align: left;"><span style="color: #3d85c6;">$ kubectl run springboot-app --image=pasapples/spring-boot-jib --port=8080<br />pod/springboot-app created</span></div><p>11. Head back to the Snyk Dashboard and click on your Kubernetes Integration Tile and you should see a list of applicable workloads to monitor in our case we just have the single app called "<b>springboot-app</b>".</p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi9Sgv7Jedj20Ug2SvOb8rhL1I5ZAsrkNycYfI0WObZ9io7BccVhxjqoF5j0UF9e-XxNTosSYAiChXtqzPaJ-TKS_rF1V00tN-O58vm-PHMMbPovB0u7SKWlZUJXpW3b8MGfnGt0mYxfikR/" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="764" data-original-width="3004" height="162" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi9Sgv7Jedj20Ug2SvOb8rhL1I5ZAsrkNycYfI0WObZ9io7BccVhxjqoF5j0UF9e-XxNTosSYAiChXtqzPaJ-TKS_rF1V00tN-O58vm-PHMMbPovB0u7SKWlZUJXpW3b8MGfnGt0mYxfikR/w640-h162/Screen+Shot+2021-06-03+at+3.11.33+pm.png" width="640" /></a></div><br /><br /><p></p><p>12. Add the selected workload and your done!!!</p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiHB7dIaMbdvFxVlmNtJUqsYEvStnHijaw1PUqRD2kZp1oc7JEPgngh2sBswmDt8AgVbBpCwcDRG_fIDNTVSWmEX6fNrUbhq0x-3-_TBquGiJBMztp_H3eWyTUrdBGWUelnIP259bOuuHXk/" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="1078" data-original-width="2917" height="236" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiHB7dIaMbdvFxVlmNtJUqsYEvStnHijaw1PUqRD2kZp1oc7JEPgngh2sBswmDt8AgVbBpCwcDRG_fIDNTVSWmEX6fNrUbhq0x-3-_TBquGiJBMztp_H3eWyTUrdBGWUelnIP259bOuuHXk/w640-h236/Screen+Shot+2021-06-03+at+3.13.13+pm.png" width="640" /></a></div><br /><br /><p></p><h2 style="text-align: left;">More Information</h2><div style="text-align: left;">Kubernetes integration overview<br /><a href="https://support.snyk.io/hc/en-us/articles/360003916138-Kubernetes-integration-overview">https://support.snyk.io/hc/en-us/articles/360003916138-Kubernetes-integration-overview</a></div><div style="text-align: left;"><br /></div><div style="text-align: left;">Install the Snyk controller with Helm</div><div style="text-align: left;"><a href="https://support.snyk.io/hc/en-us/articles/360003916158#UUID-753328ea-3d73-0eeb-4301-c22522273797">https://support.snyk.io/hc/en-us/articles/360003916158#UUID-753328ea-3d73-0eeb-4301-c22522273797</a></div><div style="text-align: left;"><br /></div><div class="blogger-post-footer">http://feeds.feedburner.com/TheBlasFromPas</div>Pas Apicellahttp://www.blogger.com/profile/09389663166398991762noreply@blogger.com0tag:blogger.com,1999:blog-6527688743456205256.post-85274389573760398752021-05-27T22:35:00.001+10:002021-05-27T22:37:12.958+10:00Cloud Native Buildpacks meets Snyk Container<p><a href="http://buildpacks.io" target="_blank"><em>Cloud Native Buildpacks</em></a> transform your application source code into images that can run on any cloud and avoid ever having a write Dockerfile again after all why would you even care about a Dockerfile?</p><p>So in today's post we are going to take a look at how <a href="http://snyk.io" target="_blank">Snyk App</a> with Snyk Container can scan those Cloud Native Buildpack OCI images for you. </p><p>Snyk Container equips developers to quickly fix container issues. Use the following to find out more about Snyk Container - <a href="https://snyk.io/product/container-vulnerability-management/">https://snyk.io/product/container-vulnerability-management/</a></p><p>You may not always have access to the original source code that runs in your containers, but vulnerabilities in your code dependencies are still important. Snyk can detect and monitor open source dependencies for popular languages as part of the container scan which is important because Cloud Native Buildpacks are building container images from source code using popular programming languages.</p><h2 style="text-align: left;">Demo</h2><p>1. First let's clone spring petclinic app and create our artifact for deployment.</p><div style="text-align: left;"><span style="color: #3d85c6;">$ git clone https://github.com/spring-projects/spring-petclinic<br />$ cd spring-petclinic</span></div><div style="text-align: left;"><span style="color: #3d85c6;">$ /mvnw package </span></div><div style="text-align: left;"><br /></div><div style="text-align: left;">2. Install pack so we can build some OCI images directly into our registries from a simple CLI. Instructions as follows</div><div style="text-align: left;"><br /></div><div style="text-align: left;"><a href="https://buildpacks.io/docs/tools/pack/">https://buildpacks.io/docs/tools/pack/</a></div><div style="text-align: left;"><br /></div><div style="text-align: left;">3. Test kpack is installed by listing the suggested builders</div><div style="text-align: left;"><br /></div><div style="text-align: left;"><span style="color: #3d85c6;">$ pack --version<br />0.18.1+git-b5c1a96.build-2373</span></div><div style="text-align: left;"><span style="color: #3d85c6;"><div><br /></div><div>$ pack builder suggest</div><div>Suggested builders:</div><div><span style="white-space: pre;"> </span>Google: gcr.io/buildpacks/builder:v1 </div><div><span style="white-space: pre;"> </span>Heroku: heroku/buildpacks:18 </div><div><span style="white-space: pre;"> </span>Heroku: heroku/buildpacks:20 </div><div><span style="white-space: pre;"> </span>Paketo Buildpacks: paketobuildpacks/builder:base </div><div><span style="white-space: pre;"> </span>Paketo Buildpacks: paketobuildpacks/builder:full </div><div><span style="white-space: pre;"> </span>Paketo Buildpacks: paketobuildpacks/builder:tiny</div></span></div><p>At this point we are ready to go so let's create 3 OCI images and push them to 3 separate regitries as follows</p><div style="text-align: left;"><ul style="text-align: left;"><li>Dockerhub</li><li>Amazon Elastic Container Registry (ECR)</li><li>Google Cloud Registry (GCR)</li></ul></div><div style="text-align: left;"><br /></div><div style="text-align: left;">Note: Your local docker desktop needs to be able to establish connections to the 3 registries above</div><div style="text-align: left;"><br /></div><div style="text-align: left;">4. Build an OCI image of petclinic to Dockerhub.</div><div style="text-align: left;"><br /></div><div style="text-align: left;"><div><span style="color: #3d85c6;">$ pack build pasapples/petclinic:latest --builder paketobuildpacks/builder:base --publish --path ./spring-petclinic/target/spring-petclinic-2.4.5.jar</span></div><div><span style="color: #3d85c6;">base: Pulling from paketobuildpacks/builder</span></div><div><span style="color: #3d85c6;">Digest: sha256:a6f81cb029d4d3272981c12dad7212a5063ec0076e2438b7b5bb702f2e1fd11a</span></div><div><span style="color: #3d85c6;">Status: Image is up to date for paketobuildpacks/builder:base</span></div><div><span style="color: #3d85c6;">===> DETECTING</span></div><div><span style="color: #3d85c6;">5 of 18 buildpacks participating</span></div><div><span style="color: #3d85c6;">paketo-buildpacks/ca-certificates 2.2.0</span></div><div><span style="color: #3d85c6;">paketo-buildpacks/bellsoft-liberica 8.0.0</span></div><div><span style="color: #3d85c6;"><br /></span></div><div><span style="color: #3d85c6;">...</span></div><div><span style="color: #3d85c6;"><br /></span></div><div><div><span style="color: #3d85c6;">Adding label 'org.opencontainers.image.version'</span></div><div><span style="color: #3d85c6;">Adding label 'org.springframework.boot.version'</span></div><div><span style="color: #3d85c6;">Setting default process type 'web'</span></div><div><span style="color: #3d85c6;">Saving pasapples/petclinic:latest...</span></div><div><span style="color: #3d85c6;">*** Images (sha256:d730612833826cd9e39a7241c1fba411eacd9b5f771915b00af4b3b499838bd5):</span></div><div><span style="color: #3d85c6;"> pasapples/petclinic:latest</span></div><div><span style="color: #3d85c6;">Successfully built image pasapples/petclinic:latest</span></div></div><div><br /></div></div><div style="text-align: left;"><br /></div><div style="text-align: left;">5. Build an OCI image of petclinic to ECR.</div><div style="text-align: left;"><br /></div><div style="text-align: left;"><div><span style="color: #3d85c6;">$ pack build 300326902600.dkr.ecr.us-east-1.amazonaws.com/petclinic:latest --builder paketobuildpacks/builder:base --publish --path ./spring-petclinic/target/spring-petclinic-2.4.5.jar</span></div><div><span style="color: #3d85c6;">base: Pulling from paketobuildpacks/builder</span></div><div><span style="color: #3d85c6;">Digest: sha256:a6f81cb029d4d3272981c12dad7212a5063ec0076e2438b7b5bb702f2e1fd11a</span></div><div><span style="color: #3d85c6;">Status: Image is up to date for paketobuildpacks/builder:base</span></div><div><span style="color: #3d85c6;">===> DETECTING</span></div><div><span style="color: #3d85c6;">5 of 18 buildpacks participating</span></div><div><span style="color: #3d85c6;">paketo-buildpacks/ca-certificates 2.2.0</span></div><div><span style="color: #3d85c6;">paketo-buildpacks/bellsoft-liberica 8.0.0</span></div><div><span style="color: #3d85c6;"><br /></span></div><div><span style="color: #3d85c6;">...</span></div><div><span style="color: #3d85c6;"><br /></span></div><div><div><span style="color: #3d85c6;">Adding label 'org.opencontainers.image.version'</span></div><div><span style="color: #3d85c6;">Adding label 'org.springframework.boot.version'</span></div><div><span style="color: #3d85c6;">Setting default process type 'web'</span></div><div><span style="color: #3d85c6;">Saving 300326902600.dkr.ecr.us-east-1.amazonaws.com/petclinic:latest...</span></div><div><span style="color: #3d85c6;">*** Images (sha256:d730612833826cd9e39a7241c1fba411eacd9b5f771915b00af4b3b499838bd5):</span></div><div><span style="color: #3d85c6;"> 300326902600.dkr.ecr.us-east-1.amazonaws.com/petclinic:latest</span></div><div><span style="color: #3d85c6;">Successfully built image 300326902600.dkr.ecr.us-east-1.amazonaws.com/petclinic:latest</span></div></div><div><br /></div><div><br /></div></div><div style="text-align: left;">6. Build an OCI image of petclinic to GCR.</div><div style="text-align: left;"><br /></div><div style="text-align: left;"><div><span style="color: #3d85c6;">$ pack build us.gcr.io/snyk-cx-se-demo/petclinic-google:latest --builder paketobuildpacks/builder:base --publish --path ./spring-petclinic/target/spring-petclinic-2.4.5.jar</span></div><div><span style="color: #3d85c6;">base: Pulling from paketobuildpacks/builder</span></div><div><span style="color: #3d85c6;">Digest: sha256:a6f81cb029d4d3272981c12dad7212a5063ec0076e2438b7b5bb702f2e1fd11a</span></div><div><span style="color: #3d85c6;">Status: Image is up to date for paketobuildpacks/builder:base</span></div><div><span style="color: #3d85c6;">===> DETECTING</span></div><div><span style="color: #3d85c6;">5 of 18 buildpacks participating</span></div><div><span style="color: #3d85c6;">paketo-buildpacks/ca-certificates 2.2.0</span></div><div><span style="color: #3d85c6;">paketo-buildpacks/bellsoft-liberica 8.0.0</span></div><div><span style="color: #3d85c6;"><br /></span></div><div><span style="color: #3d85c6;">...</span></div><div><span style="color: #3d85c6;"><br /></span></div><div><div><span style="color: #3d85c6;">Adding label 'org.opencontainers.image.version'</span></div><div><span style="color: #3d85c6;">Adding label 'org.springframework.boot.version'</span></div><div><span style="color: #3d85c6;">Setting default process type 'web'</span></div><div><span style="color: #3d85c6;">Saving us.gcr.io/snyk-cx-se-demo/petclinic-google:latest...</span></div><div><span style="color: #3d85c6;">*** Images (sha256:d730612833826cd9e39a7241c1fba411eacd9b5f771915b00af4b3b499838bd5):</span></div><div><span style="color: #3d85c6;"> us.gcr.io/snyk-cx-se-demo/petclinic-google:latest</span></div><div><span style="color: #3d85c6;">Successfully built image us.gcr.io/snyk-cx-se-demo/petclinic-google:latest</span></div></div><div><br /></div><div><br /></div></div><div style="text-align: left;">Three OCI compliant images of petclinic in our 3 registries done!!!</div><div style="text-align: left;"><br /></div><div style="text-align: left;">So now it's over to Snyk.</div><div style="text-align: left;"><br /></div><div style="text-align: left;">7. Login Snyk App - <a href="https://app.snyk.io">https://app.snyk.io</a> </div><p>8. Lets do a scan from the CLI by first authenticating with Snyk App as follows which will direct you to a browser to authenticate once done return to the prompt.</p><p>$ snyk auth </p><p>9. Now you can run a snyk container test to any of the 3 registries we used as follows.</p><p><span style="color: #3d85c6;">$ snyk container test us.gcr.io/snyk-cx-se-demo/petclinic-google:latest</span></p><p><span style="color: #3d85c6;">$ snyk container test 300326902600.dkr.ecr.us-east-1.amazonaws.com/petclinic:latest</span></p><p><span style="color: #3d85c6;">$ snyk container test pasapples/petclinic:latest</span></p><p><b>Demo Output</b></p><div style="text-align: left;"><div><span style="color: #3d85c6;">Testing us.gcr.io/snyk-cx-se-demo/petclinic-google:latest...</span></div><div><span style="color: #3d85c6;"><br /></span></div><div><span style="color: #3d85c6;">✗ Low severity vulnerability found in shadow/passwd</span></div><div><span style="color: #3d85c6;"> Description: Time-of-check Time-of-use (TOCTOU)</span></div><div><span style="color: #3d85c6;"> Info: https://snyk.io/vuln/SNYK-UBUNTU1804-SHADOW-306209</span></div><div><span style="color: #3d85c6;"> Introduced through: shadow/passwd@1:4.5-1ubuntu2, apt@1.6.13, shadow/login@1:4.5-1ubuntu2</span></div><div><span style="color: #3d85c6;"> From: shadow/passwd@1:4.5-1ubuntu2</span></div><div><span style="color: #3d85c6;"> From: apt@1.6.13 > adduser@3.116ubuntu1 > shadow/passwd@1:4.5-1ubuntu2</span></div><div><span style="color: #3d85c6;"> From: shadow/login@1:4.5-1ubuntu2</span></div></div><p><span style="color: #3d85c6;">...</span></p><div style="text-align: left;"><span style="color: #3d85c6;">✗ Medium severity vulnerability found in gcc-8/libstdc++6<br /> Description: Information Exposure<br /> Info: https://snyk.io/vuln/SNYK-UBUNTU1804-GCC8-572149<br /> Introduced through: gcc-8/libstdc++6@8.4.0-1ubuntu1~18.04, apt/libapt-pkg5.0@1.6.13, apt@1.6.13, meta-common-packages@meta<br /> From: gcc-8/libstdc++6@8.4.0-1ubuntu1~18.04<br /> From: apt/libapt-pkg5.0@1.6.13 > gcc-8/libstdc++6@8.4.0-1ubuntu1~18.04<br /> From: apt@1.6.13 > gcc-8/libstdc++6@8.4.0-1ubuntu1~18.04<br /> and 2 more...<br /><br />Organization: pas.apicella-41p<br />Package manager: deb<br />Project name: docker-image|us.gcr.io/snyk-cx-se-demo/petclinic-google<br />Docker image: us.gcr.io/snyk-cx-se-demo/petclinic-google:latest<br />Platform: linux/amd64<br />Licenses: enabled</span></div><p style="text-align: left;"><span style="color: #3d85c6;">Tested 97 dependencies for known issues, found 25 issues.</span></p><p>10. Back to Snyk App we can import all 3 OCI images from all 3 registries once we configure each integration for the registries as shown below</p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhdiUTlWy62T3QB3nsD_Ogw46HLXfFAU9xr2b_gkJ4_Md-F0hrryPCImo4W-CMkZ-7fSwigpWjEEVkH2SvTaiHO0N924hujLWTosXjwMXH67IPSVqM-DfaIr7BzK2u2eHh4Oj0Dn6y6b_Yy/" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="446" data-original-width="1310" height="218" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhdiUTlWy62T3QB3nsD_Ogw46HLXfFAU9xr2b_gkJ4_Md-F0hrryPCImo4W-CMkZ-7fSwigpWjEEVkH2SvTaiHO0N924hujLWTosXjwMXH67IPSVqM-DfaIr7BzK2u2eHh4Oj0Dn6y6b_Yy/w640-h218/Screen+Shot+2021-05-27+at+10.31.59+pm.png" width="640" /></a></div><br /><br /><p></p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg1sJgHGklQ6xAuKPXl-nPsiUZoDxEe3PbVBwybn-XjnA4a8ym1lqaJq2_O-ufdVIRTEbumboUTeapsVVAiUpTGI12lxHIzxnwMwiqmTYqV_Jg-kqed_fcMcAWNe7AAa_kAU_lh9SQXDIw3/" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="866" data-original-width="2774" height="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg1sJgHGklQ6xAuKPXl-nPsiUZoDxEe3PbVBwybn-XjnA4a8ym1lqaJq2_O-ufdVIRTEbumboUTeapsVVAiUpTGI12lxHIzxnwMwiqmTYqV_Jg-kqed_fcMcAWNe7AAa_kAU_lh9SQXDIw3/w640-h200/Screen+Shot+2021-05-27+at+10.24.37+pm.png" width="640" /></a></div><br /><br /><p></p><p>11. And take a look at where the vulnerabilities exist within those open source dependencies used in our petclinic source code as well as base image layer vulnerabilities</p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgcWSM2i_yQsp3OIF8O_vfZeKkx7klJgLbOHuP_cu8URiIMf_3CExbkp7HMhLZEWaTPBAAujwg9wFt9SES5jhQNE8Pg_Duy0RRZqRh6UUoWNaSXoi8GLWXh8BaPQSfDZ866aSi8W8jtpKif/" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="1617" data-original-width="2048" height="505" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgcWSM2i_yQsp3OIF8O_vfZeKkx7klJgLbOHuP_cu8URiIMf_3CExbkp7HMhLZEWaTPBAAujwg9wFt9SES5jhQNE8Pg_Duy0RRZqRh6UUoWNaSXoi8GLWXh8BaPQSfDZ866aSi8W8jtpKif/w640-h505/Screen+Shot+2021-05-27+at+10.25.59+pm.png" width="640" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgeSAnYx-DLSlhD3XJrwhm4iMsQHO928COSyP05PdiZyKMF5zfBXAhwH2F8cF75jkHS5hMqjO_0QwRiV55t7AsZEZXgBODhdKwmRt6uTeeP7MuBHmSpwkNsiQm0zNq_cPnYlssA99UZwF6q/" style="margin-left: 1em; margin-right: 1em;"><img alt="" data-original-height="1570" data-original-width="2048" height="491" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgeSAnYx-DLSlhD3XJrwhm4iMsQHO928COSyP05PdiZyKMF5zfBXAhwH2F8cF75jkHS5hMqjO_0QwRiV55t7AsZEZXgBODhdKwmRt6uTeeP7MuBHmSpwkNsiQm0zNq_cPnYlssA99UZwF6q/w640-h491/Screen+Shot+2021-05-27+at+10.28.57+pm.png" width="640" /></a></div><br /><br /></div><br />Give <a href="https://app.snyk.io" target="_blank">Snyk App</a> a go yourself and start scanning those OCI container images built using Cloud Native Buildpacks now!!!<p></p><h2 style="text-align: left;">More Information</h2><div>Main Snyk Web Page</div><div><a href="http://snyk.io">http://snyk.io</a></div><div><br /></div><div>Snyk Container</div><div><a href="https://snyk.io/product/container-vulnerability-management/">https://snyk.io/product/container-vulnerability-management/</a></div><div class="blogger-post-footer">http://feeds.feedburner.com/TheBlasFromPas</div>Pas Apicellahttp://www.blogger.com/profile/09389663166398991762noreply@blogger.com0tag:blogger.com,1999:blog-6527688743456205256.post-81896510005045738042021-02-25T20:02:00.003+11:002021-02-25T20:02:58.155+11:00Elastic Cloud with Observability 7.11 Using Rancher k3d for K8s <p>Started working locally with rancher's <a href="https://k3d.io/" target="_blank">k3d</a> to spin up K8s clusters so I thought why not use <a href="https://cloud.elastic.co/" target="_blank">Elastic Cloud </a>with Elastic Observability to monitor this local K8s cluster and even run container workloads that use Elastic APM to monitor the application through tracing.</p><div style="text-align: left;">A successful Kubernetes monitoring solution has a few requirements:<br /><ul style="text-align: left;"><li>Monitors all layers of your technology stack, including:</li><ul><li>The host systems where Kubernetes is running.</li><li>Kubernetes core components, nodes, pods, and containers running within the cluster.</li><li>All of the applications and services running in Kubernetes containers.</li></ul><li>Automatically detects and monitors services as they appear dynamically.</li><li>Provides a way to correlate related data so that you can group and explore related metrics, logs, and other observability data.</li></ul></div><p>Some basic steps to get this running as per below. This was tested with Elastic Stack 7.11 using the Elastic Cloud Service</p><h2 style="text-align: left;">Pre Steps</h2><p>1. I have my Elastic stack running using Elastic Cloud as per the screen shot below</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg2YdmNZyYiXtCQuqGLUlP9pvnipvvAbbJYxWyaWF9hJAeu2Xy6HRKoAHmBNl_JbsH7NE_CD3xsvGo5H-_burXA2uXCcWHWn0mupr6zUczTtcjUw74tNmVA6l4Qvsf_prcrMRH7QFC-x8ig/s3049/Screen+Shot+2021-02-25+at+7.13.43+pm.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1032" data-original-width="3049" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg2YdmNZyYiXtCQuqGLUlP9pvnipvvAbbJYxWyaWF9hJAeu2Xy6HRKoAHmBNl_JbsH7NE_CD3xsvGo5H-_burXA2uXCcWHWn0mupr6zUczTtcjUw74tNmVA6l4Qvsf_prcrMRH7QFC-x8ig/s320/Screen+Shot+2021-02-25+at+7.13.43+pm.png" width="320" /></a></div><p>2. I have k3d installed on my Mac as follows</p><div style="text-align: left;"><span style="color: #3d85c6;">$ k3d --version<br />k3d version v4.0.0<br />k3s version latest (default)</span></div><p>3. You will need the <a href="https://v1-18.docs.kubernetes.io/docs/tasks/tools/install-kubectl/" target="_blank">kubectl</a> CLI as well</p><h2 style="text-align: left;">Steps</h2><p>1. First let's start a K8s cluster as follows. I have created a 5 node K8s cluster with 1 master node but you can reduce the number of worker nodes if you don't have the memory to support this</p><p><span style="color: #3d85c6;">$ k3d cluster create elastic-k3d --servers 1 --agents 5</span></p><p>2. Start your k3d cluster as follows if not already started</p><div style="text-align: left;"><span style="color: #3d85c6;">$ k3d cluster start elastic-k3d<br />INFO[0000] Starting cluster 'elastic-k3d'<br />INFO[0000] Starting Node 'k3d-elastic-k3d-agent-4'<br />INFO[0000] Starting Node 'k3d-elastic-k3d-agent-3'<br />INFO[0000] Starting Node 'k3d-elastic-k3d-agent-2'<br />INFO[0001] Starting Node 'k3d-elastic-k3d-agent-1'<br />INFO[0001] Starting Node 'k3d-elastic-k3d-agent-0'<br />INFO[0001] Starting Node 'k3d-elastic-k3d-server-0'<br />INFO[0002] Starting Node 'k3d-elastic-k3d-serverlb'</span></div><p>3. Let's list out our nodes just to verify what you created</p><div style="text-align: left;"><span style="color: #3d85c6;">$ k get nodes<br />NAME STATUS ROLES AGE VERSION<br />k3d-elastic-k3d-server-0 Ready control-plane,master 34h v1.20.2+k3s1<br />k3d-elastic-k3d-agent-4 Ready <none> 34h v1.20.2+k3s1<br />k3d-elastic-k3d-agent-1 Ready <none> 34h v1.20.2+k3s1<br />k3d-elastic-k3d-agent-3 Ready <none> 34h v1.20.2+k3s1<br />k3d-elastic-k3d-agent-2 Ready <none> 34h v1.20.2+k3s1<br />k3d-elastic-k3d-agent-0 Ready <none> 34h v1.20.2+k3s1</span></div><p>Now it's time to "Monitor Kubernetes: Observe the health and performance of your Kubernetes deployments" To do that we could just follow this tutorial as shown below</p><p><a href="https://www.elastic.co/guide/en/observability/7.11/monitor-kubernetes.html">https://www.elastic.co/guide/en/observability/7.11/monitor-kubernetes.html</a></p><div>This tutorial will walk you through how to do the following</div><div><br /></div><div>Deploy filebeat into your k3d cluster</div><div>Deploy metricbeat into your k3d cluster</div><div>Deploy an application which is instrumented using the Elastic APM agent </div><p>4. When installing filebeat and metricbeat make sure you add your Elastic Cloud credentials which can be obtained from the Elastic Cloud deployments page and would have also been provided as a XLS file to download once you create your deployment</p><div style="text-align: left;"><b>filebeat-kubernetes.yaml</b></div><div style="text-align: left;"><br /></div><div style="text-align: left;"><div><span style="color: #3d85c6;"> processors:</span></div><div><span style="color: #3d85c6;"> - add_cloud_metadata:</span></div><div><span style="color: #3d85c6;"> - add_host_metadata:</span></div><div><span style="color: #3d85c6;"><br /></span></div><div><span style="color: #3d85c6;"> cloud.id: pas-K8s:{PASSWORD}<br /> cloud.auth: elastic:{PASSWORD}</span></div></div><div style="text-align: left;"><br /><b>metricbeat-kubernetes.yaml (Notice how I have added a kubernetes_metadata processor)</b></div><div style="text-align: left;"><br /><span style="color: #3d85c6;"> processors:<br /> - add_host_metadata:<br /> - add_kubernetes_metadata:</span></div><div style="text-align: left;"><span style="color: #3d85c6;"><br /> cloud.id: pas-K8s:{PASSWORD}<br /> cloud.auth: elastic:{PASSWORD}</span></div><p style="text-align: left;">5. Once installed filebeat and metricbeat PODS should be running as per the output below</p><div style="text-align: left;"><b>Filebeat</b></div><div style="text-align: left;"><br /></div><div style="text-align: left;"><div><span style="color: #3d85c6;">$ kubectl get pods -n kube-system -l k8s-app=filebeat</span></div><div><span style="color: #3d85c6;">NAME READY STATUS RESTARTS AGE</span></div><div><span style="color: #3d85c6;">filebeat-m8s6s 1/1 Running 2 32h</span></div><div><span style="color: #3d85c6;">filebeat-g86vf 1/1 Running 2 32h</span></div><div><span style="color: #3d85c6;">filebeat-bj548 1/1 Running 2 32h</span></div><div><span style="color: #3d85c6;">filebeat-5cpcn 1/1 Running 2 32h</span></div><div><span style="color: #3d85c6;">filebeat-nwj2h 1/1 Running 2 32h</span></div><div><span style="color: #3d85c6;">filebeat-4hs8j 1/1 Running 2 32h</span></div><div><br /></div><b>Metricbeat</b><br /></div><div style="text-align: left;"><br /></div><div style="text-align: left;"><span style="color: #3d85c6;">$ kubectl get pods -n kube-system -l k8s-app=metricbeat<br />NAME READY STATUS RESTARTS AGE<br />metricbeat-pbfvs 1/1 Running 2 32h<br />metricbeat-v5n8l 1/1 Running 2 32h<br />metricbeat-cdfz9 1/1 Running 2 32h<br />metricbeat-z85g8 1/1 Running 2 32h<br />metricbeat-4fxhh 1/1 Running 2 32h</span></div><div style="text-align: left;"><span style="color: #3d85c6;">metricbeat-g25lh 1/1 Running 2 32h</span></div><p style="text-align: left;">6. At this point it's worth heading to Kibana Obserability page and from there you will see we have logs and metrics from our local k3d K8s cluster as shown below</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiIWmUGWKw2ysysW_T37CoyXhKNIELvC2p79XkdG9ORVGQCaZwnz5pCSWCNeLrBKvD75vcYYm9CdHd0ujYBYBlqiGfOKizR7vCH_n-0kv7Ouc0LzPXGTmcRCkM9g5mE9I7kEq0P1UlrqKlb/s2048/Screen+Shot+2021-02-25+at+7.50.38+pm.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1064" data-original-width="2048" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiIWmUGWKw2ysysW_T37CoyXhKNIELvC2p79XkdG9ORVGQCaZwnz5pCSWCNeLrBKvD75vcYYm9CdHd0ujYBYBlqiGfOKizR7vCH_n-0kv7Ouc0LzPXGTmcRCkM9g5mE9I7kEq0P1UlrqKlb/s320/Screen+Shot+2021-02-25+at+7.50.38+pm.png" width="320" /></a></div><br /><p style="text-align: left;">7. Now click on Metrics view and you can clearly see each of your k3d K8s nodes and by clicking on any node you get a very convenient view of metrics, process, metadata and more all from a single page. This was one of the new Elastic 7.11 features for Observability</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgs0ubUGdHKl8h7dztCZeu_SrLWinTQo5QPHYVxvT1zn5kt7hS1o6qEVc-kRxMlLamdUnYjqreUF7jJHMNdq8MWwAVb9c-IiLAga68S8lLOGZp4lxI9jYG8xTWUze4Ifv8cz0rKB_br4NSf/s2805/Screen+Shot+2021-02-25+at+7.55.02+pm.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1121" data-original-width="2805" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgs0ubUGdHKl8h7dztCZeu_SrLWinTQo5QPHYVxvT1zn5kt7hS1o6qEVc-kRxMlLamdUnYjqreUF7jJHMNdq8MWwAVb9c-IiLAga68S8lLOGZp4lxI9jYG8xTWUze4Ifv8cz0rKB_br4NSf/s320/Screen+Shot+2021-02-25+at+7.55.02+pm.png" width="320" /></a></div><br /><p style="text-align: left;"><br /></p><p style="text-align: left;">8. Finally to deploy a application to this K8s cluster and have Elastic APM instrument it's activity you can either use the example in the <a href="https://www.elastic.co/guide/en/observability/7.11/monitor-kubernetes.html" target="_blank">tutorial link</a> provided above OR just follow this very simple example to get this done</p><p style="text-align: left;"><a href="https://github.com/papicella/elastic-customer-api-rest">https://github.com/papicella/elastic-customer-api-rest</a></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh3D9Mb-76zYB_tdNHeqcnIu3MW5q8cx2xTo9jYvxQ699RrjSg6x_CKd8F6Y1bpv4RExNjOo5Q5DZHq5PYsBCYAstY7wottY3wBGT3EpsmYCasJMgIvR665eza3_OQSp61wZrDTmV4WBpMg/s3350/Screen+Shot+2021-02-25+at+7.59.45+pm.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="939" data-original-width="3350" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh3D9Mb-76zYB_tdNHeqcnIu3MW5q8cx2xTo9jYvxQ699RrjSg6x_CKd8F6Y1bpv4RExNjOo5Q5DZHq5PYsBCYAstY7wottY3wBGT3EpsmYCasJMgIvR665eza3_OQSp61wZrDTmV4WBpMg/s320/Screen+Shot+2021-02-25+at+7.59.45+pm.png" width="320" /></a></div><br /><p style="text-align: left;"><br /></p><h2 style="text-align: left;">More Information</h2><div style="text-align: left;">k3d Home Page</div><div style="text-align: left;"><a href="https://k3d.io/" target="_blank">k3d</a> </div><div style="text-align: left;"><br />Elastic Cloud Service<br /><a href="https://cloud.elastic.co/" target="_blank">Elastic Cloud </a></div><div style="text-align: left;"><br /></div><div class="blogger-post-footer">http://feeds.feedburner.com/TheBlasFromPas</div>Pas Apicellahttp://www.blogger.com/profile/09389663166398991762noreply@blogger.com0tag:blogger.com,1999:blog-6527688743456205256.post-8112059900482510512021-02-03T12:11:00.002+11:002021-02-03T12:11:55.521+11:00Elastic APM with Java Applications on Kubernetes <p>This GitHub demo is a Spring Boot application which accesses relational database with Spring Data JPA through a hypermedia-based RESTful front end. You can use the repo instructions to deploy to K8s and inject automatically an Elastic Agent for APM Monitoring with Elastic Observability.</p><p>There is no code changes required simply use an Init Container on K8s to instrument your application on Elastic APM server as shown by this GitHub repo below.</p><p><a href="https://github.com/papicella/elastic-customer-api-rest">https://github.com/papicella/elastic-customer-api-rest</a></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgH2riZ8CJHor4M3eZrSigGGJcsAWm7c-rZh446uYcCDmJA3kpwCawnYeLwzugvhqwB0r_ZeYE6mmtnoCKa4Fy4DDAfWvFnK-B215z6qsKkOhMehAi2OlciZv2IKTId4hL8phbrSso7RCzk/s2048/Screen+Shot+2021-02-03+at+12.07.32+pm.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1311" data-original-width="2048" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgH2riZ8CJHor4M3eZrSigGGJcsAWm7c-rZh446uYcCDmJA3kpwCawnYeLwzugvhqwB0r_ZeYE6mmtnoCKa4Fy4DDAfWvFnK-B215z6qsKkOhMehAi2OlciZv2IKTId4hL8phbrSso7RCzk/s320/Screen+Shot+2021-02-03+at+12.07.32+pm.png" width="320" /></a></div><br /><p><br /></p><p>Try it out on your Kubernetes cluster using <a href="https://www.elastic.co/cloud/" target="_blank">Elastic Cloud</a> using our free 14 day trial</p><p></p><div class="separator" style="clear: both; text-align: center;"><br /></div><br /><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEje4W_-f2IJf-ExT3VDf77oP3fAdnc0NMAS5301PWG4dXO8WBNGchmMOJV0QTvcKvWZHGVy8lyEBvsfxT1rFYjkA2Cucdf6NsMuJEVf2eNjlYcHk4RNT14sTWk53W25Ui-NLhyphenhyphenuR89ZT9ck/s3620/K8s-init-apm-10.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="869" data-original-width="3620" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEje4W_-f2IJf-ExT3VDf77oP3fAdnc0NMAS5301PWG4dXO8WBNGchmMOJV0QTvcKvWZHGVy8lyEBvsfxT1rFYjkA2Cucdf6NsMuJEVf2eNjlYcHk4RNT14sTWk53W25Ui-NLhyphenhyphenuR89ZT9ck/s320/K8s-init-apm-10.png" width="320" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiCgR4IfqQ4YJEQiAdwBu0kUoGh1QY9RXEhT8iakgnZ8t3-359elWYr2z9CucfHc4wo4bacG7VrTNZJ_el3xoSkZgDcCI5jQ5W6s7RcLxhkLZgMNV8kbdUC32j37W1k0__EZrQeuOpIlDDQ/s2048/K8s-init-apm-9.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1284" data-original-width="2048" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiCgR4IfqQ4YJEQiAdwBu0kUoGh1QY9RXEhT8iakgnZ8t3-359elWYr2z9CucfHc4wo4bacG7VrTNZJ_el3xoSkZgDcCI5jQ5W6s7RcLxhkLZgMNV8kbdUC32j37W1k0__EZrQeuOpIlDDQ/s320/K8s-init-apm-9.png" width="320" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbGJXyFT0lt7MXGUnuhtF-G5Yw2lBkIjP500Q9LPv00Zta8TOfhs0PcZJ7Wh6tQ9yRlYk4hokpF-nKNtn22jFiOG1o1KxVsSaf8W64XBOz8hqvHqmaYnjofTrsYestGRORJP0ZvaONtA8E/s2048/K8s-init-apm-8.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1574" data-original-width="2048" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbGJXyFT0lt7MXGUnuhtF-G5Yw2lBkIjP500Q9LPv00Zta8TOfhs0PcZJ7Wh6tQ9yRlYk4hokpF-nKNtn22jFiOG1o1KxVsSaf8W64XBOz8hqvHqmaYnjofTrsYestGRORJP0ZvaONtA8E/s320/K8s-init-apm-8.png" width="320" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi-wNY_3d1WbgxjghbFLjHWA4SPMpIphk3v5JQJvlPBLnWbltLryfa5dDlImSLKQl1CJOUV66iU89rMSUlo9jyWi9KT0N6cjJJhBoFVCw6A-Hm7my_xqnsb_8QLmtYjenZIyK2gJYpS9s_5/s2048/K8s-init-apm-7.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1158" data-original-width="2048" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi-wNY_3d1WbgxjghbFLjHWA4SPMpIphk3v5JQJvlPBLnWbltLryfa5dDlImSLKQl1CJOUV66iU89rMSUlo9jyWi9KT0N6cjJJhBoFVCw6A-Hm7my_xqnsb_8QLmtYjenZIyK2gJYpS9s_5/s320/K8s-init-apm-7.png" width="320" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEim_8Qug3T89bQfvPUzDD766dAWwDi0E3nSKWDtz-7Yh3QyIItBaJt2YVYmAko1xslXVGLyA270DY6zPbGkOy-QC0xQL6vghNqERORkpJ7_FolPwEpzJsM8sAodGY8Z0GdPL7boJg4MYauE/s3481/K8s-init-apm-6.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="904" data-original-width="3481" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEim_8Qug3T89bQfvPUzDD766dAWwDi0E3nSKWDtz-7Yh3QyIItBaJt2YVYmAko1xslXVGLyA270DY6zPbGkOy-QC0xQL6vghNqERORkpJ7_FolPwEpzJsM8sAodGY8Z0GdPL7boJg4MYauE/s320/K8s-init-apm-6.png" width="320" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjERCvYIjU1QtfQ_VNRg3wa-0lTKgcB6ZaLsninYXQ5bePGxdjVVJLqtHfRSYVNsafvHwQrf95utpw9OZYL0iJ_3mbwasBzaZk9f7NzLuOTO6n73YE13UViTMBKfv6p8vvJOe1DDYLtEsuS/s2774/K8s-init-apm-5.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1134" data-original-width="2774" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjERCvYIjU1QtfQ_VNRg3wa-0lTKgcB6ZaLsninYXQ5bePGxdjVVJLqtHfRSYVNsafvHwQrf95utpw9OZYL0iJ_3mbwasBzaZk9f7NzLuOTO6n73YE13UViTMBKfv6p8vvJOe1DDYLtEsuS/s320/K8s-init-apm-5.png" width="320" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgsWCaz8xKJ8Ljc4psBgBP9cPE23Yj0Yu4b8fVVnoppCMkafhcHzPIUCMqD5m5_Ebwo6Efy9wtYoWnLlgJbiaLhPTcrx9WDg6uFrnAejeV2TdgHrBGxIO2VIX5bc4ELxnQ443F7negkw9rc/s2048/K8s-init-apm-4.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1289" data-original-width="2048" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgsWCaz8xKJ8Ljc4psBgBP9cPE23Yj0Yu4b8fVVnoppCMkafhcHzPIUCMqD5m5_Ebwo6Efy9wtYoWnLlgJbiaLhPTcrx9WDg6uFrnAejeV2TdgHrBGxIO2VIX5bc4ELxnQ443F7negkw9rc/s320/K8s-init-apm-4.png" width="320" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEipcTAPWY0yottrPVtuim-5EMpgMyS_vKeRBN9XjjsC-e6frYPOKzSu7dwrR-fWoDW1G7YEuIbaZ0dy9jp0sC_CDuAQzOu6Pbee0rq8sf8r8LZT9uYciZHL4ugykCUFthCi9la-LdcodbSS/s3378/K8s-init-apm-3.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="931" data-original-width="3378" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEipcTAPWY0yottrPVtuim-5EMpgMyS_vKeRBN9XjjsC-e6frYPOKzSu7dwrR-fWoDW1G7YEuIbaZ0dy9jp0sC_CDuAQzOu6Pbee0rq8sf8r8LZT9uYciZHL4ugykCUFthCi9la-LdcodbSS/s320/K8s-init-apm-3.png" width="320" /></a></div><br /><div class="blogger-post-footer">http://feeds.feedburner.com/TheBlasFromPas</div>Pas Apicellahttp://www.blogger.com/profile/09389663166398991762noreply@blogger.com0tag:blogger.com,1999:blog-6527688743456205256.post-66182020430629139582020-12-29T11:41:00.003+11:002021-02-05T11:11:39.330+11:00Loading Australian Football League (AFL) Data into the Elastic Stack with some cool visulaizations<p>I decided to load some AFL data into the Elastic Stack and do some basic visualisations. I loaded data for all home and away plus finals games since 2017 so four seasons in total. Follow below if you want to do the same. </p><h2 style="text-align: left;">Steps</h2><p><span style="color: #e69138;"><i>Note: We already have Elasticsearch cluster running for this demo</i></span></p><pre class="brush: json">$ curl -u "elastic:welcome1" localhost:9200
{
"name" : "node1",
"cluster_name" : "apples-cluster",
"cluster_uuid" : "hJrp2eJaRGCfBt7Zg_-EJQ",
"version" : {
"number" : "7.10.0",
"build_flavor" : "default",
"build_type" : "tar",
"build_hash" : "51e9d6f22758d0374a0f3f5c6e8f3a7997850f96",
"build_date" : "2020-11-09T21:30:33.964949Z",
"build_snapshot" : false,
"lucene_version" : "8.7.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
</pre> <p></p><p>First I need the data loaded into the Elastic Stack I did that using <a href="https://api.squiggle.com.au/" target="_blank">Squiggle API</a> which you would do as follows</p><p>1. I use <a href="https://httpie.io/" target="_blank">HTTPie</a> rather then curl. </p><p><span style="color: #3d85c6;">http "https://api.squiggle.com.au/?q=games;complete=100" > games-2017-2020.json</span></p><p>2. Now this data itself needs to be altered slightly so I can BULK load it into Elasticsearch cluster and I do that as follows. I use JQ to do this.</p><p><span style="color: #3d85c6;">$ cat games-2017-2020.json | jq -c '.games[] | {"index": {"_id": .id}}, .' > converted-games-2017-2020.json</span></p><p>Snippet I what the JSON file now looks like</p><p><span style="color: #3d85c6;">{"index":{"_id":1}}</span></p><p><span style="color: #3d85c6;">{"round":1,"hgoals":14,"roundname":"Round 1","hteamid":3,"hscore":89,"winner":"Richmond","ateam":"Richmond","hbehinds":5,"venue":"M.C.G.","year":2017,"complete":100,"id":1,"localtime":"2017-03-23 19:20:00","agoals":20,"date":"2017-03-23 19:20:00","hteam":"Carlton","updated":"2017-04-15 15:59:16","tz":"+11:00","ascore":132,"ateamid":14,"winnerteamid":14,"is_grand_final":0,"abehinds":12,"is_final":0}</span></p><p><span style="color: #3d85c6;">{"index":{"_id":2}}</span></p><p><span style="color: #3d85c6;">{"date":"2017-03-24 19:50:00","agoals":15,"ateamid":18,"winnerteamid":18,"hteam":"Collingwood","updated":"2017-04-15 15:59:16","tz":"+11:00","ascore":100,"is_grand_final":0,"abehinds":10,"is_final":0,"round":1,"hgoals":12,"hscore":86,"winner":"Western Bulldogs","ateam":"Western Bulldogs","roundname":"Round 1","hteamid":4,"hbehinds":14,"venue":"M.C.G.","year":2017,"complete":100,"id":2,"localtime":"2017-03-24 19:50:00"}</span></p><p><span style="color: #3d85c6;">{"index":{"_id":3}}</span></p><p><span style="color: #3d85c6;">{"hscore":82,"ateam":"Port Adelaide","winner":"Port Adelaide","roundname":"Round 1","hteamid":16,"round":1,"hgoals":12,"complete":100,"id":3,"localtime":"2017-03-25 16:35:00","venue":"S.C.G.","hbehinds":10,"year":2017,"ateamid":13,"winnerteamid":13,"updated":"2017-04-15 15:59:16","hteam":"Sydney","tz":"+11:00","ascore":110,"date":"2017-03-25 16:35:00","agoals":17,"is_final":0,"is_grand_final":0,"abehinds":8}</span></p><p>Load data into Elasticsearch cluster as follows</p><p><span style="color: #3d85c6;">$ curl -u "elastic:welcome1" -H "Content-Type: application/json" -XPOST "localhost:9200/afl_games/_bulk?pretty&refresh" --data-binary "@converted-games-2017-2020.json"</span></p><p>3. Using DevTools with Kibana we can run a query as follows</p><p><span style="color: #cc0000;">Question: Get each teams winning games for the season 2020 before finals - Final Ladder</span></p><p><b>Query:</b></p><pre class="brush: json">GET afl_games/_search
{
"size": 0,
"query": {
"bool": {
"must": [
{
"match": {
"year": 2020
}
},
{
"match": {
"is_final": 0
}
}
]
}
},
"aggs": {
"group_by_winner": {
"terms": {
"field": "winner.keyword",
"size": 20
}
}
}
}
</pre> <p></p><p><b>Results:</b></p><p></p><pre class="brush: json">Results
{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 153,
"relation" : "eq"
},
"max_score" : null,
"hits" : [ ]
},
"aggregations" : {
"group_by_winner" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"key" : "Brisbane Lions",
"doc_count" : 14
},
{
"key" : "Port Adelaide",
"doc_count" : 14
},
{
"key" : "Geelong",
"doc_count" : 12
},
{
"key" : "Richmond",
"doc_count" : 12
},
{
"key" : "West Coast",
"doc_count" : 12
},
{
"key" : "St Kilda",
"doc_count" : 10
},
{
"key" : "Western Bulldogs",
"doc_count" : 10
},
{
"key" : "Collingwood",
"doc_count" : 9
},
{
"key" : "Melbourne",
"doc_count" : 9
},
{
"key" : "Greater Western Sydney",
"doc_count" : 8
},
{
"key" : "Carlton",
"doc_count" : 7
},
{
"key" : "Fremantle",
"doc_count" : 7
},
{
"key" : "Essendon",
"doc_count" : 6
},
{
"key" : "Gold Coast",
"doc_count" : 5
},
{
"key" : "Hawthorn",
"doc_count" : 5
},
{
"key" : "Sydney",
"doc_count" : 5
},
{
"key" : "Adelaide",
"doc_count" : 3
},
{
"key" : "North Melbourne",
"doc_count" : 3
}
]
}
}
}
</pre> <p></p><p>4. Finally using <a href="https://www.elastic.co/kibana/kibana-lens" target="_blank">Kibana Lens </a>to easily visualize this data using a Kibana Dasboard</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgsT1hoCj12_wdlU9baXsO_et2_KeAZIS_jqlgwU3i-xpDb6Wt2Mi-5W_9zDkdvPahbsVVHRugRAWKgvdTvBjoCUrUdgHVuYMUDEmCEEkCeG2lS91DbaUU6AVdU_9epkUxd8JSWjcPTbGym/s2048/Screen+Shot+2020-12-26+at+12.14.33+pm.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1074" data-original-width="2048" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgsT1hoCj12_wdlU9baXsO_et2_KeAZIS_jqlgwU3i-xpDb6Wt2Mi-5W_9zDkdvPahbsVVHRugRAWKgvdTvBjoCUrUdgHVuYMUDEmCEEkCeG2lS91DbaUU6AVdU_9epkUxd8JSWjcPTbGym/s320/Screen+Shot+2020-12-26+at+12.14.33+pm.png" width="320" /></a></div><p><br /></p><p>Of course you could do much more plus load more data from Squiggle and with the power of <a href="https://www.elastic.co/kibana" target="_blank">Kibana</a> feel free to create your own visualizations.</p><h2 style="text-align: left;">More Information</h2><p>Squiggle API</p><p><a href="https://api.squiggle.com.au/">https://api.squiggle.com.au/</a></p><p>Getting Started with the Elastic Stack</p><p><a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started.html">https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started.html</a></p><div class="blogger-post-footer">http://feeds.feedburner.com/TheBlasFromPas</div>Pas Apicellahttp://www.blogger.com/profile/09389663166398991762noreply@blogger.com0tag:blogger.com,1999:blog-6527688743456205256.post-37177993098907059862020-12-22T20:08:00.002+11:002020-12-22T20:08:33.888+11:00VMware Solutions Hub - Elastic Cloud on Kubernetes - the official Elasticsearch Operator from the creators<p>Proud to have worked on this with the VMware Tanzu team and Elastic team to add this to VMware Solution Hub page clearly highlighting what the Elastic Stack on Kubernetes really means.</p><p>Do you need to run your Elastic Stack on a certified Kubernetes distribution, bolstered
by the global Kubernetes community allowing you to focus on delivering
innovative applications powered by Elastic?</p><p>If so click below to get started:</p><p><a href="https://tanzu.vmware.com/solutions-hub/data-management/elastic">https://tanzu.vmware.com/solutions-hub/data-management/elastic</a></p><p><b>More Information</b></p><p><a href="https://tanzu.vmware.com/solutions-hub/data-management/elastic">https://tanzu.vmware.com/solutions-hub/data-management/elastic</a></p><div class="blogger-post-footer">http://feeds.feedburner.com/TheBlasFromPas</div>Pas Apicellahttp://www.blogger.com/profile/09389663166398991762noreply@blogger.com0tag:blogger.com,1999:blog-6527688743456205256.post-69458669938181251422020-10-28T09:18:00.003+11:002020-10-28T09:18:53.685+11:00How to Become a Kubernetes Admin from the Comfort of Your vSphere<p> My Talk at VMworld 2020 with Olive power can be found here.</p><p><b>Talk Details</b></p><p>In this session, we will walk through the integration of VMware vSphere and Kubernetes, and how this union of technologies can fundamentally change how virtual infrastructure and operational engineers view the management of Kubernetes platforms. We will demonstrate the capability of vSphere to host Kubernetes clusters internally, allocate capacity to those clusters, and monitor them side by side with virtual machines (VMs). We will talk about how extended vSphere functionality eases the transition of enterprises to running yet another platform (Kubernetes) by treating all managed endpoints—be they VMs, Kubernetes clusters or pods—as one platform. We want to demonstrate that platforms for running modern applications can be facilitated through the intuitive interface of vSphere and its ecosystem of automation tooling</p><p><a href="https://www.vmworld.com/en/video-library/search.html#text=%22KUB2038%22&year=2020">https://www.vmworld.com/en/video-library/search.html#text=%22KUB2038%22&year=2020</a></p><div class="blogger-post-footer">http://feeds.feedburner.com/TheBlasFromPas</div>Pas Apicellahttp://www.blogger.com/profile/09389663166398991762noreply@blogger.com0tag:blogger.com,1999:blog-6527688743456205256.post-86508954503448165972020-09-03T10:19:00.006+10:002020-09-03T10:19:33.921+10:00 java-cfenv : A library for accessing Cloud Foundry Services on the new Tanzu Application Service for Kubernetes<p>
The Spring Cloud Connectors library has been with us since the launch event of
Cloud Foundry itself back in 2011. This library would create the required
Spring Beans from bound VCAP_SERVICE ENV variable from a pushed Cloud Foundry
Application such as connecting to databases for example. The java buildpack
then replaces these bean definitions you had in your application with those
created by the connector library through a feature called
‘auto-reconfiguration’
</p>
<p>
Auto-reconfiguration is great for getting started. However, it is not so great
when you want more control, for example changing the size of the connection
pool associated with a DataSource.
</p>
<p>
With the up coming
<a href="https://network.pivotal.io/products/tas-for-kubernetes/" target="_blank">Tanzu Application Service for Kubernetes</a>
the original Cloud Foundry buildpacks are now replaced with the new Tanzu
Buildpacks which are based on the
<a href="https://buildpacks.io/" target="_blank">Cloud Native Buildpacks</a>
CNCF Sandbox project. As a result of this auto-reconfiguration is no
longer included in java cloud native buildpacks which means auto-configuration
for the backing services is no longer available.
</p>
<p>
So is their another option for this? The answer is "<b>Java CFEnv</b>". This provide
a simple API for retrieving credentials from the JSON strings contained inside
the <code><b>VCAP_SERVICES</b></code> environment variable.
</p>
<p>
<a href="https://github.com/pivotal-cf/java-cfenv">https://github.com/pivotal-cf/java-cfenv</a>
</p>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiufADUQazS8Nl8LCCze5cWBP6vCeGmQef-sSPoGO1UOMk6bgMV2G6x9TZenIm_ghAX-0-63mBkEuTdyCwRpMb34dQgle51M7VMenyXI2KJVGYxELt_Y1EyBgNsmwlxmLF4_DM5jVu2M3gZ/s1410/Screen+Shot+2020-09-03+at+10.06.25+am.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="920" data-original-width="1410" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiufADUQazS8Nl8LCCze5cWBP6vCeGmQef-sSPoGO1UOMk6bgMV2G6x9TZenIm_ghAX-0-63mBkEuTdyCwRpMb34dQgle51M7VMenyXI2KJVGYxELt_Y1EyBgNsmwlxmLF4_DM5jVu2M3gZ/s640/Screen+Shot+2020-09-03+at+10.06.25+am.png" width="640" /></a>
</div>
<br />
<p><br /></p>
<p>
So if you after exactly how it worked previously all you need to do is add
this maven dependancy to your project as shown below.</p>
<pre class="brush: xml">
<dependency>
<groupId>io.pivotal.cfenv</groupId>
<artifactId>java-cfenv-boot</artifactId>
</dependency>
</pre>
<p>Of course this new library is much more flexible then this and by using the class <b>CfEnv</b> as the entry point to the API for accessing Cloud Foundry environment variables your free to use the Spring Expression Language to invoke methods on the bean of type CfEnv to set properties for example plus more.</p>
<p>For more information read the full blog post as per below</p><p><a href="https://spring.io/blog/2019/02/15/introducing-java-cfenv-a-new-library-for-accessing-cloud-foundry-services">https://spring.io/blog/2019/02/15/introducing-java-cfenv-a-new-library-for-accessing-cloud-foundry-services</a></p><p>Finally this Spring Boot application is an example of using this new library with an application deployed to the new Tanzu Application Service for Kubernetes.</p><p><a href="https://github.com/papicella/spring-book-service">https://github.com/papicella/spring-book-service</a></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhVAdzDmG41DBk8sBLGQwo66B9xjtyNPICJzxZ9OErU5KYjDSmHfE8Q8XUolQ2Ukxk7mUtXhzO8VJBkkB7aH4JUK0E0VSrGj9hkamFtTBOhIdkMbIlLXiK3-Ff0oITQACnIDtnVHxHHMDsF/s1367/Screen+Shot+2020-09-03+at+10.16.51+am.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="884" data-original-width="1367" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhVAdzDmG41DBk8sBLGQwo66B9xjtyNPICJzxZ9OErU5KYjDSmHfE8Q8XUolQ2Ukxk7mUtXhzO8VJBkkB7aH4JUK0E0VSrGj9hkamFtTBOhIdkMbIlLXiK3-Ff0oITQACnIDtnVHxHHMDsF/s640/Screen+Shot+2020-09-03+at+10.16.51+am.png" width="640" /></a></div><p><br /></p><p><b>More Information</b></p>
<p>
1. Introducing java-cfenv: A new library for accessing Cloud Foundry Services
</p>
<p>
<a href="https://spring.io/blog/2019/02/15/introducing-java-cfenv-a-new-library-for-accessing-cloud-foundry-services">https://spring.io/blog/2019/02/15/introducing-java-cfenv-a-new-library-for-accessing-cloud-foundry-services</a>
</p>
<p>2. Java CFEnv GitHub Repo</p>
<p>
<a href="https://github.com/pivotal-cf/java-cfenv#pushing-your-application-to-cloud-foundry">https://github.com/pivotal-cf/java-cfenv#pushing-your-application-to-cloud-foundry</a>
</p>
<div class="blogger-post-footer">http://feeds.feedburner.com/TheBlasFromPas</div>Pas Apicellahttp://www.blogger.com/profile/09389663166398991762noreply@blogger.com0tag:blogger.com,1999:blog-6527688743456205256.post-15209268933853277342020-08-06T15:35:00.000+10:002020-08-06T15:35:58.232+10:00Configure a MySQL Marketplace service for the new Tanzu Application Service on Kubernetes using Container Services Manager for VMware TanzuThe following post shows how to configure a MySQL service into the new Tanzu
Application Service BETA version 0.3.0. For instructions on how to install
the Container Services Manager for VMware Tanzu (KSM) see post below.
<div><br /></div>
<div>
<a href="http://www.clue2solve.io/tanzu/2020/07/14/install-ksm-and-configure-the-cf-marketplace.html">http://www.clue2solve.io/tanzu/2020/07/14/install-ksm-and-configure-the-cf-marketplace.html</a><div>
<h2 style="text-align: left;"><b>Steps</b></h2>
<div>
It's assumed you have already installed KSM into your Kubernetes Cluster
as shown below. If not please refer to the documentation to get this done
first
</div>
<div><br /></div>
<div>
<a href="https://docs.pivotal.io/ksm/0-10/installing.html">https://docs.pivotal.io/ksm/0-10/installing.html</a>
</div>
<div><br /></div>
<pre class="brush: xml">$ kubectl get all -n ksm
NAME READY STATUS RESTARTS AGE
pod/ksm-chartmuseum-78d5d5bfb-2ggdg 1/1 Running 0 15d
pod/ksm-ksm-broker-6db696894c-blvpp 1/1 Running 0 15d
pod/ksm-ksm-broker-6db696894c-mnshg 1/1 Running 0 15d
pod/ksm-ksm-daemon-587b6fd549-cc7sv 1/1 Running 1 15d
pod/ksm-ksm-daemon-587b6fd549-fgqx5 1/1 Running 1 15d
pod/ksm-postgresql-0 1/1 Running 0 15d
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/ksm-chartmuseum ClusterIP 10.100.200.107 <none> 8080/TCP 15d
service/ksm-ksm-broker LoadBalancer 10.100.200.229 10.195.93.188 80:30086/TCP 15d
service/ksm-ksm-daemon LoadBalancer 10.100.200.222 10.195.93.179 80:31410/TCP 15d
service/ksm-postgresql ClusterIP 10.100.200.213 <none> 5432/TCP 15d
service/ksm-postgresql-headless ClusterIP None <none> 5432/TCP 15d
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/ksm-chartmuseum 1/1 1 1 15d
deployment.apps/ksm-ksm-broker 2/2 2 2 15d
deployment.apps/ksm-ksm-daemon 2/2 2 2 15d
NAME DESIRED CURRENT READY AGE
replicaset.apps/ksm-chartmuseum-78d5d5bfb 1 1 1 15d
replicaset.apps/ksm-ksm-broker-6db696894c 2 2 2 15d
replicaset.apps/ksm-ksm-broker-8645dfcf98 0 0 0 15d
replicaset.apps/ksm-ksm-daemon-587b6fd549 2 2 2 15d
NAME READY AGE
statefulset.apps/ksm-postgresql 1/1 15d
</pre>
<div><br /></div>
<div>
1. let's start by getting the Broker IP address which when installed using
LoadBalancer type can be retrieved as shown below.
</div>
<div><br /></div>
<div>
<span style="color: #3d85c6;">$ kubectl get service ksm-ksm-broker -n ksm
-o=jsonpath='{@.status.loadBalancer.ingress[0].ip}'</span>
</div>
<div><span style="color: #3d85c6;">10.195.93.188</span></div>
<div><br /></div>
<div>
2. Upgrade your Helm release by running the following using the IP address
from above
</div>
<div><br /></div>
<div>
<div>
<span style="color: #3d85c6;">$ export BROKER_IP=$(kubectl get service ksm-ksm-broker -n ksm
-o=jsonpath='{@.status.loadBalancer.ingress[0].ip}')</span>
</div>
<div>
<span style="color: #3d85c6;">$ helm upgrade ksm ./ksm -n ksm --reuse-values \</span>
</div>
<div>
<span style="color: #3d85c6;"> --set
cf.brokerUrl="http://$BROKER_IP" \</span>
</div>
<div>
<span style="color: #3d85c6;"> --set cf.brokerName=KSM
\</span>
</div>
<div>
<span style="color: #3d85c6;"> --set
cf.apiAddress="https://api.system.run.haas-210.pez.pivotal.io" \</span>
</div>
<div>
<span style="color: #3d85c6;"> --set cf.username="admin"
\</span>
</div>
<div>
<span style="color: #3d85c6;"> --set
cf.password="admin-password"</span>
</div>
</div>
<div><br /></div>
<div>
3. Next we configure the ksm CLI. You can download the CLI from
<a href="https://docs.pivotal.io/ksm/0-10/installing.html#cli" target="_blank">here</a>
</div>
<div><br /></div>
<div><b>configure-ksm-cli.sh</b></div>
<div><br /></div>
<div>
<div>
<span style="color: #3d85c6;">export KSM_IP=$(kubectl get service ksm-ksm-daemon -n ksm
-o=jsonpath='{@.status.loadBalancer.ingress[0].ip}')</span>
</div>
<div>
<span style="color: #3d85c6;">export KSM_TARGET=http://$KSM_IP:$(kubectl get svc ksm-ksm-daemon -n
ksm -o=jsonpath='{@.spec.ports[0].port}')</span>
</div>
<div><span style="color: #3d85c6;">export KSM_USER=admin</span></div>
<div>
<span style="color: #3d85c6;">export KSM_PASSWORD=$(kubectl get secret -n ksm ksm-ksm-daemon
-o=jsonpath='{@.data.SECURITY_USER_PASSWORD}' | base64 --decode)</span>
</div>
</div>
<div><br /></div>
<div>4. Verify ksm CLI is configured correctly</div>
<div><br /></div>
<div>
<div><span style="color: #3d85c6;">$ ksm version</span></div>
<div><span style="color: #3d85c6;">Client Version [0.10.80]</span></div>
<div><span style="color: #3d85c6;">Server Version [0.10.80]</span></div>
</div>
<div><br /></div>
<div>
5. Create a YAML file for the KSM service account and
<code>ClusterRoleBinding</code> using the following YAML:
</div>
<div><br /></div>
<div><b>ksm-sa.yml</b></div>
<div><br /></div>
<div>
<div><span style="color: #3d85c6;">---</span></div>
<div><span style="color: #3d85c6;">apiVersion: v1</span></div>
<div><span style="color: #3d85c6;">kind: ServiceAccount</span></div>
<div><span style="color: #3d85c6;">metadata:</span></div>
<div><span style="color: #3d85c6;"> name: ksm-admin</span></div>
<div>
<span style="color: #3d85c6;"> namespace: kube-system</span>
</div>
<div><span style="color: #3d85c6;">---</span></div>
<div>
<span style="color: #3d85c6;">apiVersion: rbac.authorization.k8s.io/v1beta1</span>
</div>
<div><span style="color: #3d85c6;">kind: ClusterRoleBinding</span></div>
<div><span style="color: #3d85c6;">metadata:</span></div>
<div>
<span style="color: #3d85c6;"> name: ksm-cluster-admin</span>
</div>
<div><span style="color: #3d85c6;">roleRef:</span></div>
<div>
<span style="color: #3d85c6;"> apiGroup: rbac.authorization.k8s.io</span>
</div>
<div><span style="color: #3d85c6;"> kind: ClusterRole</span></div>
<div><span style="color: #3d85c6;"> name: cluster-admin</span></div>
<div><span style="color: #3d85c6;">subjects:</span></div>
<div>
<span style="color: #3d85c6;"> - kind: ServiceAccount</span>
</div>
<div>
<span style="color: #3d85c6;"> name: ksm-admin</span>
</div>
<div>
<span style="color: #3d85c6;"> namespace: kube-system</span>
</div>
</div>
<div><br /></div>
<div>Apply as follows</div>
<div><br /></div>
<div>
<span style="color: #3d85c6;">$ kubectl apply -f ksm-sa.yml</span>
</div>
<div><br /></div>
<div>
6. You need a cluster credential file to register and set default
Kubernetes clusters that is done as follows
</div>
<div><br /></div>
<div><b>cluster-creds.sh</b></div>
<div><br /></div>
<div>
<div>
<span style="color: #3d85c6;">export kube_config="/Users/papicella/.kube/config"</span>
</div>
<div>
<span style="color: #3d85c6;"><br /></span>
</div>
<div>
<span style="color: #3d85c6;">cluster=`grep current $kube_config|sed "s/ //g"|cut -d ":" -f
2`</span>
</div>
<div>
<span style="color: #3d85c6;"><br /></span>
</div>
<div>
<span style="color: #3d85c6;">echo "Using cluster $cluster"</span>
</div>
<div>
<span style="color: #3d85c6;"><br /></span>
</div>
<div>
<span style="color: #3d85c6;">export server=`grep -B 2 "name: $cluster" $kube_config \</span>
</div>
<div>
<span style="color: #3d85c6;"> |grep server|sed "s/ //g"|sed "s/^[^:]*://g"`</span>
</div>
<div>
<span style="color: #3d85c6;"><br /></span>
</div>
<div>
<span style="color: #3d85c6;">export certificate=`grep -B 2 "name: $cluster" $kube_config \</span>
</div>
<div>
<span style="color: #3d85c6;"> |grep certificate|sed "s/ //g"|sed "s/.*://"`</span>
</div>
<div>
<span style="color: #3d85c6;"><br /></span>
</div>
<div>
<span style="color: #3d85c6;">export secret_name=$(kubectl get serviceaccount ksm-admin \</span>
</div>
<div>
<span style="color: #3d85c6;"> --namespace=kube-system -o
jsonpath='{.secrets[0].name}')</span>
</div>
<div>
<span style="color: #3d85c6;"><br /></span>
</div>
<div>
<span style="color: #3d85c6;">export secret_val=$(kubectl --namespace=kube-system get secret
$secret_name \</span>
</div>
<div>
<span style="color: #3d85c6;"> -o jsonpath='{.data.token}')</span>
</div>
<div>
<span style="color: #3d85c6;"><br /></span>
</div>
<div>
<span style="color: #3d85c6;">export secret_val=$(echo ${secret_val} | base64 --decode)</span>
</div>
<div>
<span style="color: #3d85c6;"><br /></span>
</div>
<div>
<span style="color: #3d85c6;">cat > cluster-creds.yaml << EOF</span>
</div>
<div><span style="color: #3d85c6;">token: ${secret_val}</span></div>
<div><span style="color: #3d85c6;">server: ${server}</span></div>
<div><span style="color: #3d85c6;">caData: ${certificate}</span></div>
<div><span style="color: #3d85c6;">EOF</span></div>
<div>
<span style="color: #3d85c6;"><br /></span>
</div>
<div><span style="color: #3d85c6;">echo ""</span></div>
<div><span style="color: #3d85c6;">echo "ready to roll!!!!"</span></div>
<div><span style="color: #3d85c6;">echo ""</span></div>
</div>
<div><br /></div>
<div>
Before running this script it's best to make sure you have targeted the
correct K8s cluster you wish to. You can run a command as follows to
verify that
</div>
<div><br /></div>
<div>
<div>
<span style="color: #3d85c6;">$ kubectl config current-context</span>
</div>
<div><span style="color: #3d85c6;">tas4k8s</span></div>
</div>
<div> </div>
<div>7. Now we have a "<b>cluster-creds.yaml</b>" file we can go ahead and
register the Kubernetes cluster with KSM as follows</div>
<div><br /></div>
<div>
<div>
<span style="color: #3d85c6;">$ ksm cluster register ksm-svcs ./cluster-creds.yaml</span>
</div>
<div>
<span style="color: #3d85c6;">$ ksm cluster set-default ksm-svcs</span>
</div>
</div>
<div><br /></div>
<div>Verify as follows:</div>
<div><br /></div>
<div>
<div><span style="color: #3d85c6;">$ ksm cluster list</span></div>
<div>
<span style="color: #3d85c6;">CLUSTER NAME<span style="white-space: pre;"> </span>IP ADDRESS
<span style="white-space: pre;"> </span>DEFAULT</span>
</div>
<div>
<span style="color: #3d85c6;">ksm-svcs
<span style="white-space: pre;"> </span>https://tas4k8s.run.haas-210.pez.pivotal.io:8443<span style="white-space: pre;">
</span>true</span>
</div>
</div>
<div><br /></div>
<div>
8. Now we can go ahead and create a Marketplace offering for MySQL. To do
that we will use the Bitnami MySQL chart as shown below
</div>
<div><br /></div>
<div>
<div>
<span style="color: #3d85c6;">$ git clone https://github.com/bitnami/charts.git</span>
</div>
<div>
<span style="color: #3d85c6;">$ cd ./charts/bitnami/mysql</span>
</div>
<div><br /></div>
<div>
<b>** create bind.yaml as follows which is required so our service
binding from Tanzu Application Service will inject the right JSON we
are expecting or requiring at bind time **</b>
</div>
<div><br /></div>
<div><span style="color: #3d85c6;">$ cat bind.yaml</span></div>
<div><span style="color: #3d85c6;">template: |</span></div>
<div>
<span style="color: #3d85c6;"> local filterfunc(j) = std.length(std.findSubstr("mysql",
j.name)) > 0;</span>
</div>
<div>
<span style="color: #3d85c6;"> local s1 = std.filter(filterfunc, $.services);</span>
</div>
<div><span style="color: #3d85c6;"> {</span></div>
<div>
<span style="color: #3d85c6;"> hostname:
s1[0].status.loadBalancer.ingress[0].ip,</span>
</div>
<div>
<span style="color: #3d85c6;"> name: s1[0].name,</span>
</div>
<div>
<span style="color: #3d85c6;"> jdbcUrl: "jdbc:mysql://" + self.hostname +
"/my_db?user=" + self.username + "&password=" + self.password +
"&useSSL=false",</span>
</div>
<div>
<span style="color: #3d85c6;"> uri: "mysql://" + self.username + ":" + self.password +
"@" + self.hostname + ":" + self.port + "/my_db?reconnect=true",</span>
</div>
<div>
<span style="color: #3d85c6;"> password:
$.secrets[0].data['mysql-root-password'],</span>
</div>
<div><span style="color: #3d85c6;"> port: 3306,</span></div>
<div>
<span style="color: #3d85c6;"> username: "root"</span>
</div>
<div><span style="color: #3d85c6;"> }</span></div>
<div>
<span style="color: #3d85c6;"><br /></span>
</div>
<div><span style="color: #3d85c6;">$ helm package .</span></div>
<div><span style="color: #3d85c6;"># cd ..</span></div>
<div>
<span style="color: #3d85c6;">$ ksm offer save ./mysql ./mysql/mysql-6.14.7.tgz</span>
</div>
</div>
<div><br /></div>
<div>Verify MySQL is now part of the offer list as follows</div>
<div>
<pre class="brush: xml">
$ ksm offer list
MARKETPLACE NAME INCLUDED CHARTS VERSION PLANS
rabbitmq rabbitmq 6.18.1 [persistent ephemeral]
mysql mysql 6.14.7 [default]
</pre>
</div>
<div><br /></div>
<div>9. Now we need to login as an ADMIN user</div>
<div><br /></div>
<div>Verify you are logged in as admin user using the CF CLI:</div>
<div><br /></div>
<div>
<div><span style="color: #3d85c6;">$ cf target</span></div>
<div>
<span style="color: #3d85c6;">api endpoint:
https://api.system.run.haas-210.pez.pivotal.io</span>
</div>
<div>
<span style="color: #3d85c6;">api version: 2.151.0</span>
</div>
<div>
<span style="color: #3d85c6;">user: admin</span>
</div>
<div>
<span style="color: #3d85c6;">org: system</span>
</div>
<div>
<span style="color: #3d85c6;">space: development</span>
</div>
</div>
<div><br /></div>
<div>
10. At this point you can see the KSM service broker registered with
TAS4K8s as follows
</div>
<div><br /></div>
<div>
<div><span style="color: #3d85c6;">$ cf service-brokers</span></div>
<div>
<span style="color: #3d85c6;">Getting service brokers as admin...</span>
</div>
<div>
<span style="color: #3d85c6;"><br /></span>
</div>
<div><span style="color: #3d85c6;">name url</span></div>
<div>
<span style="color: #3d85c6;">KSM http://10.195.93.188</span>
</div>
</div>
<div><br /></div>
<div>11. Enable access to the MySQL service as follows</div>
<div><br /></div>
<div>
<span style="color: #3d85c6;">$ cf enable-service-access mysql</span>
</div>
<div><br /></div>
<div>Verify it's enabled:</div>
<div><br /></div>
<div>
<div><span style="color: #3d85c6;">$ cf service-access</span></div>
<div>
<span style="color: #3d85c6;">Getting service access as admin...</span>
</div>
<div><span style="color: #3d85c6;">broker: KSM</span></div>
<div>
<span style="color: #3d85c6;"> service plan
access orgs</span>
</div>
<div>
<span style="color: #3d85c6;"> mysql default
all</span>
</div>
<div>
<span style="color: #3d85c6;"> rabbitmq ephemeral all</span>
</div>
<div>
<span style="color: #3d85c6;"> rabbitmq persistent all</span>
</div>
</div>
<div><br /></div>
<div>
12. At this point it's best to log out of admin and log back in as a user
that is not admin
</div>
<div><br /></div>
<div>
<div><span style="color: #3d85c6;">$ cf target</span></div>
<div>
<span style="color: #3d85c6;">api endpoint:
https://api.system.run.haas-210.pez.pivotal.io</span>
</div>
<div>
<span style="color: #3d85c6;">api version: 2.151.0</span>
</div>
<div>
<span style="color: #3d85c6;">user: pas</span>
</div>
<div>
<span style="color: #3d85c6;">org: apples-org</span>
</div>
<div>
<span style="color: #3d85c6;">space: development</span>
</div>
</div>
<div><br /></div>
<div>13. Create a MySQL service as follows. I passing in some JSON to indicate
that my K8s cluster support's a LoadBalancer type so use that as part of the
creation of the service.</div>
<div><br /></div>
<div>
<span style="color: #3d85c6;">$ cf create-service mysql default pas-mysql -c
'{"service":{"type":"LoadBalancer"}}'</span>
</div>
<div><br /></div>
<div>
14. Check that the service has created correctly it will take a few
minutes
</div>
<div><br /></div>
<div>
<div><span style="color: #3d85c6;">$ cf services</span></div>
<div>
<span style="color: #3d85c6;">Getting services in org apples-org / space development as
pas...</span>
</div>
<div>
<span style="color: #3d85c6;"><br /></span>
</div>
<div>
<span style="color: #3d85c6;">name service plan
bound apps last
operation broker upgrade
available</span>
</div>
<div>
<span style="color: #3d85c6;">pas-mysql mysql default
my-springboot-app create succeeded
KSM no</span>
</div>
</div>
<div><br /></div>
<div>
15. Your service is created in it's own K8s namespace BUT that may not be
the case at some point.
</div>
<pre class="brush: xml">$ kubectl get all -n ksm-2e526124-11a3-4d38-966c-b3ffd45471d7
NAME READY STATUS RESTARTS AGE
pod/k-wqo5mubw-mysql-master-0 1/1 Running 0 15d
pod/k-wqo5mubw-mysql-slave-0 1/1 Running 0 15d
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/k-wqo5mubw-mysql LoadBalancer 10.100.200.12 10.195.93.192 3306:30563/TCP 15d
service/k-wqo5mubw-mysql-slave LoadBalancer 10.100.200.130 10.195.93.191 3306:31982/TCP 15d
NAME READY AGE
statefulset.apps/k-wqo5mubw-mysql-master 1/1 15d
statefulset.apps/k-wqo5mubw-mysql-slave 1/1 15d
</pre>
<div><br /></div>
<div>
16. At this point we can now test our new MySQL service we created and use
a Spring Boot application to test this out with.
</div>
<div><br /></div>
<div>
The following GitHub repo can be used for that. Ignore the steps to create
a service as you have already done that
</div>
<div><br /></div>
<div>
<a href="https://github.com/papicella/spring-book-service">https://github.com/papicella/spring-book-service</a>
</div>
<div><br /></div>
<div class="separator" style="clear: both;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEghU4C3wODqgjNe2sosnWCvK_lB1ldpdude1nItD8ii7Pi3s7Do2ofr0l1QSNgJVg_R_aL5OgWrHwlZIp1cmLjApANrNPPc9jdhY-BgeTf62gPXJnzYaIiqB0gY6yuFMb3Zww4ztLX2RPXL/s1613/Screen+Shot+2020-08-06+at+3.25.48+pm.png" style="display: block; padding: 1em 0px;"><img border="0" data-original-height="809" data-original-width="1613" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEghU4C3wODqgjNe2sosnWCvK_lB1ldpdude1nItD8ii7Pi3s7Do2ofr0l1QSNgJVg_R_aL5OgWrHwlZIp1cmLjApANrNPPc9jdhY-BgeTf62gPXJnzYaIiqB0gY6yuFMb3Zww4ztLX2RPXL/s640/Screen+Shot+2020-08-06+at+3.25.48+pm.png" width="640" /></a>
</div>
<div><br /></div>
<div><br /></div>
<div>Finally to define service plans see the link below</div>
<div><br /></div>
<div>
<a href="https://docs.pivotal.io/ksm/0-10/prepare-offer.html#plans">https://docs.pivotal.io/ksm/0-10/prepare-offer.html#plans</a></div><h2 style="text-align: left;"><b>More Information</b></h2></div>
<div>Container Services Manager(KSM)</div>
<div>
<a href="https://network.pivotal.io/products/container-services-manager/">https://network.pivotal.io/products/container-services-manager/</a>
</div>
<div><br /></div>
<div>Tanzu Application Service for Kubernetes</div>
<div>
<a href="https://network.pivotal.io/products/tas-for-kubernetes/">https://network.pivotal.io/products/tas-for-kubernetes/</a>
</div>
<div><br /></div>
</div>
<div class="blogger-post-footer">http://feeds.feedburner.com/TheBlasFromPas</div>Pas Apicellahttp://www.blogger.com/profile/09389663166398991762noreply@blogger.com0tag:blogger.com,1999:blog-6527688743456205256.post-82194416713186419472020-08-03T13:45:00.003+10:002020-08-03T13:45:35.828+10:00Using CNCF Sandbox Project Strimzi for Kafka Clusters on VMware Tanzu Kubernetes Grid Integrated Edition (TKGI)<div>
Strimzi a
<a href="https://www.cncf.io/sandbox-projects/" target="_blank">CNCF sandbox project</a>
provides a way to run an Apache Kafka cluster on Kubernetes in various
deployment configurations. In this post we will take a look at how to get this
running on VMware Tanzu Kubernetes Grid Integrated Edition (TKGI) and
consume the Kafka cluster from a Springboot application.
</div>
<div>
<b><br /></b>
</div>
<div>
If you have a K8s cluster that's all you need to follow along in this exampleI
am using VMware Tanzu Kubernetes Grid Integrated Edition (TKGI) but you can
use any K8s cluster you have such as GKE, AKS, EKS etc.
</div>
<div><br /></div>
<b>Steps</b>
<div><br /></div>
<div>
1. Installing Strimzi is pretty straight forward so we can do that as follows.
I am using the namespace "<b>kafka</b>" which needs to be created prior to
running this command.</div>
<div><br /></div>
<div>
<font color="#3d85c6">kubectl apply -f 'https://strimzi.io/install/latest?namespace=kafka' -n
kafka</font>
</div>
<div><br /></div>
<div>
2. Verify that the operator was installed correctly and we have a running POD
as shown below
</div>
<pre class="brush: xml">
$ kubectl get pods -n kafka
NAME READY STATUS RESTARTS AGE
strimzi-cluster-operator-6c9d899778-4mdtg 1/1 Running 0 6d22h</pre>
<div><br /></div>
<div>
3. Next let's ensure we have a default storage class for the cluster as shown
below.
</div>
<div><br /></div>
<div>
<div><font color="#3d85c6">$ kubectl get storageclass</font></div>
<div>
<font color="#3d85c6">NAME PROVISIONER
AGE</font>
</div>
<div>
<font color="#3d85c6">fast (default) kubernetes.io/vsphere-volume
47d</font>
</div>
</div>
<div><br /></div>
<div>
4. Now at this point we are ready to create a Kafka cluster. For this example
we will create a 3 node cluster defined in YML as follows.</div>
<div><br /></div>
<div><b>kafka-persistent-MULTI_NODE.yaml</b></div>
<div><br /></div>
<div>
<div><font color="#3d85c6">apiVersion: kafka.strimzi.io/v1beta1</font></div>
<div><font color="#3d85c6">kind: Kafka</font></div>
<div><font color="#3d85c6">metadata:</font></div>
<div><font color="#3d85c6"> name: apples-kafka-cluster</font></div>
<div><font color="#3d85c6">spec:</font></div>
<div><font color="#3d85c6"> kafka:</font></div>
<div><font color="#3d85c6"> version: 2.5.0</font></div>
<div><font color="#3d85c6"> replicas: 3</font></div>
<div><font color="#3d85c6"> listeners:</font></div>
<div><font color="#3d85c6"> external:</font></div>
<div>
<font color="#3d85c6"> type: loadbalancer</font>
</div>
<div><font color="#3d85c6"> tls: false</font></div>
<div><font color="#3d85c6"> plain: {}</font></div>
<div><font color="#3d85c6"> tls: {}</font></div>
<div><font color="#3d85c6"> config:</font></div>
<div>
<font color="#3d85c6"> offsets.topic.replication.factor: 3</font>
</div>
<div>
<font color="#3d85c6"> transaction.state.log.replication.factor: 3</font>
</div>
<div>
<font color="#3d85c6"> transaction.state.log.min.isr: 2</font>
</div>
<div>
<font color="#3d85c6"> log.message.format.version: "2.5"</font>
</div>
<div><font color="#3d85c6"> storage:</font></div>
<div><font color="#3d85c6"> type: jbod</font></div>
<div><font color="#3d85c6"> volumes:</font></div>
<div><font color="#3d85c6"> - id: 0</font></div>
<div>
<font color="#3d85c6"> type: persistent-claim</font>
</div>
<div>
<font color="#3d85c6"> size: 100Gi</font>
</div>
<div>
<font color="#3d85c6"> deleteClaim: false</font>
</div>
<div><font color="#3d85c6"> zookeeper:</font></div>
<div><font color="#3d85c6"> replicas: 3</font></div>
<div><font color="#3d85c6"> storage:</font></div>
<div>
<font color="#3d85c6"> type: persistent-claim</font>
</div>
<div><font color="#3d85c6"> size: 100Gi</font></div>
<div>
<font color="#3d85c6"> deleteClaim: false</font>
</div>
<div><font color="#3d85c6"> entityOperator:</font></div>
<div><font color="#3d85c6"> topicOperator: {}</font></div>
<div><font color="#3d85c6"> userOperator: {}</font></div>
</div>
<div><br /></div>
<div>Few things to note:</div>
<div>
<ul style="text-align: left;">
<li>
We have enable access to the cluster using the type LoadBalancer which
means your K8s cluster needs to support such a Type
</li>
<li>
We need to create dynamic Persistence claim's in the cluster so ensure #3
above is in place
</li>
<li>We have disabled TLS given this is a demo </li>
</ul>
</div>
<div>
5. Create the Kafka cluster as shown below ensuring we target the namespace
"<b>kafka</b>"
</div>
<div><br /></div>
<div>
<font color="#3d85c6">$ kubectl apply -f kafka-persistent-MULTI_NODE.yaml -n
kafka</font>
</div>
<div><br /></div>
<div>
6. Now we can view the status/creation of our cluster one of two ways as shown
below. You will need to wait a few minutes for everything to start up.</div>
<div><br /></div>
<div><b>Option 1:</b></div>
<pre class="brush: xml">
$ kubectl get Kafka -n kafka
NAME DESIRED KAFKA REPLICAS DESIRED ZK REPLICAS
apples-kafka-cluster 3 3 1/1 Running 0 6d22h
</pre>
<div><br /></div>
<div><b>Option 2:</b></div>
<pre class="brush: xml">
$ kubectl get all -n kafka
NAME READY STATUS RESTARTS AGE
pod/apples-kafka-cluster-entity-operator-58685b8fbd-r4wxc 3/3 Running 0 6d21h
pod/apples-kafka-cluster-kafka-0 2/2 Running 0 6d21h
pod/apples-kafka-cluster-kafka-1 2/2 Running 0 6d21h
pod/apples-kafka-cluster-kafka-2 2/2 Running 0 6d21h
pod/apples-kafka-cluster-zookeeper-0 1/1 Running 0 6d21h
pod/apples-kafka-cluster-zookeeper-1 1/1 Running 0 6d21h
pod/apples-kafka-cluster-zookeeper-2 1/1 Running 0 6d21h
pod/strimzi-cluster-operator-6c9d899778-4mdtg 1/1 Running 0 6d23h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/apples-kafka-cluster-kafka-0 LoadBalancer 10.100.200.90 10.195.93.200 9094:30362/TCP 6d21h
service/apples-kafka-cluster-kafka-1 LoadBalancer 10.100.200.179 10.195.93.197 9094:32022/TCP 6d21h
service/apples-kafka-cluster-kafka-2 LoadBalancer 10.100.200.155 10.195.93.201 9094:32277/TCP 6d21h
service/apples-kafka-cluster-kafka-bootstrap ClusterIP 10.100.200.77 <none> 9091/TCP,9092/TCP,9093/TCP 6d21h
service/apples-kafka-cluster-kafka-brokers ClusterIP None <none> 9091/TCP,9092/TCP,9093/TCP 6d21h
service/apples-kafka-cluster-kafka-external-bootstrap LoadBalancer 10.100.200.58 10.195.93.196 9094:30735/TCP 6d21h
service/apples-kafka-cluster-zookeeper-client ClusterIP 10.100.200.22 <none> 2181/TCP 6d21h
service/apples-kafka-cluster-zookeeper-nodes ClusterIP None <none> 2181/TCP,2888/TCP,3888/TCP 6d21h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/apples-kafka-cluster-entity-operator 1/1 1 1 6d21h
deployment.apps/strimzi-cluster-operator 1/1 1 1 6d23h
NAME DESIRED CURRENT READY AGE
replicaset.apps/apples-kafka-cluster-entity-operator-58685b8fbd 1 1 1 6d21h
replicaset.apps/strimzi-cluster-operator-6c9d899778 1 1 1 6d23h
NAME READY AGE
statefulset.apps/apples-kafka-cluster-kafka 3/3 6d21h
statefulset.apps/apples-kafka-cluster-zookeeper 3/3 6d21h 3 1/1 Running 0 6d22h
</pre>
<div><br /></div>
<div>
7. Our entry point into the cluster is a service of type LoadBalancer which we
asked for as per our Kafka cluster YML config. To find the IP address we can
run a command as follow using the cluster name from above.</div>
<div><br /></div>
<div>
<div>
<font color="#3d85c6">$ kubectl get service -n kafka
apples-kafka-cluster-kafka-external-bootstrap
-o=jsonpath='{.status.loadBalancer.ingress[0].ip}{"\n"}'</font>
</div>
<div><font color="#3d85c6">10.195.93.196</font></div>
</div>
<div><br /></div>
<div>
<font color="#e69138">Note: Make a not of this IP address as we will need it shortly</font>
</div>
<div><br /></div>
<div>
8. Let's create a Kafka Topic using YML as follows. In this YML we actually
ensure we are using the namespace "<b>kafka</b>".
</div>
<div><br /></div>
<div><b>create-kafka-topic.yaml</b></div>
<div><br /></div>
<div>
<div><font color="#3d85c6">apiVersion: kafka.strimzi.io/v1beta1</font></div>
<div><font color="#3d85c6">kind: KafkaTopic</font></div>
<div><font color="#3d85c6">metadata:</font></div>
<div><font color="#3d85c6"> name: apples-topic</font></div>
<div><font color="#3d85c6"> namespace: kafka</font></div>
<div><font color="#3d85c6"> labels:</font></div>
<div>
<font color="#3d85c6"> strimzi.io/cluster: apples-kafka-cluster</font>
</div>
<div><font color="#3d85c6">spec:</font></div>
<div><font color="#3d85c6"> partitions: 1</font></div>
<div><font color="#3d85c6"> replicas: 1</font></div>
<div><font color="#3d85c6"> config:</font></div>
<div><font color="#3d85c6"> retention.ms: 7200000</font></div>
<div>
<font color="#3d85c6"> segment.bytes: 1073741824</font>
</div>
</div>
<div><br /></div>
<div><br /></div>
<div>9. Create a Kafka topic as shown below.</div>
<div><br /></div>
<div>
<font color="#3d85c6">$ kubectl apply -f create-kafka-topic.yaml</font>
</div>
<div><br /></div>
<div>10. We can view the Kafka topics as shown below.</div>
<pre class="brush: xml">
$ kubectl get KafkaTopic -n kafka
NAME PARTITIONS REPLICATION FACTOR
apples-topic 1 1
</pre>
<div><br /></div>
<div>
11. Now at this point we ready to send some messages to our topic
"apples-topic" as well as consume messages so to do that we are going to use a
Springboot Application in fact two of them which exist on GitHub.
</div>
<div><br /></div>
<div>
<a href="https://github.com/papicella/demo-kafka-producer" target="_blank">Producer Springboot</a>
</div>
<div>
<a href="https://github.com/papicella/demo-kafka-consumer" target="_blank">Consumer Springboot</a>
</div>
<div><br /></div>
<div>Download or clone those onto your file system. </div>
<div><br /></div>
<div>
12.With both downloaded you will need to set the
<b>spring.kafka.bootstrap-servers</b> with the IP address we retrieved from #7
above. That needs to be done in both GitHub downloaded/cloned repo's above.
The file we need to edit for both repo's is as follows.
</div>
<div><br /></div>
<div>File: <b>src/main/resources/application.yml</b> </div>
<div><br /></div>
<div>Example:</div>
<div><br /></div>
<div>
<div><font color="#3d85c6">spring:</font></div>
<div><font color="#3d85c6"> kafka:</font></div>
<div>
<font color="#3d85c6"> bootstrap-servers: IP-ADDRESS:9094</font>
</div>
</div>
<div><br /></div>
<div><font color="#e69138">
Note: Make sure you do this for both downloaded repo
<b>application.yml</b> files
</font></div>
<div><br /></div>
<div>
13. Now let's run the producer and consumer Springboot application using a
command as follows in seperate terminal windows. One will use PORT 8080 while
the other uses port 8081.</div>
<div><br /></div>
<div><b>$ ./mvnw spring-boot:run</b></div>
<div><br /></div>
<div>Consumer:</div>
<div><br /></div>
<div>
<font color="#3d85c6">papicella@papicella:~/pivotal/DemoProjects/spring-starter/pivotal/KAFKA/demo-kafka-producer$
./mvnw spring-boot:run</font>
</div>
<div>
<font color="#3d85c6"><br /></font>
</div>
<div><font color="#3d85c6">...</font></div>
<div>
<div>
<font color="#3d85c6">2020-08-03 11:41:46.742 INFO 34025 --- [
main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat
started on port(s): 8080 (http) with context path ''</font>
</div>
<div>
<font color="#3d85c6">2020-08-03 11:41:46.754 INFO 34025 --- [
main] a.a.t.k.DemoKafkaProducerApplication
: Started DemoKafkaProducerApplication in 1.775 seconds (JVM running
for 2.102)</font>
</div>
</div>
<div>
<font color="#3d85c6"><br /></font>
</div>
<div>Producer:</div>
<div>
<font color="#3d85c6"><br /></font>
</div>
<div>
<div>
<font color="#3d85c6">papicella@papicella:~/pivotal/DemoProjects/spring-starter/pivotal/KAFKA/demo-kafka-consumer$
./mvnw spring-boot:run</font>
</div>
</div>
<div>
<font color="#3d85c6"><br /></font>
</div>
<div><font color="#3d85c6">...</font></div>
<div>
<div>
<font color="#3d85c6">2020-08-03 11:43:53.423 INFO 34056 --- [
main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat
started on port(s): 8081 (http) with context path ''</font>
</div>
<div>
<font color="#3d85c6">2020-08-03 11:43:53.440 INFO 34056 --- [
main] a.a.t.k.DemoKafkaConsumerApplication
: Started DemoKafkaConsumerApplication in 1.666 seconds (JVM running
for 1.936)</font>
</div>
</div>
<div><br /></div>
<div>
14. Start by opening up the the Producer UI by navigating to
<a href="http://localhost:8080/">http://localhost:8080/</a>
</div>
<div><br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjhyphenhyphenmnl_6OiB_9_Vgkf0ziyNKneRvQJVDGpnrdI_XPjYxXy9nOpbV27TSqUCC8Q1OD4dg2uO8-pw1CL01nP9Zf8s_v_3YOSZCJmA0-dD8e5WW2zCRUbalweGJvE0Bh_CyGotSIWz_YrHgJ_/s1377/Screen+Shot+2020-08-03+at+11.47.20+am.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="337" data-original-width="1377" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjhyphenhyphenmnl_6OiB_9_Vgkf0ziyNKneRvQJVDGpnrdI_XPjYxXy9nOpbV27TSqUCC8Q1OD4dg2uO8-pw1CL01nP9Zf8s_v_3YOSZCJmA0-dD8e5WW2zCRUbalweGJvE0Bh_CyGotSIWz_YrHgJ_/s640/Screen+Shot+2020-08-03+at+11.47.20+am.png" width="640" /></a>
</div>
<div><br /></div>
<div><br /></div>
<div>
15. Now let's not add any messages yet and also open up the Consumer UI by
navigating to <a href="http://localhost:8081/">http://localhost:8081/</a>
</div>
<div><br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjmlOQU1kv9lU80iJKLVU-eQfZ1yBcuIqSdPGpPLvgG93mILREdnKB65F8-me9s4VWA5Pv6iVIwnUvWfuXho_MifU86Us_ezUKn5dEWKCsG27Mwvq2UIMgiTdxawPUVkbOBjJaqRNF3GgCS/s1377/Screen+Shot+2020-08-03+at+11.49.01+am.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="353" data-original-width="1377" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjmlOQU1kv9lU80iJKLVU-eQfZ1yBcuIqSdPGpPLvgG93mILREdnKB65F8-me9s4VWA5Pv6iVIwnUvWfuXho_MifU86Us_ezUKn5dEWKCsG27Mwvq2UIMgiTdxawPUVkbOBjJaqRNF3GgCS/s640/Screen+Shot+2020-08-03+at+11.49.01+am.png" width="640" /></a>
</div>
<div><br /></div>
<div><br /></div>
<div>
<font color="#e69138">Note: This application will automatically refresh the page every 2 seconds
to show which messages have been sent to the Kafka Topic</font>
</div>
<div><br /></div>
<div>
16. Return to the Producer UI <a href="http://localhost:8080/">http://localhost:8080/</a> and add two messages using whatever text you like as shown below.</div>
<div><br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-AvXSLJKmEmEQFO98HepF7C7c_GIFkV435w9QjRCw0ly7cBpoDAX4IhdvycpKHCBHf3LjHv1MVmrM3QR1sst9JKv7qTUr6HdmekA50WP8ha0PBKqM_gSLxRC6-haFhviKVBhnlnb1i4iH/s1373/Screen+Shot+2020-08-03+at+11.53.07+am.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="394" data-original-width="1373" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg-AvXSLJKmEmEQFO98HepF7C7c_GIFkV435w9QjRCw0ly7cBpoDAX4IhdvycpKHCBHf3LjHv1MVmrM3QR1sst9JKv7qTUr6HdmekA50WP8ha0PBKqM_gSLxRC6-haFhviKVBhnlnb1i4iH/s640/Screen+Shot+2020-08-03+at+11.53.07+am.png" width="640" /></a>
</div>
<div><br /></div>
<div>
17. Return to the Consumer UI <a href="http://localhost:8081/">http://localhost:8081/</a> to verify the two messages sent to the Kafka topic has been consumed
</div>
<div><br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhJSbe8j0ELWS72D08LF5w2mQlQtYfMKzVHW5OwCWTHutjMnLDK8PntAlpE_7haFR85LaYqPS2tXzJqSbq15ZJ8X_UJ8JSHRY3DPi4AsrdkyXcznkc0z-s9hP5cG5aeSoMzT38ST_Nn6G8q/s1373/Screen+Shot+2020-08-03+at+11.53.22+am.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="497" data-original-width="1373" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhJSbe8j0ELWS72D08LF5w2mQlQtYfMKzVHW5OwCWTHutjMnLDK8PntAlpE_7haFR85LaYqPS2tXzJqSbq15ZJ8X_UJ8JSHRY3DPi4AsrdkyXcznkc0z-s9hP5cG5aeSoMzT38ST_Nn6G8q/s640/Screen+Shot+2020-08-03+at+11.53.22+am.png" width="640" /></a>
</div>
<div><br /></div>
<div><br /></div>
<div>
18. Both these Springboot applications are using "<a href="https://spring.io/projects/spring-kafka" target="_blank">Spring for Apache Kafka</a>"
</div>
<div><br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjMADw3NmkoUXbxN6YMrX3Y0XkOqSjQ4clv1tw-jLYQ_RbuVfuXQsm5wpgV16iavBA0oZa26jzc5XXhuoC8O0oyu6MDVTH5ok2a7w38224mGl_iSURXGcOtRNSKIaRwVKVrhBgHOhRCGcEp/s1171/Screen+Shot+2020-08-03+at+11.57.22+am.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="702" data-original-width="1171" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjMADw3NmkoUXbxN6YMrX3Y0XkOqSjQ4clv1tw-jLYQ_RbuVfuXQsm5wpgV16iavBA0oZa26jzc5XXhuoC8O0oyu6MDVTH5ok2a7w38224mGl_iSURXGcOtRNSKIaRwVKVrhBgHOhRCGcEp/s640/Screen+Shot+2020-08-03+at+11.57.22+am.png" width="640" /></a>
</div>
<div><br /></div>
<div>
Both Springboot application use a <b>application.yml</b> to bootstrap access to the
Kafka cluster
</div>
<div><br /></div>
<div>
The Producer Springboot application is using a KafkaTemplate to send messages
to our Kafka Topic as shown below.</div>
<pre class="brush: java">
@Controller
@Slf4j
public class TopicMessageController {
private KafkaTemplate<String, String> kafkaTemplate;
@Autowired
public TopicMessageController(KafkaTemplate<String, String> kafkaTemplate) {
this.kafkaTemplate = kafkaTemplate;
}
final private String topicName = "apples-topic";
@GetMapping("/")
public String indexPage (Model model){
model.addAttribute("topicMessageAddSuccess", "N");
return "home";
}
@PostMapping("/addentry")
public String addNewTopicMessage (@RequestParam(value="message") String message, Model model){
kafkaTemplate.send(topicName, message);
log.info("Sent single message: " + message);
model.addAttribute("message", message);
model.addAttribute("topicMessageAddSuccess", "Y");
return "home";
}
}
</pre>
<div><br /></div>
<div>
The Consumer Springboot application is configured with a KafkaListener as
shown below</div>
<pre class="brush: java">
@Controller
@Slf4j
public class TopicConsumerController {
private static ArrayList<String> topicMessages = new ArrayList<String>();
@GetMapping("/")
public String indexPage (Model model){
model.addAttribute("topicMessages", topicMessages);
model.addAttribute("topicMessagesCount", topicMessages.size());
return "home";
}
@KafkaListener(topics = "apples-topic")
public void listen(String message) {
log.info("Received Message: " + message);
topicMessages.add(message);
}
}
</pre>
<div><br /></div>
<div>In this post we did not setup any client authentication against the cluster for the producer or consumer given this was just a demo.</div>
<div><br /></div>
<div><br /></div>
<div><br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiFCDnKuMWHbb6jsVUIMytUwm192N4Vo1L50YE4Am6B4wofZB2RApVCxcLKElxEZn_EBkrgap83f5YGKRj7ERJVs8fOSy_NDH6viFVOOemsrcJYKYNUrR57ol-u0MG1uehpqx9xg6tbqann/s176/Screen+Shot+2020-08-03+at+10.09.22+am.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="176" data-original-width="159" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiFCDnKuMWHbb6jsVUIMytUwm192N4Vo1L50YE4Am6B4wofZB2RApVCxcLKElxEZn_EBkrgap83f5YGKRj7ERJVs8fOSy_NDH6viFVOOemsrcJYKYNUrR57ol-u0MG1uehpqx9xg6tbqann/s0/Screen+Shot+2020-08-03+at+10.09.22+am.png" /></a>
</div>
<div><br /></div>
<div>
<br />
<div><b>More Information</b></div>
<div><br /></div><div>Spring for Apache Kafka</div>
<div><a href="https://spring.io/projects/spring-kafka">https://spring.io/projects/spring-kafka</a></div><div><br /></div><div>CNCF Sanbox projects</div>
<div>
<a href="https://www.cncf.io/sandbox-projects/">https://www.cncf.io/sandbox-projects/</a>
</div>
<div><br /></div>
<div>Strimzi</div>
<div><a href="https://strimzi.io/">https://strimzi.io/</a></div>
</div>
<div class="blogger-post-footer">http://feeds.feedburner.com/TheBlasFromPas</div>Pas Apicellahttp://www.blogger.com/profile/09389663166398991762noreply@blogger.com0tag:blogger.com,1999:blog-6527688743456205256.post-54501935523484035262020-07-17T12:57:00.004+10:002020-07-17T12:57:48.181+10:00Stumbled upon this today : Lens | The Kubernetes IDE Lens is the only IDE you’ll ever need to take control of your Kubernetes
clusters. It is a standalone application for MacOS, Windows and Linux
operating systems. It is open source and free.<br />
<br />
I installed it today and was impressed. Below is some screen shots of new Tanzu Application Service running on my Kubernetes cluster using Lens IDE. Simply point it to your Kube Config for the cluster you wish to examine.<br />
<br />
On Mac SX it's installed as follows<br />
<br />
<span style="color: #3d85c6;">$ brew cask install lens</span><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiuDe2pzE5BMpe-5hGnmvD6LlctiPoaBUZvXqaWoNp8F-CcZdHUlg7WmKldVP-fo0302252WBFX-cBSTb7n6IqB8uhhigbwbh8x8dEhc7X2BosKkAqTDv0bECKuh9XNfoAQtm3FyoSEWZCv/s1600/Screen+Shot+2020-07-17+at+12.54.22+pm.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="965" data-original-width="1600" height="193" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiuDe2pzE5BMpe-5hGnmvD6LlctiPoaBUZvXqaWoNp8F-CcZdHUlg7WmKldVP-fo0302252WBFX-cBSTb7n6IqB8uhhigbwbh8x8dEhc7X2BosKkAqTDv0bECKuh9XNfoAQtm3FyoSEWZCv/s320/Screen+Shot+2020-07-17+at+12.54.22+pm.png" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhPZMgkF3B55X9umKckLOoCFDcJk5bbu8tbHSvlINSTRjoyInWVGrEuzUSWDXVk7UVoCO9tJzKVvAhcUC90YtJaWvjFV5w6TMiTplYXL8ahxjMwDUUHg37GaDmO96MGqC6gxGvD_tJD0bb9/s1600/Screen+Shot+2020-07-17+at+12.54.44+pm.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="968" data-original-width="1600" height="193" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhPZMgkF3B55X9umKckLOoCFDcJk5bbu8tbHSvlINSTRjoyInWVGrEuzUSWDXVk7UVoCO9tJzKVvAhcUC90YtJaWvjFV5w6TMiTplYXL8ahxjMwDUUHg37GaDmO96MGqC6gxGvD_tJD0bb9/s320/Screen+Shot+2020-07-17+at+12.54.44+pm.png" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgEbGEXSdRH00Fl0P_d0-Lnq3phuLCKZlpkaq7hl3z_KQ6KnncHrAcn2XV_LCPRhJiZeXGv0CyR5sLDeEgAopf-GyLdOE3ILKAO-HenXAaL4gIWR-G6n4JmLFH64oMPrefztiLACSe1J0iP/s1600/Screen+Shot+2020-07-17+at+12.55.51+pm.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="970" data-original-width="1600" height="193" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgEbGEXSdRH00Fl0P_d0-Lnq3phuLCKZlpkaq7hl3z_KQ6KnncHrAcn2XV_LCPRhJiZeXGv0CyR5sLDeEgAopf-GyLdOE3ILKAO-HenXAaL4gIWR-G6n4JmLFH64oMPrefztiLACSe1J0iP/s320/Screen+Shot+2020-07-17+at+12.55.51+pm.png" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgdWz9hz2PkgRKj5nD182LGL8OA_9CtvcwR3oi8hgnJ21XsIyB8kQeEJk4IwnaS7SZA0t_AkaL0btOV1urgsUw9bLpTm_yTRru0YFiF-dYculMhl7epEBGTiPkXokobmLnQaDXPkYlUEgBy/s1600/Screen+Shot+2020-07-17+at+12.54.32+pm.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="966" data-original-width="1600" height="193" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgdWz9hz2PkgRKj5nD182LGL8OA_9CtvcwR3oi8hgnJ21XsIyB8kQeEJk4IwnaS7SZA0t_AkaL0btOV1urgsUw9bLpTm_yTRru0YFiF-dYculMhl7epEBGTiPkXokobmLnQaDXPkYlUEgBy/s320/Screen+Shot+2020-07-17+at+12.54.32+pm.png" width="320" /></a></div>
<br />
<br />
<b>More Information</b><br />
<br />
<a href="https://github.com/lensapp/lens">https://github.com/lensapp/lens</a><br />
<br />
<br /><div class="blogger-post-footer">http://feeds.feedburner.com/TheBlasFromPas</div>Pas Apicellahttp://www.blogger.com/profile/09389663166398991762noreply@blogger.com0tag:blogger.com,1999:blog-6527688743456205256.post-60259749641135446912020-07-14T13:50:00.001+10:002020-07-14T14:28:47.095+10:00Spring Data Elasticsearch using Elastic Cloud on Kubernetes (ECK) on VMware Tanzu Kubernetes Grid Integrated Edition (TKGI)VMware Tanzu Kubernetes Grid Integrated Edition (formerly known as
VMware Enterprise PKS) is a Kubernetes-based container solution with
advanced networking, a private container registry, and life cycle
management.<br />
<br />
In this post I show how to get <a href="https://www.elastic.co/guide/en/cloud-on-k8s/current/index.html" target="_blank">Elastic Cloud on Kubernetes</a> (ECK) up and running on VMware Tanzu Kubernetes Grid Integrated Edition and how to access it using a Spring Boot Application using Spring Data Elasticsearch.<br />
<br />
With <a href="https://www.elastic.co/guide/en/cloud-on-k8s/current/index.html" target="_blank">ECK</a>, users now have a seamless way of deploying, managing, and operating the Elastic Stack on Kubernetes.<br />
<br />
If you have a K8s cluster that's all you need to follow along.<br />
<br />
<b>Steps</b><br />
<br />
1. Let's install ECK on our cluster we do that as follows<br />
<br />
<span style="color: #e69138;">Note: There is a 1.1 version as the latest BUT I installing a slightly older one here</span><br />
<br />
<span style="color: #3d85c6;">$ kubectl apply -f https://download.elastic.co/downloads/eck/1.0.1/all-in-one.yaml</span><br />
<br />
2. Make sure the operator is up and running as shown below<br />
<pre class="brush: xml">
$ kubectl get all -n elastic-system
NAME READY STATUS RESTARTS AGE
pod/elastic-operator-0 1/1 Running 0 26d
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/elastic-webhook-server ClusterIP 10.100.200.55 <none> 443/TCP 26d
NAME READY AGE
statefulset.apps/elastic-operator 1/1 26d
</pre>
<br />
3. We can also see a CRD for Elasticsearch as shown below.<br />
<br />
<span style="color: #3d85c6;">elasticsearches.elasticsearch.k8s.elastic.co</span><br />
<pre class="brush: xml">
$ kubectl get crd
NAME CREATED AT
apmservers.apm.k8s.elastic.co 2020-06-17T00:37:32Z
clusterlogsinks.pksapi.io 2020-06-16T23:04:43Z
clustermetricsinks.pksapi.io 2020-06-16T23:04:44Z
elasticsearches.elasticsearch.k8s.elastic.co 2020-06-17T00:37:33Z
kibanas.kibana.k8s.elastic.co 2020-06-17T00:37:34Z
loadbalancers.vmware.com 2020-06-16T22:51:52Z
logsinks.pksapi.io 2020-06-16T23:04:43Z
metricsinks.pksapi.io 2020-06-16T23:04:44Z
nsxerrors.nsx.vmware.com 2020-06-16T22:51:52Z
nsxlbmonitors.vmware.com 2020-06-16T22:51:52Z
nsxlocks.nsx.vmware.com 2020-06-16T22:51:51Z
</pre>
<br />
4. We are now ready to create our first Elasticsearch cluster. To do that create a file YML file as shown below<br />
<br />
<b>create-elastic-cluster-from-operator.yaml</b><br />
<br />
<span style="color: #3d85c6;">apiVersion: elasticsearch.k8s.elastic.co/v1</span><br />
<span style="color: #3d85c6;">kind: Elasticsearch</span><br />
<span style="color: #3d85c6;">metadata:</span><br />
<span style="color: #3d85c6;"> name: quickstart</span><br />
<span style="color: #3d85c6;">spec:</span><br />
<span style="color: #3d85c6;"> version: 7.7.0</span><br />
<span style="color: #3d85c6;"> http:</span><br />
<span style="color: #3d85c6;"> service:</span><br />
<span style="color: #3d85c6;"> spec:</span><br />
<span style="color: #3d85c6;"> type: LoadBalancer # default is ClusterIP</span><br />
<span style="color: #3d85c6;"> tls:</span><br />
<span style="color: #3d85c6;"> selfSignedCertificate:</span><br />
<span style="color: #3d85c6;"> disabled: true</span><br />
<span style="color: #3d85c6;"> nodeSets:</span><br />
<span style="color: #3d85c6;"> - name: default</span><br />
<span style="color: #3d85c6;"> count: 2</span><br />
<span style="color: #3d85c6;"> volumeClaimTemplates:</span><br />
<span style="color: #3d85c6;"> - metadata:</span><br />
<span style="color: #3d85c6;"> name: elasticsearch-data</span><br />
<span style="color: #3d85c6;"> spec:</span><br />
<span style="color: #3d85c6;"> accessModes:</span><br />
<span style="color: #3d85c6;"> - ReadWriteOnce</span><br />
<span style="color: #3d85c6;"> resources:</span><br />
<span style="color: #3d85c6;"> requests:</span><br />
<span style="color: #3d85c6;"> storage: 1Gi</span><br />
<span style="color: #3d85c6;"> config:</span><br />
<span style="color: #3d85c6;"> node.master: true</span><br />
<span style="color: #3d85c6;"> node.data: true</span><br />
<span style="color: #3d85c6;"> node.ingest: true</span><br />
<span style="color: #3d85c6;"> node.store.allow_mmap: false</span><br />
<br />
From the YML a few things to note:<br />
<br />
<ul>
<li>We are creating two pods for our Elasticsearch cluster</li>
<li>We are using a K8s LoadBalancer to expose access to the cluster through HTTP</li>
<li>We are using version 7.7.0 but this is not the latest Elasticsearch version</li>
<li>We have disabled the use of TLS given this is just a demo</li>
</ul>
5. Apply that as shown below.<br />
<br />
<span style="color: #3d85c6;">$ kubectl apply -f create-elastic-cluster-from-operator.yaml</span><br />
<b><br /></b>
6. After about a minute we should have our Elasticsearch cluster running. The following commands show that<br />
<pre class="brush: xml">
$ kubectl get elasticsearch
NAME HEALTH NODES VERSION PHASE AGE
quickstart green 2 7.7.0 Ready 47h
$ kubectl get all -n default
NAME READY STATUS RESTARTS AGE
pod/quickstart-es-default-0 1/1 Running 0 47h
pod/quickstart-es-default-1 1/1 Running 0 47h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.100.200.1 <none> 443/TCP 27d
service/quickstart-es-default ClusterIP None <none> <none> 47h
service/quickstart-es-http LoadBalancer 10.100.200.92 10.195.93.137 9200:30590/TCP 47h
NAME READY AGE
statefulset.apps/quickstart-es-default 2/2 47h
</pre>
<br />
7. Let's deploy a Kibana instance. To do that create a YML as shown below<br />
<br />
<b>create-kibana.yaml</b><br />
<br />
<span style="color: #3d85c6;">apiVersion: kibana.k8s.elastic.co/v1</span><br />
<span style="color: #3d85c6;">kind: Kibana</span><br />
<span style="color: #3d85c6;">metadata:</span><br />
<span style="color: #3d85c6;"> name: kibana-sample</span><br />
<span style="color: #3d85c6;">spec:</span><br />
<span style="color: #3d85c6;"> version: 7.7.0</span><br />
<span style="color: #3d85c6;"> count: 1</span><br />
<span style="color: #3d85c6;"> elasticsearchRef:</span><br />
<span style="color: #3d85c6;"> name: quickstart</span><br />
<span style="color: #3d85c6;"> namespace: default</span><br />
<span style="color: #3d85c6;"> http:</span><br />
<span style="color: #3d85c6;"> service:</span><br />
<span style="color: #3d85c6;"> spec:</span><br />
<span style="color: #3d85c6;"> type: LoadBalancer # default is ClusterIP</span><br />
<br />
8. Apply that as shown below.<br />
<br />
<span style="color: #3d85c6;"><span style="color: #3d85c6;">$ kubectl apply -f </span>create-kibana.yaml</span><br />
<br />
9. To verify everything is up and running we can run a command as follows<br />
<pre class="brush: xml">
$ kubectl get all
NAME READY STATUS RESTARTS AGE
pod/kibana-sample-kb-f8fcb88d5-jdzh5 1/1 Running 0 2d
pod/quickstart-es-default-0 1/1 Running 0 2d
pod/quickstart-es-default-1 1/1 Running 0 2d
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kibana-sample-kb-http LoadBalancer 10.100.200.46 10.195.93.174 5601:32459/TCP 2d
service/kubernetes ClusterIP 10.100.200.1 <none> 443/TCP 27d
service/quickstart-es-default ClusterIP None <none> <none> 2d
service/quickstart-es-http LoadBalancer 10.100.200.92 10.195.93.137 9200:30590/TCP 2d
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/kibana-sample-kb 1/1 1 1 2d
NAME DESIRED CURRENT READY AGE
replicaset.apps/kibana-sample-kb-f8fcb88d5 1 1 1 2d
NAME READY AGE
statefulset.apps/quickstart-es-default 2/2 2d
</pre>
<br />
10. So to access out cluster we will need to obtain the following which we can do using a script as follows. This was tested on Mac OSX<br />
<br />
What do we need?<br />
<br />
<ul>
<li>Elasticsearch password</li>
<li>IP address of the LoadBalancer service we created</li>
</ul>
<br />
<br />
<b>access.sh</b><br />
<br />
<span style="color: #3d85c6;">export PASSWORD=`kubectl get secret quickstart-es-elastic-user -o go-template='{{.data.elastic | base64decode}}'`</span><br />
<span style="color: #3d85c6;">export IP=`kubectl get svc quickstart-es-http -o jsonpath='{.status.loadBalancer.ingress[0].ip}'`</span><br />
<span style="color: #3d85c6;"><br /></span>
<span style="color: #3d85c6;">echo ""</span><br />
<span style="color: #3d85c6;">echo $IP</span><br />
<span style="color: #3d85c6;">echo ""</span><br />
<span style="color: #3d85c6;"><br /></span>
<span style="color: #3d85c6;">curl -u "elastic:$PASSWORD" "http://$IP:9200"</span><br />
<span style="color: #3d85c6;"><br /></span>
<span style="color: #3d85c6;">echo ""</span><br />
<span style="color: #3d85c6;"><br /></span>
<span style="color: #3d85c6;">curl -u "elastic:$PASSWORD" "http://$IP:9200/_cat/health?v"</span><br />
<br />
Output:<br />
<br />
<span style="color: #3d85c6;">10.195.93.137</span><br />
<span style="color: #3d85c6;"><br /></span>
<span style="color: #3d85c6;">{</span><br />
<span style="color: #3d85c6;"> "name" : "quickstart-es-default-1",</span><br />
<span style="color: #3d85c6;"> "cluster_name" : "quickstart",</span><br />
<span style="color: #3d85c6;"> "cluster_uuid" : "Bbpb7Pu7SmaQaCmEY2Er8g",</span><br />
<span style="color: #3d85c6;"> "version" : {</span><br />
<span style="color: #3d85c6;"> "number" : "7.7.0",</span><br />
<span style="color: #3d85c6;"> "build_flavor" : "default",</span><br />
<span style="color: #3d85c6;"> "build_type" : "docker",</span><br />
<span style="color: #3d85c6;"> "build_hash" : "81a1e9eda8e6183f5237786246f6dced26a10eaf",</span><br />
<span style="color: #3d85c6;"> "build_date" : "2020-05-12T02:01:37.602180Z",</span><br />
<span style="color: #3d85c6;"> "build_snapshot" : false,</span><br />
<span style="color: #3d85c6;"> "lucene_version" : "8.5.1",</span><br />
<span style="color: #3d85c6;"> "minimum_wire_compatibility_version" : "6.8.0",</span><br />
<span style="color: #3d85c6;"> "minimum_index_compatibility_version" : "6.0.0-beta1"</span><br />
<span style="color: #3d85c6;"> },</span><br />
<span style="color: #3d85c6;"> "tagline" : "You Know, for Search"</span><br />
<span style="color: #3d85c6;">}</span><br />
<span style="color: #3d85c6;"><br /></span>
<span style="color: #3d85c6;">.....</span><br />
<br />
11. Ideally I would load some data into the Elasticsearch cluster BUT let's do that as part of a sample application using "<a href="https://spring.io/projects/spring-data-elasticsearch" target="_blank">Spring Data Elasticsearch</a>". Clone the demo project as shown below.<br />
<br />
<span style="color: #3d85c6;">$ git clone https://github.com/papicella/boot-elastic-demo.git</span><br />
<span style="color: #3d85c6;">Cloning into 'boot-elastic-demo'...</span><br />
<span style="color: #3d85c6;">remote: Enumerating objects: 36, done.</span><br />
<span style="color: #3d85c6;">remote: Counting objects: 100% (36/36), done.</span><br />
<span style="color: #3d85c6;">remote: Compressing objects: 100% (26/26), done.</span><br />
<span style="color: #3d85c6;">remote: Total 36 (delta 1), reused 36 (delta 1), pack-reused 0</span><br />
<span style="color: #3d85c6;">Unpacking objects: 100% (36/36), done.</span><br />
<br />
12. Edit "<b>./src/main/resources/application.yml</b>" with your details for the Elasticsearch cluster above.<br />
<br />
<span style="color: #3d85c6;">spring:</span><br />
<span style="color: #3d85c6;"> elasticsearch:</span><br />
<span style="color: #3d85c6;"> rest:</span><br />
<span style="color: #3d85c6;"> username: elastic</span><br />
<span style="color: #3d85c6;"> password: {PASSWORD}</span><br />
<span style="color: #3d85c6;"> uris: http://{IP}:9200</span><br />
<br />
13. Package as follows<br />
<br />
<span style="color: #3d85c6;">$ ./mvnw -DskipTests package</span><br />
<br />
14. Run as follows<br />
<br />
<span style="color: #3d85c6;">$ ./mvnw spring-boot:run</span><br />
<span style="color: #3d85c6;"><br /></span>
<span style="color: #3d85c6;">....</span><br />
<span style="color: #3d85c6;">2020-07-14 11:10:11.947 INFO 76260 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path ''</span><br />
<span style="color: #3d85c6;">2020-07-14 11:10:11.954 INFO 76260 --- [ main] c.e.e.demo.BootElasticDemoApplication : Started BootElasticDemoApplication in 2.495 seconds (JVM running for 2.778)</span><br />
<div>
<span style="color: #3d85c6;">....</span></div>
<br />
15. Access application using "<b>http://localhost:8080/</b>"<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhKEEQks0K7xmhc4mZkexW1mNcIGMv9O79dhbP7v1A49brHeeypZY-TKDJys5CiG1wTKJZeWPc7f05djlsQn2Rp8CajXgsB3egkJb2pMNlNHT54TVR3lV1ycQbO5IXQMMjZM_GYR8zWm_3R/s1600/Screen+Shot+2020-07-14+at+11.13.04+am.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="967" data-original-width="1600" height="193" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhKEEQks0K7xmhc4mZkexW1mNcIGMv9O79dhbP7v1A49brHeeypZY-TKDJys5CiG1wTKJZeWPc7f05djlsQn2Rp8CajXgsB3egkJb2pMNlNHT54TVR3lV1ycQbO5IXQMMjZM_GYR8zWm_3R/s320/Screen+Shot+2020-07-14+at+11.13.04+am.png" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjIQKFkMBqxCqfAx33VMNQivnDcHEW_x8y7jnq4c2f1ryuzcXI-DSsDQ9KvComZZNIb25ynVwfnLgKJIcGBVKcR4he5_Z2SwZE7GTnCE-RwYVBpqrkj-eHHtAc0xw_-kWBPATx6avC8SNIx/s1600/Screen+Shot+2020-07-14+at+11.13.23+am.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="521" data-original-width="1600" height="104" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjIQKFkMBqxCqfAx33VMNQivnDcHEW_x8y7jnq4c2f1ryuzcXI-DSsDQ9KvComZZNIb25ynVwfnLgKJIcGBVKcR4he5_Z2SwZE7GTnCE-RwYVBpqrkj-eHHtAc0xw_-kWBPATx6avC8SNIx/s320/Screen+Shot+2020-07-14+at+11.13.23+am.png" width="320" /></a></div>
<br />
<br />
16. If we look at our code we will see the data was loaded into the Elasticsearch cluster using a java class called "<b>LoadData.java</b>". Ideally data should already exist in the cluster but for demo purposes we load some data as part of the Spring Boot Application and clear the data prior to each application run given it's just a demo.<br />
<br />
<span style="color: #3d85c6;">2020-07-14 11:12:33.109 INFO 76277 --- [ main] com.example.elastic.demo.LoadData : Pre loading Car{id='OjThSnMBLjyTRl7lZsDL', make='holden', model='commodore', bodystyles=[BodyStyle{type='2-door'}, BodyStyle{type='4-door'}, BodyStyle{type='5-door'}]}</span><br />
<span style="color: #3d85c6;">2020-07-14 11:12:33.584 INFO 76277 --- [ main] com.example.elastic.demo.LoadData : Pre loading Car{id='OzThSnMBLjyTRl7laMCo', make='holden', model='astra', bodystyles=[BodyStyle{type='2-door'}, BodyStyle{type='4-door'}]}</span><br />
<span style="color: #3d85c6;">2020-07-14 11:12:34.189 INFO 76277 --- [ main] com.example.elastic.demo.LoadData : Pre loading Car{id='PDThSnMBLjyTRl7lasCC', make='nissan', model='skyline', bodystyles=[BodyStyle{type='4-door'}]}</span><br />
<span style="color: #3d85c6;">2020-07-14 11:12:34.744 INFO 76277 --- [ main] com.example.elastic.demo.LoadData : Pre loading Car{id='PTThSnMBLjyTRl7lbMDe', make='nissan', model='pathfinder', bodystyles=[BodyStyle{type='5-door'}]}</span><br />
<span style="color: #3d85c6;">2020-07-14 11:12:35.227 INFO 76277 --- [ main] com.example.elastic.demo.LoadData : Pre loading Car{id='PjThSnMBLjyTRl7lb8AL', make='ford', model='falcon', bodystyles=[BodyStyle{type='4-door'}, BodyStyle{type='5-door'}]}</span><br />
<span style="color: #3d85c6;">2020-07-14 11:12:36.737 INFO 76277 --- [ main] com.example.elastic.demo.LoadData : Pre loading Car{id='QDThSnMBLjyTRl7lcMDu', make='ford', model='territory', bodystyles=[BodyStyle{type='5-door'}]}</span><br />
<span style="color: #3d85c6;">2020-07-14 11:12:37.266 INFO 76277 --- [ main] com.example.elastic.demo.LoadData : Pre loading Car{id='QTThSnMBLjyTRl7ldsDU', make='toyota', model='camry', bodystyles=[BodyStyle{type='4-door'}, BodyStyle{type='5-door'}]}</span><br />
<span style="color: #3d85c6;">2020-07-14 11:12:37.777 INFO 76277 --- [ main] com.example.elastic.demo.LoadData : Pre loading Car{id='QjThSnMBLjyTRl7leMDk', make='toyota', model='corolla', bodystyles=[BodyStyle{type='2-door'}, BodyStyle{type='5-door'}]}</span><br />
<span style="color: #3d85c6;">2020-07-14 11:12:38.285 INFO 76277 --- [ main] com.example.elastic.demo.LoadData : Pre loading Car{id='QzThSnMBLjyTRl7lesDj', make='kia', model='sorento', bodystyles=[BodyStyle{type='5-door'}]}</span><br />
<span style="color: #3d85c6;">2020-07-14 11:12:38.800 INFO 76277 --- [ main] com.example.elastic.demo.LoadData : Pre loading Car{id='RDThSnMBLjyTRl7lfMDg', make='kia', model='sportage', bodystyles=[BodyStyle{type='4-door'}]}</span><br />
<br />
<b>LoadData.java</b><br />
<pre class="brush: java">
package com.example.elastic.demo;
import com.example.elastic.demo.indices.BodyStyle;
import com.example.elastic.demo.indices.Car;
import com.example.elastic.demo.repo.CarRepository;
import org.springframework.boot.CommandLineRunner;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import lombok.extern.slf4j.Slf4j;
import static java.util.Arrays.asList;
@Configuration
@Slf4j
public class LoadData {
@Bean
public CommandLineRunner initElasticsearchData(CarRepository carRepository) {
return args -> {
carRepository.deleteAll();
log.info("Pre loading " + carRepository.save(new Car("holden", "commodore", asList(new BodyStyle("2-door"), new BodyStyle("4-door"), new BodyStyle("5-door")))));
log.info("Pre loading " + carRepository.save(new Car("holden", "astra", asList(new BodyStyle("2-door"), new BodyStyle("4-door")))));
log.info("Pre loading " + carRepository.save(new Car("nissan", "skyline", asList(new BodyStyle("4-door")))));
log.info("Pre loading " + carRepository.save(new Car("nissan", "pathfinder", asList(new BodyStyle("5-door")))));
log.info("Pre loading " + carRepository.save(new Car("ford", "falcon", asList(new BodyStyle("4-door"), new BodyStyle("5-door")))));
log.info("Pre loading " + carRepository.save(new Car("ford", "territory", asList(new BodyStyle("5-door")))));
log.info("Pre loading " + carRepository.save(new Car("toyota", "camry", asList(new BodyStyle("4-door"), new BodyStyle("5-door")))));
log.info("Pre loading " + carRepository.save(new Car("toyota", "corolla", asList(new BodyStyle("2-door"), new BodyStyle("5-door")))));
log.info("Pre loading " + carRepository.save(new Car("kia", "sorento", asList(new BodyStyle("5-door")))));
log.info("Pre loading " + carRepository.save(new Car("kia", "sportage", asList(new BodyStyle("4-door")))));
};
}
}
</pre>
<br />
17. Our CarRepository interface is defined as follows<br />
<br />
<b>CarRepository.java</b><br />
<pre class="brush: java">
package com.example.elastic.demo.repo;
import com.example.elastic.demo.indices.Car;
import org.springframework.data.domain.Page;
import org.springframework.data.domain.Pageable;
import org.springframework.data.elasticsearch.repository.ElasticsearchRepository;
import org.springframework.stereotype.Repository;
@Repository
public interface CarRepository extends ElasticsearchRepository <Car, String> {
Page<Car> findByMakeContaining(String make, Pageable page);
}
</pre>
<br />
18. So let's also via this data using "<b>curl</b>" and Kibana as shown below.<br />
<br />
<span style="color: #3d85c6;">curl -X GET -u "elastic:{PASSWORD}" "http://{IP}:9200/vehicle/_search?pretty" -H 'Content-Type: application/json' -d'</span><br />
<span style="color: #3d85c6;">{</span><br />
<span style="color: #3d85c6;"> "query": { "match_all": {} },</span><br />
<span style="color: #3d85c6;"> "sort": [</span><br />
<span style="color: #3d85c6;"> { "_id": "asc" }</span><br />
<span style="color: #3d85c6;"> ]</span><br />
<span style="color: #3d85c6;">}</span><br />
<span style="color: #3d85c6;">'</span><br />
<br />
Output:<br />
<br />
<span style="color: #3d85c6;">{</span><br />
<span style="color: #3d85c6;"> "took" : 2,</span><br />
<span style="color: #3d85c6;"> "timed_out" : false,</span><br />
<span style="color: #3d85c6;"> "_shards" : {</span><br />
<span style="color: #3d85c6;"> "total" : 1,</span><br />
<span style="color: #3d85c6;"> "successful" : 1,</span><br />
<span style="color: #3d85c6;"> "skipped" : 0,</span><br />
<span style="color: #3d85c6;"> "failed" : 0</span><br />
<span style="color: #3d85c6;"> },</span><br />
<span style="color: #3d85c6;"> "hits" : {</span><br />
<span style="color: #3d85c6;"> "total" : {</span><br />
<span style="color: #3d85c6;"> "value" : 10,</span><br />
<span style="color: #3d85c6;"> "relation" : "eq"</span><br />
<span style="color: #3d85c6;"> },</span><br />
<span style="color: #3d85c6;"> "max_score" : null,</span><br />
<span style="color: #3d85c6;"> "hits" : [</span><br />
<span style="color: #3d85c6;"> {</span><br />
<span style="color: #3d85c6;"> "_index" : "vehicle",</span><br />
<span style="color: #3d85c6;"> "_type" : "_doc",</span><br />
<span style="color: #3d85c6;"> "_id" : "OjThSnMBLjyTRl7lZsDL",</span><br />
<span style="color: #3d85c6;"> "_score" : null,</span><br />
<span style="color: #3d85c6;"> "_source" : {</span><br />
<span style="color: #3d85c6;"> "_class" : "com.example.elastic.demo.indices.Car",</span><br />
<span style="color: #3d85c6;"> "make" : "holden",</span><br />
<span style="color: #3d85c6;"> "model" : "commodore",</span><br />
<span style="color: #3d85c6;"> "bodystyles" : [</span><br />
<span style="color: #3d85c6;"> {</span><br />
<span style="color: #3d85c6;"> "type" : "2-door"</span><br />
<span style="color: #3d85c6;"> },</span><br />
<span style="color: #3d85c6;"> {</span><br />
<span style="color: #3d85c6;"> "type" : "4-door"</span><br />
<span style="color: #3d85c6;"> },</span><br />
<span style="color: #3d85c6;"> {</span><br />
<span style="color: #3d85c6;"> "type" : "5-door"</span><br />
<span style="color: #3d85c6;"> }</span><br />
<span style="color: #3d85c6;"> ]</span><br />
<span style="color: #3d85c6;"> },</span><br />
<span style="color: #3d85c6;"> "sort" : [</span><br />
<span style="color: #3d85c6;"> "OjThSnMBLjyTRl7lZsDL"</span><br />
<span style="color: #3d85c6;"> ]</span><br />
<span style="color: #3d85c6;"> },</span><br />
<span style="color: #3d85c6;"> {</span><br />
<span style="color: #3d85c6;"> "_index" : "vehicle",</span><br />
<span style="color: #3d85c6;"> "_type" : "_doc",</span><br />
<span style="color: #3d85c6;"> "_id" : "OzThSnMBLjyTRl7laMCo",</span><br />
<span style="color: #3d85c6;"> "_score" : null,</span><br />
<span style="color: #3d85c6;"> "_source" : {</span><br />
<span style="color: #3d85c6;"> "_class" : "com.example.elastic.demo.indices.Car",</span><br />
<span style="color: #3d85c6;"> "make" : "holden",</span><br />
<span style="color: #3d85c6;"> "model" : "astra",</span><br />
<span style="color: #3d85c6;"> "bodystyles" : [</span><br />
<span style="color: #3d85c6;"> {</span><br />
<span style="color: #3d85c6;"> "type" : "2-door"</span><br />
<span style="color: #3d85c6;"> },</span><br />
<span style="color: #3d85c6;"> {</span><br />
<span style="color: #3d85c6;"> "type" : "4-door"</span><br />
<span style="color: #3d85c6;"> }</span><br />
<span style="color: #3d85c6;"> ]</span><br />
<span style="color: #3d85c6;"> },</span><br />
<span style="color: #3d85c6;"> "sort" : [</span><br />
<span style="color: #3d85c6;"> "OzThSnMBLjyTRl7laMCo"</span><br />
<span style="color: #3d85c6;"> ]</span><br />
<span style="color: #3d85c6;"> },</span><br />
<span style="color: #3d85c6;"> {</span><br />
<span style="color: #3d85c6;"> "_index" : "vehicle",</span><br />
<span style="color: #3d85c6;"> "_type" : "_doc",</span><br />
<span style="color: #3d85c6;"> "_id" : "PDThSnMBLjyTRl7lasCC",</span><br />
<span style="color: #3d85c6;"> "_score" : null,</span><br />
<span style="color: #3d85c6;"> "_source" : {</span><br />
<span style="color: #3d85c6;"> "_class" : "com.example.elastic.demo.indices.Car",</span><br />
<span style="color: #3d85c6;"> "make" : "nissan",</span><br />
<span style="color: #3d85c6;"> "model" : "skyline",</span><br />
<span style="color: #3d85c6;"> "bodystyles" : [</span><br />
<span style="color: #3d85c6;"> {</span><br />
<span style="color: #3d85c6;"> "type" : "4-door"</span><br />
<span style="color: #3d85c6;"> }</span><br />
<span style="color: #3d85c6;"> ]</span><br />
<span style="color: #3d85c6;"> },</span><br />
<span style="color: #3d85c6;"> "sort" : [</span><br />
<span style="color: #3d85c6;"> "PDThSnMBLjyTRl7lasCC"</span><br />
<span style="color: #3d85c6;"> ]</span><br />
<span style="color: #3d85c6;"> },</span><br />
<span style="color: #3d85c6;"> {</span><br />
<span style="color: #3d85c6;"> "_index" : "vehicle",</span><br />
<span style="color: #3d85c6;"> "_type" : "_doc",</span><br />
<span style="color: #3d85c6;"> "_id" : "PTThSnMBLjyTRl7lbMDe",</span><br />
<span style="color: #3d85c6;"> "_score" : null,</span><br />
<span style="color: #3d85c6;"> "_source" : {</span><br />
<span style="color: #3d85c6;"> "_class" : "com.example.elastic.demo.indices.Car",</span><br />
<span style="color: #3d85c6;"> "make" : "nissan",</span><br />
<span style="color: #3d85c6;"> "model" : "pathfinder",</span><br />
<span style="color: #3d85c6;"> "bodystyles" : [</span><br />
<span style="color: #3d85c6;"> {</span><br />
<span style="color: #3d85c6;"> "type" : "5-door"</span><br />
<span style="color: #3d85c6;"> }</span><br />
<span style="color: #3d85c6;"> ]</span><br />
<span style="color: #3d85c6;"> },</span><br />
<span style="color: #3d85c6;"> "sort" : [</span><br />
<span style="color: #3d85c6;"> "PTThSnMBLjyTRl7lbMDe"</span><br />
<span style="color: #3d85c6;"> ]</span><br />
<span style="color: #3d85c6;"> },</span><br />
<span style="color: #3d85c6;"> {</span><br />
<span style="color: #3d85c6;"> "_index" : "vehicle",</span><br />
<span style="color: #3d85c6;"> "_type" : "_doc",</span><br />
<span style="color: #3d85c6;"> "_id" : "PjThSnMBLjyTRl7lb8AL",</span><br />
<span style="color: #3d85c6;"> "_score" : null,</span><br />
<span style="color: #3d85c6;"> "_source" : {</span><br />
<span style="color: #3d85c6;"> "_class" : "com.example.elastic.demo.indices.Car",</span><br />
<span style="color: #3d85c6;"> "make" : "ford",</span><br />
<span style="color: #3d85c6;"> "model" : "falcon",</span><br />
<span style="color: #3d85c6;"> "bodystyles" : [</span><br />
<span style="color: #3d85c6;"> {</span><br />
<span style="color: #3d85c6;"> "type" : "4-door"</span><br />
<span style="color: #3d85c6;"> },</span><br />
<span style="color: #3d85c6;"> {</span><br />
<span style="color: #3d85c6;"> "type" : "5-door"</span><br />
<span style="color: #3d85c6;"> }</span><br />
<span style="color: #3d85c6;"> ]</span><br />
<span style="color: #3d85c6;"> },</span><br />
<span style="color: #3d85c6;"> "sort" : [</span><br />
<span style="color: #3d85c6;"> "PjThSnMBLjyTRl7lb8AL"</span><br />
<span style="color: #3d85c6;"> ]</span><br />
<span style="color: #3d85c6;"> },</span><br />
<span style="color: #3d85c6;"> {</span><br />
<span style="color: #3d85c6;"> "_index" : "vehicle",</span><br />
<span style="color: #3d85c6;"> "_type" : "_doc",</span><br />
<span style="color: #3d85c6;"> "_id" : "QDThSnMBLjyTRl7lcMDu",</span><br />
<span style="color: #3d85c6;"> "_score" : null,</span><br />
<span style="color: #3d85c6;"> "_source" : {</span><br />
<span style="color: #3d85c6;"> "_class" : "com.example.elastic.demo.indices.Car",</span><br />
<span style="color: #3d85c6;"> "make" : "ford",</span><br />
<span style="color: #3d85c6;"> "model" : "territory",</span><br />
<span style="color: #3d85c6;"> "bodystyles" : [</span><br />
<span style="color: #3d85c6;"> {</span><br />
<span style="color: #3d85c6;"> "type" : "5-door"</span><br />
<span style="color: #3d85c6;"> }</span><br />
<span style="color: #3d85c6;"> ]</span><br />
<span style="color: #3d85c6;"> },</span><br />
<span style="color: #3d85c6;"> "sort" : [</span><br />
<span style="color: #3d85c6;"> "QDThSnMBLjyTRl7lcMDu"</span><br />
<span style="color: #3d85c6;"> ]</span><br />
<span style="color: #3d85c6;"> },</span><br />
<span style="color: #3d85c6;"> {</span><br />
<span style="color: #3d85c6;"> "_index" : "vehicle",</span><br />
<span style="color: #3d85c6;"> "_type" : "_doc",</span><br />
<span style="color: #3d85c6;"> "_id" : "QTThSnMBLjyTRl7ldsDU",</span><br />
<span style="color: #3d85c6;"> "_score" : null,</span><br />
<span style="color: #3d85c6;"> "_source" : {</span><br />
<span style="color: #3d85c6;"> "_class" : "com.example.elastic.demo.indices.Car",</span><br />
<span style="color: #3d85c6;"> "make" : "toyota",</span><br />
<span style="color: #3d85c6;"> "model" : "camry",</span><br />
<span style="color: #3d85c6;"> "bodystyles" : [</span><br />
<span style="color: #3d85c6;"> {</span><br />
<span style="color: #3d85c6;"> "type" : "4-door"</span><br />
<span style="color: #3d85c6;"> },</span><br />
<span style="color: #3d85c6;"> {</span><br />
<span style="color: #3d85c6;"> "type" : "5-door"</span><br />
<span style="color: #3d85c6;"> }</span><br />
<span style="color: #3d85c6;"> ]</span><br />
<span style="color: #3d85c6;"> },</span><br />
<span style="color: #3d85c6;"> "sort" : [</span><br />
<span style="color: #3d85c6;"> "QTThSnMBLjyTRl7ldsDU"</span><br />
<span style="color: #3d85c6;"> ]</span><br />
<span style="color: #3d85c6;"> },</span><br />
<span style="color: #3d85c6;"> {</span><br />
<span style="color: #3d85c6;"> "_index" : "vehicle",</span><br />
<span style="color: #3d85c6;"> "_type" : "_doc",</span><br />
<span style="color: #3d85c6;"> "_id" : "QjThSnMBLjyTRl7leMDk",</span><br />
<span style="color: #3d85c6;"> "_score" : null,</span><br />
<span style="color: #3d85c6;"> "_source" : {</span><br />
<span style="color: #3d85c6;"> "_class" : "com.example.elastic.demo.indices.Car",</span><br />
<span style="color: #3d85c6;"> "make" : "toyota",</span><br />
<span style="color: #3d85c6;"> "model" : "corolla",</span><br />
<span style="color: #3d85c6;"> "bodystyles" : [</span><br />
<span style="color: #3d85c6;"> {</span><br />
<span style="color: #3d85c6;"> "type" : "2-door"</span><br />
<span style="color: #3d85c6;"> },</span><br />
<span style="color: #3d85c6;"> {</span><br />
<span style="color: #3d85c6;"> "type" : "5-door"</span><br />
<span style="color: #3d85c6;"> }</span><br />
<span style="color: #3d85c6;"> ]</span><br />
<span style="color: #3d85c6;"> },</span><br />
<span style="color: #3d85c6;"> "sort" : [</span><br />
<span style="color: #3d85c6;"> "QjThSnMBLjyTRl7leMDk"</span><br />
<span style="color: #3d85c6;"> ]</span><br />
<span style="color: #3d85c6;"> },</span><br />
<span style="color: #3d85c6;"> {</span><br />
<span style="color: #3d85c6;"> "_index" : "vehicle",</span><br />
<span style="color: #3d85c6;"> "_type" : "_doc",</span><br />
<span style="color: #3d85c6;"> "_id" : "QzThSnMBLjyTRl7lesDj",</span><br />
<span style="color: #3d85c6;"> "_score" : null,</span><br />
<span style="color: #3d85c6;"> "_source" : {</span><br />
<span style="color: #3d85c6;"> "_class" : "com.example.elastic.demo.indices.Car",</span><br />
<span style="color: #3d85c6;"> "make" : "kia",</span><br />
<span style="color: #3d85c6;"> "model" : "sorento",</span><br />
<span style="color: #3d85c6;"> "bodystyles" : [</span><br />
<span style="color: #3d85c6;"> {</span><br />
<span style="color: #3d85c6;"> "type" : "5-door"</span><br />
<span style="color: #3d85c6;"> }</span><br />
<span style="color: #3d85c6;"> ]</span><br />
<span style="color: #3d85c6;"> },</span><br />
<span style="color: #3d85c6;"> "sort" : [</span><br />
<span style="color: #3d85c6;"> "QzThSnMBLjyTRl7lesDj"</span><br />
<span style="color: #3d85c6;"> ]</span><br />
<span style="color: #3d85c6;"> },</span><br />
<span style="color: #3d85c6;"> {</span><br />
<span style="color: #3d85c6;"> "_index" : "vehicle",</span><br />
<span style="color: #3d85c6;"> "_type" : "_doc",</span><br />
<span style="color: #3d85c6;"> "_id" : "RDThSnMBLjyTRl7lfMDg",</span><br />
<span style="color: #3d85c6;"> "_score" : null,</span><br />
<span style="color: #3d85c6;"> "_source" : {</span><br />
<span style="color: #3d85c6;"> "_class" : "com.example.elastic.demo.indices.Car",</span><br />
<span style="color: #3d85c6;"> "make" : "kia",</span><br />
<span style="color: #3d85c6;"> "model" : "sportage",</span><br />
<span style="color: #3d85c6;"> "bodystyles" : [</span><br />
<span style="color: #3d85c6;"> {</span><br />
<span style="color: #3d85c6;"> "type" : "4-door"</span><br />
<span style="color: #3d85c6;"> }</span><br />
<span style="color: #3d85c6;"> ]</span><br />
<span style="color: #3d85c6;"> },</span><br />
<span style="color: #3d85c6;"> "sort" : [</span><br />
<span style="color: #3d85c6;"> "RDThSnMBLjyTRl7lfMDg"</span><br />
<span style="color: #3d85c6;"> ]</span><br />
<span style="color: #3d85c6;"> }</span><br />
<span style="color: #3d85c6;"> ]</span><br />
<span style="color: #3d85c6;"> }</span><br />
<span style="color: #3d85c6;">}</span><br />
<br />
<b>Kibana</b><br />
<br />
Obtain Kibana HTTP IP as shown below and login using username "<b>elastic</b>" and password we obtained previously.<br />
<br />
<span style="color: #3d85c6;">$ kubectl get svc kibana-sample-kb-http -o jsonpath='{.status.loadBalancer.ingress[0].ip}'</span><br />
<span style="color: #3d85c6;">10.195.93.174</span><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiAAT6TjTV9S9oT9J8SOXldyRTfoK4EP-6ggcjLEMiI4O4kU7dNX7FAbRdBwSPXQPopyOtD2HxcPs8hgqR94h9XSzcFuFN0kxTiI5WXBhpaEaOUmvyZYNS4eLVrTLR04tDS5huw3YOiolcO/s1600/Screen+Shot+2020-07-14+at+11.35.09+am.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="979" data-original-width="1600" height="195" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiAAT6TjTV9S9oT9J8SOXldyRTfoK4EP-6ggcjLEMiI4O4kU7dNX7FAbRdBwSPXQPopyOtD2HxcPs8hgqR94h9XSzcFuFN0kxTiI5WXBhpaEaOUmvyZYNS4eLVrTLR04tDS5huw3YOiolcO/s320/Screen+Shot+2020-07-14+at+11.35.09+am.png" width="320" /></a></div>
<br />
<br />
<br />
Finally maybe you want to deploy the application to Kubernetes. To do that take a look at <a href="https://buildpacks.io/" target="_blank">Cloud Native Buildpacks</a> CNCF project and/or <a href="https://docs.pivotal.io/build-service/0-2-0/index.html" target="_blank">Tanzu Build Service</a> to turn your code into a Container Image stored in a registry.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhq9Oa9WCe0M5MJ8KXaOcptWn2pWhdGAAs7YWDEYngUWy2lWH9Q8w2f33zdkVAeko7hRSoUKxkWTo008YgPnQ-pmtMzO1MG8rei5LDvWcPml5eqL8d-DJa578lJpS4nOVtvR4mAtBOf-BDP/s1600/Screen+Shot+2020-07-14+at+1.45.00+pm.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="470" data-original-width="1426" height="105" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhq9Oa9WCe0M5MJ8KXaOcptWn2pWhdGAAs7YWDEYngUWy2lWH9Q8w2f33zdkVAeko7hRSoUKxkWTo008YgPnQ-pmtMzO1MG8rei5LDvWcPml5eqL8d-DJa578lJpS4nOVtvR4mAtBOf-BDP/s320/Screen+Shot+2020-07-14+at+1.45.00+pm.png" width="320" /></a></div>
<br />
<br />
<b>More Information</b><br />
<br />
Spring Data Elasticsearch<br />
<a href="https://spring.io/projects/spring-data-elasticsearch">https://spring.io/projects/spring-data-elasticsearch</a><br />
<br />
VMware Tanzu Kubernetes Grid Integrated Edition Documentation<br />
<a href="https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid-Integrated-Edition/index.html">https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid-Integrated-Edition/index.html</a><div class="blogger-post-footer">http://feeds.feedburner.com/TheBlasFromPas</div>Pas Apicellahttp://www.blogger.com/profile/09389663166398991762noreply@blogger.com0tag:blogger.com,1999:blog-6527688743456205256.post-11242268265446670002020-07-10T14:22:00.002+10:002020-07-10T16:56:35.049+10:00Multi-Factor Authentication (MFA) using OKTA with Spring Boot and Tanzu Application ServiceRecently I was asked to build a quick demo showing how to use MFA with OKTA and Spring Boot application running on <a href="https://tanzu.vmware.com/application-service" target="_blank">Tanzu Application Service</a>. Here is the demo application plus how to setup and run this yourself.<br />
<br />
<b>Steps</b><br />
<br />
1. Clone the existing repo as shown below<br />
<br />
<span style="color: #3d85c6;">$ git clone https://github.com/papicella/mfa-boot-fsi</span><br />
<span style="color: #3d85c6;">Cloning into 'mfa-boot-fsi'...</span><br />
<span style="color: #3d85c6;">remote: Enumerating objects: 47, done.</span><br />
<span style="color: #3d85c6;">remote: Counting objects: 100% (47/47), done.</span><br />
<span style="color: #3d85c6;">remote: Compressing objects: 100% (31/31), done.</span><br />
<span style="color: #3d85c6;">remote: Total 47 (delta 2), reused 47 (delta 2), pack-reused 0</span><br />
<span style="color: #3d85c6;">Unpacking objects: 100% (47/47), done.</span><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhBivZWgcyC6vhbMaJo6hdV3bXvbwPbsD0SoyIYXjYRr84RFelt6q025OHWy3XSaHgbcToVXclJvLuYs42XjmioCEtHBVb64jsgbl-xPNZHcmUgMrHDVJUNrtcKZm-pAbhlHhzPtuP50LAT/s1600/Screen+Shot+2020-07-10+at+2.19.39+pm.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="652" data-original-width="1600" height="130" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhBivZWgcyC6vhbMaJo6hdV3bXvbwPbsD0SoyIYXjYRr84RFelt6q025OHWy3XSaHgbcToVXclJvLuYs42XjmioCEtHBVb64jsgbl-xPNZHcmUgMrHDVJUNrtcKZm-pAbhlHhzPtuP50LAT/s320/Screen+Shot+2020-07-10+at+2.19.39+pm.png" width="320" /></a></div>
<br />
<br />
2. Create a free account of <a href="https://developer.okta.com/">https://developer.okta.com/</a><br />
<br />
Once created login to the dev account. Your account URL will look like something as follows<br />
<br />
<b>https://dev-{ID}-admin.okta.com</b><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEimFYLXR9XoPMIrRmyR8PD_jCXbe4x4ksJQc2Cg4DwGsx-SdOl5vI2YJXpL5-vuOeEPWX5U9Q-Vb-YAVsANeOuL4LRwldNahkRiOZ5UYp8nvDDE0QiZ9FDgxpR_rWgdn-gQnEfzV_JNDJJf/s1600/Screen+Shot+2020-07-10+at+1.06.27+pm.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="463" data-original-width="1046" height="141" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEimFYLXR9XoPMIrRmyR8PD_jCXbe4x4ksJQc2Cg4DwGsx-SdOl5vI2YJXpL5-vuOeEPWX5U9Q-Vb-YAVsANeOuL4LRwldNahkRiOZ5UYp8nvDDE0QiZ9FDgxpR_rWgdn-gQnEfzV_JNDJJf/s320/Screen+Shot+2020-07-10+at+1.06.27+pm.png" width="320" /></a></div>
<br />
<br />
3. You will need your default authorization server settings. From the top menu in the <b>developer.okta.com</b> dashboard, go to <strong>API</strong> -> <strong>Authorization Servers</strong> and click on the <strong>default</strong> server<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhel0E8BpUzoOsBT6nsJ4jsArBShhiAhGzY-KbhVq-1KxHsrBTnTaN7Sb-Dc9BYy4wzHUL0sARMHwT4EXXuba1buzcNsJQnZ3CQfnq2Vop5EEIpGtCbiQzLGrDpAlcFbEqMf2dtthwcrNjY/s1600/Screen+Shot+2020-07-10+at+1.20.36+pm.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="741" data-original-width="1072" height="221" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhel0E8BpUzoOsBT6nsJ4jsArBShhiAhGzY-KbhVq-1KxHsrBTnTaN7Sb-Dc9BYy4wzHUL0sARMHwT4EXXuba1buzcNsJQnZ3CQfnq2Vop5EEIpGtCbiQzLGrDpAlcFbEqMf2dtthwcrNjY/s320/Screen+Shot+2020-07-10+at+1.20.36+pm.png" width="320" /></a></div>
<br />
You will need this data shortly. Image above is an example those details won't work for your own setup.<br />
<br />
4. From the top menu, go to <strong>Applications</strong> and click the <strong>Add Application</strong> button. Click on the <strong>Web</strong> button and click <strong>Next. </strong>Name your app whatever you like. I named mine "<b>pas-okta-springapp"</b>. Otherwise the default settings are fine. Click <strong>Done</strong>.<br />
<br />
From this screen shot you can see that the default's refer to localhost which for DEV purposes is fine.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjpF6yOsdzqItoCO5AbQGqOuS9ofQOYBo5Zh6Q3jZv0wWTuxTpK3KxlGNlabgyUMNMNv2QqV2IPRNBFd8g8nI08LNdtjLcyladYeSolnFXCV4tOOp_ODQiR-o2cV-Nyvt_4oPLGQ7eIiUt9/s1600/Screen+Shot+2020-07-10+at+1.26.30+pm.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1027" data-original-width="1039" height="316" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjpF6yOsdzqItoCO5AbQGqOuS9ofQOYBo5Zh6Q3jZv0wWTuxTpK3KxlGNlabgyUMNMNv2QqV2IPRNBFd8g8nI08LNdtjLcyladYeSolnFXCV4tOOp_ODQiR-o2cV-Nyvt_4oPLGQ7eIiUt9/s320/Screen+Shot+2020-07-10+at+1.26.30+pm.png" width="320" /></a></div>
<br />
You will need the <strong>Client ID</strong> and <strong>Client secret</strong> from the final screen so make a note of these<br />
<br />
5. Edit the "<b>./mfa-boot-fsi/src/main/resources/application-DEV.yml</b>" to include the details as per #3 and #4 above.<br />
<br />
You will need to edit<br />
<br />
<ul>
<li>issuer</li>
<li>client-id</li>
<li>client-secret</li>
</ul>
<br />
<br />
<b>application-DEV.yaml</b><br />
<br />
<span style="color: #3d85c6;">spring:</span><br />
<span style="color: #3d85c6;"> security:</span><br />
<span style="color: #3d85c6;"> oauth2:</span><br />
<span style="color: #3d85c6;"> client:</span><br />
<span style="color: #3d85c6;"> provider:</span><br />
<span style="color: #3d85c6;"> okta:</span><br />
<span style="color: #3d85c6;"> user-name-attribute: email</span><br />
<span style="color: #3d85c6;"><br /></span>
<span style="color: #3d85c6;">okta:</span><br />
<span style="color: #3d85c6;"> oauth2:</span><br />
<span style="color: #3d85c6;"> issuer: https://dev-213269.okta.com/oauth2/default</span><br />
<span style="color: #3d85c6;"> redirect-uri: /authorization-code/callback</span><br />
<span style="color: #3d85c6;"> scopes:</span><br />
<span style="color: #3d85c6;"> - profile</span><br />
<span style="color: #3d85c6;"> - email</span><br />
<span style="color: #3d85c6;"> - openid</span><br />
<span style="color: #3d85c6;"> client-id: ....</span><br />
<span style="color: #3d85c6;"> client-secret: ....</span><br />
<br />
6. In order to pick up this application-DEV.yaml we have to set the spring profile correctly. That can be done using a JVM property as follows.<br />
<br />
<b>-Dspring.profiles.active=DEV</b><br />
<br />
In my example I use IntelliJ IDEA so I set it on the run configurations dialog as follows<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi7utuy6Xbha_oVcJFQ9T3xzJKMtJGd4KGQ1lbRk6CcpJV4Sji2dfnm59l22ymQPyrHrm_PuHgfr6lCATbYgOh5-8gCrfJ2EnGZjoBo8Ko4so0ij9mbhU2KDzBz7OYtEVLIXohXaR1lDl1d/s1600/Screen+Shot+2020-07-10+at+1.29.03+pm.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="678" data-original-width="1065" height="203" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi7utuy6Xbha_oVcJFQ9T3xzJKMtJGd4KGQ1lbRk6CcpJV4Sji2dfnm59l22ymQPyrHrm_PuHgfr6lCATbYgOh5-8gCrfJ2EnGZjoBo8Ko4so0ij9mbhU2KDzBz7OYtEVLIXohXaR1lDl1d/s320/Screen+Shot+2020-07-10+at+1.29.03+pm.png" width="320" /></a></div>
<br />
<br />
7. Finally let's setup MFA which we do as follows by switching to classic UI as shown below<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEguihesJOTBjCyNYbATBL3ykfuqfGXlpAHj528O48vvUZRjZ8WjSOE1Ld5fCqZ52Aacel5Qp-c8cmPVozYE51sdM3U6I8R6b2WMn5XRoRsI33DvGsBcf6i9AD6ZoZF6OzwXOzxFeaxm8GD-/s1600/Screen+Shot+2020-07-10+at+1.43.24+pm.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="217" data-original-width="1030" height="67" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEguihesJOTBjCyNYbATBL3ykfuqfGXlpAHj528O48vvUZRjZ8WjSOE1Ld5fCqZ52Aacel5Qp-c8cmPVozYE51sdM3U6I8R6b2WMn5XRoRsI33DvGsBcf6i9AD6ZoZF6OzwXOzxFeaxm8GD-/s320/Screen+Shot+2020-07-10+at+1.43.24+pm.png" width="320" /></a></div>
<br />
<br />
8. Click on <b>Security</b> -> <b>Multifactor </b>and setup another Multifactor policy. In the screen shot below I select "Email Policy" and make sure it is "Required" along with the default policy<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh-Omd0-yw6yhQcvLon9dSjPUEOMsgD-K5ntaBzu66RGGuPQMzmXuWc6iFRpAYdtyyaQ8nZ5AIoqpkYcUK1UUrVjIJ5OaLpAABNwUoiS3ZknXLsKDp_t389MAC2i1xVx3K-LiKwV0LLQzRb/s1600/Screen+Shot+2020-07-10+at+1.46.38+pm.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="778" data-original-width="1143" height="217" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh-Omd0-yw6yhQcvLon9dSjPUEOMsgD-K5ntaBzu66RGGuPQMzmXuWc6iFRpAYdtyyaQ8nZ5AIoqpkYcUK1UUrVjIJ5OaLpAABNwUoiS3ZknXLsKDp_t389MAC2i1xVx3K-LiKwV0LLQzRb/s320/Screen+Shot+2020-07-10+at+1.46.38+pm.png" width="320" /></a></div>
<br />
<br />
9. Now run the application making sure you set the spring active profile to DEV.<br />
<br />
<span style="color: #3d85c6;">...</span><br />
<span style="color: #3d85c6;">2020-07-10 13:34:57.528 INFO 55990 --- [ restartedMain] pas.apa.apj.mfa.demo.DemoApplication : The following profiles are active: DEV</span><br />
<span style="color: #3d85c6;">...</span><br />
<br />
10. Navigate to <b>http://localhost:8080/</b><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiRhKYBxqRnJLr-mkeYDBYZyfEaLK0_TSof8f8Kwrqe3Ihc7XWFM6AMSouK1c5d5wYNfFRqPn87CouGAS28N48uXh1BQjiKAoNJFipA2kk_Ou64QvvT_yME5Np7Y_iBTUBKpiQ65tzmyCki/s1600/Screen+Shot+2020-07-10+at+1.39.53+pm.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="411" data-original-width="1594" height="82" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiRhKYBxqRnJLr-mkeYDBYZyfEaLK0_TSof8f8Kwrqe3Ihc7XWFM6AMSouK1c5d5wYNfFRqPn87CouGAS28N48uXh1BQjiKAoNJFipA2kk_Ou64QvvT_yME5Np7Y_iBTUBKpiQ65tzmyCki/s320/Screen+Shot+2020-07-10+at+1.39.53+pm.png" width="320" /></a></div>
<br />
<br />
11. Click on the "<b>Login</b>" button<br />
<br />
Verify you are taken to the default OKTA login page<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiBc6CxQlrqdUwxyy8BPaQpdTj-e9q15wR-pUS510fJ4F4eFLdAOBzLMXR9ejOZKbh5K8eaHnlhvMT6FJ1WwYZiLf3UYsc5HXMF2FTnNBPt6wtbrQaBLkzoVFQMZM2rlrMIiF4oMuUAzbOY/s1600/Screen+Shot+2020-07-10+at+1.40.09+pm.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="778" data-original-width="836" height="297" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiBc6CxQlrqdUwxyy8BPaQpdTj-e9q15wR-pUS510fJ4F4eFLdAOBzLMXR9ejOZKbh5K8eaHnlhvMT6FJ1WwYZiLf3UYsc5HXMF2FTnNBPt6wtbrQaBLkzoVFQMZM2rlrMIiF4oMuUAzbOY/s320/Screen+Shot+2020-07-10+at+1.40.09+pm.png" width="320" /></a></div>
<br />
12. Once logged in the second factor should then ask for a verification code to be sent to your email. Press the "<b>Send me the code</b>" button<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhI8aW8yb5HN5AT6h406o7Mr1_pYC2zR23mwlnp-bnt2dyOy1X2BuFPwLIwCE_WTJEDxogCiKZNJU5C1rvw8_v4KIKN6Mtrils-d1WM4M2H2ciaQrua59-Q3nbTus3-v2JGDEuaNUzhH-lY/s1600/Screen+Shot+2020-07-10+at+1.49.11+pm.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="450" data-original-width="514" height="280" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhI8aW8yb5HN5AT6h406o7Mr1_pYC2zR23mwlnp-bnt2dyOy1X2BuFPwLIwCE_WTJEDxogCiKZNJU5C1rvw8_v4KIKN6Mtrils-d1WM4M2H2ciaQrua59-Q3nbTus3-v2JGDEuaNUzhH-lY/s320/Screen+Shot+2020-07-10+at+1.49.11+pm.png" width="320" /></a></div>
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjD82gp0LG7WGfBYaQvrhpjeJ5nDpFCugsJFwwsPj1C9A_7Ln5XRDkpo17ei6njSYXF_RoABmpYKE-FTfdDBGwyncU23ujx23PQlPg8izxBeiOnbtcp4Z4S9lgFBvR8guoNzBZXrpL5Jz6H/s1600/Screen+Shot+2020-07-10+at+1.49.22+pm.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="624" data-original-width="625" height="319" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjD82gp0LG7WGfBYaQvrhpjeJ5nDpFCugsJFwwsPj1C9A_7Ln5XRDkpo17ei6njSYXF_RoABmpYKE-FTfdDBGwyncU23ujx23PQlPg8izxBeiOnbtcp4Z4S9lgFBvR8guoNzBZXrpL5Jz6H/s320/Screen+Shot+2020-07-10+at+1.49.22+pm.png" width="320" /></a></div>
<br />
13. Once you enter the code sent to your email you will be granted access to the application endpoints<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiED-Ey5-0AD4mvY-k4wDjxRk6je7jzTpmKtwrXidhvfQ-uE6ckHw7duUPe-kS_09lgZO-gDOZjJKyZt8bg1uGU3VvmGX-KvHLYWL3XsQmE7Cu_Uj8bmzH5uiuULc1g6Vab4UGffnYGlzM4/s1600/Screen+Shot+2020-07-10+at+1.49.44+pm.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="529" data-original-width="1600" height="105" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiED-Ey5-0AD4mvY-k4wDjxRk6je7jzTpmKtwrXidhvfQ-uE6ckHw7duUPe-kS_09lgZO-gDOZjJKyZt8bg1uGU3VvmGX-KvHLYWL3XsQmE7Cu_Uj8bmzH5uiuULc1g6Vab4UGffnYGlzM4/s320/Screen+Shot+2020-07-10+at+1.49.44+pm.png" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3hVXNjQzJF2iFywx3T2ac3JmS1VgzCEsuAYSKdYHAWdmy7FRiWXsvwPvtVjnmAZV8MhjrlKT8btRvLQ5YHKU9eo-NKei8BHYOxJ-WyvZMcbSds9vSNSf_3F0_s03qMrN2VFNvb49Wx22f/s1600/Screen+Shot+2020-07-10+at+2.00.07+pm.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="634" data-original-width="701" height="289" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3hVXNjQzJF2iFywx3T2ac3JmS1VgzCEsuAYSKdYHAWdmy7FRiWXsvwPvtVjnmAZV8MhjrlKT8btRvLQ5YHKU9eo-NKei8BHYOxJ-WyvZMcbSds9vSNSf_3F0_s03qMrN2VFNvb49Wx22f/s320/Screen+Shot+2020-07-10+at+2.00.07+pm.png" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi7HaslpIYB9ZNj0-q7hzLJWYPJRnh9NNSuqOedKlid4dink3D_8iHuH_AjEstvxyXKWKFdIBYayDxU8OhFStQ8Z9M2WvIq37NG_fORoIaTb6zve1HDX4k-Nixt-vs3VMm23mWc3WJgvwGl/s1600/Screen+Shot+2020-07-10+at+1.59.57+pm.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="655" data-original-width="1598" height="131" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi7HaslpIYB9ZNj0-q7hzLJWYPJRnh9NNSuqOedKlid4dink3D_8iHuH_AjEstvxyXKWKFdIBYayDxU8OhFStQ8Z9M2WvIq37NG_fORoIaTb6zve1HDX4k-Nixt-vs3VMm23mWc3WJgvwGl/s320/Screen+Shot+2020-07-10+at+1.59.57+pm.png" width="320" /></a></div>
<br />
<br />
<br />
14. Finally to deploy the application to Tanzu Application Service perform these steps below<br />
<br />
- Create a manifest.yaml as follows<br />
<br />
<span style="color: #3d85c6;">---</span><br />
<span style="color: #3d85c6;">applications:</span><br />
<span style="color: #3d85c6;">- name: pas-okta-boot-app </span><br />
<span style="color: #3d85c6;"> memory: 1024M</span><br />
<span style="color: #3d85c6;"> buildpack: https://github.com/cloudfoundry/java-buildpack.git#v4.16</span><br />
<span style="color: #3d85c6;"> instances: 2</span><br />
<span style="color: #3d85c6;"> path: ./target/demo-0.0.1-SNAPSHOT.jar</span><br />
<span style="color: #3d85c6;"> env:</span><br />
<span style="color: #3d85c6;"> JBP_CONFIG_OPEN_JDK_JRE: '{ jre: { version: 11.+}}'</span><br />
<br />
- Package the application as follows<br />
<br />
<span style="color: #3d85c6;">$ ./mvnw -DskipTests package</span><br />
<br />
- In the DEV OTKA console create a second application which will be for the deployed application on Tanzu Application Service which refers to it's FQDN rather then localhost as shown below<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgqt4Rtdp8Th_UsevxGGf6OS2s-FQP9ktP7FSbTyVNbsWoKYf50IVuWpuLGj0PoqwYUaBMFtv8gX09MLN3cqL2ciSbFPTXwHo0Bpv5Fa-AarHPL3_qiwJfWkFA0_XQQ1j2H2JjcrloBavvg/s1600/Screen+Shot+2020-07-10+at+2.07.09+pm.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1042" data-original-width="1063" height="313" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgqt4Rtdp8Th_UsevxGGf6OS2s-FQP9ktP7FSbTyVNbsWoKYf50IVuWpuLGj0PoqwYUaBMFtv8gX09MLN3cqL2ciSbFPTXwHo0Bpv5Fa-AarHPL3_qiwJfWkFA0_XQQ1j2H2JjcrloBavvg/s320/Screen+Shot+2020-07-10+at+2.07.09+pm.png" width="320" /></a></div>
<br />
<br />
- Edit "<b>application.yml" </b>to ensure you set the following correctly for the new "<b>Application</b>" we created above.<br />
<br />
You will need to edit<br />
<br />
<ul>
<li>issuer</li>
<li>client-id</li>
<li>client-secret</li>
</ul>
<div>
- Push the application using "<b>cf push -f manifest.yaml</b>"</div>
<div>
<br /></div>
<div>
<div>
<span style="color: #3d85c6;">$ cf apps</span></div>
<div>
<span style="color: #3d85c6;">Getting apps in org papicella-org / space apple as papicella@pivotal.io...</span></div>
<div>
<span style="color: #3d85c6;">OK</span></div>
<div>
<span style="color: #3d85c6;"><br /></span></div>
<div>
<span style="color: #3d85c6;">name requested state instances memory disk urls</span></div>
<div>
<span style="color: #3d85c6;">pas-okta-boot-app started 1/1 1G 1G pas-okta-boot-app.cfapps.io</span></div>
</div>
<div>
<br /></div>
<br />
<span style="color: #e69138;">That's It!!!!</span><br />
<br /><div class="blogger-post-footer">http://feeds.feedburner.com/TheBlasFromPas</div>Pas Apicellahttp://www.blogger.com/profile/09389663166398991762noreply@blogger.com0