Friday, 27 June 2014

Pivotal Cloud Foundry Installed lets create an ORG / USER to get started

I installed Pivotal Cloud Foundry 1.2 recently and the commands below is what I run using the CLI to quickly create an ORG and a USER to get started with. Below assumes your connected as the ADMIN user to set a new ORG up.

Cloud Foundry CLI Commands as follows

cf api {cloud end point}
cf create-org pivotal
cf create-user pas pas
cf set-org-role pas pivotal OrgManager
cf target -o pivotal
cf create-space development
cf create-space test
cf create-space production
cf set-space-role pas pivotal production SpaceDeveloper
cf set-space-role pas pivotal development SpaceDeveloper
cf set-space-role pas pivotal test SpaceDeveloper
cf login -u pas -p pas -s development

Thursday, 15 May 2014

Pivotal GemFireXD*Web, Web based Interface For GemFireXD

Pivotal GemFire XD bridges GemFire’s proven in-memory intelligence and integrates it with Pivotal HD 2.0 and HAWQ. This enables businesses to make prescriptive decisions in real-time, such as stock trading, fraud detection, intelligence for energy companies, or routing for the telecom industries.

You can read more about how GemFireXD and it's integration with PHD here.

https://www.gopivotal.com/big-data/pivotal-hd

While development team worked on GemFireXD I produced another open source web based tool named GemFireXD*Web. It's available with source code as follows.

https://github.com/papicella/GemFireXD-Web

GemFireXD *Web enables schema management from a web browser with features as follows

  • Create all Schema Objects via Dialogs
  • Generate DDL
  • Run multiple SQL Commands, upload SQL files
  • Browse / Administer Objects
  • Browse / Administer HDFS stores/tables 
  • Browse / Administer Async Event Listeners
  • View data distribution
  • View Members / start parameters


etc…







Tuesday, 15 April 2014

Creating some Pivotal Cloud Foundry (PCF) PHD services

After installing PHD add on for Pivotal Cloud Foundry 1.1 I quickly created some development services for PHD using the CLI as shown below.

[Tue Apr 15 22:40:08 papicella@:~/vmware/pivotal/products/cloud-foundry ] $ cf create-service p-hd-hawq-cf free dev-hawq
Creating service dev-hawq in org pivotal / space development as pas...
OK
[Tue Apr 15 22:42:31 papicella@:~/vmware/pivotal/products/cloud-foundry ] $ cf create-service p-hd-hbase-cf free dev-hbase
Creating service dev-hbase in org pivotal / space development as pas...
OK
[Tue Apr 15 22:44:10 papicella@:~/vmware/pivotal/products/cloud-foundry ] $ cf create-service p-hd-hive-cf free dev-hive
Creating service dev-hive in org pivotal / space development as pas...
OK
[Tue Apr 15 22:44:22 papicella@:~/vmware/pivotal/products/cloud-foundry ] $ cf create-service p-hd-yarn-cf free dev-yarn
Creating service dev-yarn in org pivotal / space development as pas...
OK

Finally using the web console to brow the services in the "Development" space


Wednesday, 9 April 2014

Pivotal Greenplum GPLOAD with multiple CSV files

I recently needed to setup a cron script which loaded CSV files from a directory into Greenplum every 2 minutes. Once loaded the files are moved onto Hadoop for archive purposes. The config below shows how to use GPLOAD data load utility which utilises GPFDIST.

1. Create a load table. In this example the data is then moved to the FACT table once the load is complete

  
drop table rtiadmin.rtitrans_etl4;

CREATE TABLE rtiadmin.rtitrans_etl4 (
    imsi character varying(82),
    subscriber_mccmnc character varying(10),
    msisdn character varying(82),
    imei character varying(50),
    called_digits character varying(50),
    start_datetime integer,
    end_datetime integer,
    first_cell_lac integer,
    first_cell_idsac integer,
    current_cell_lac integer,
    current_cell_idsac integer,
    dr_type integer,
    status character varying(50),
    ingest_time bigint,
    processed_time bigint,
    export_time bigint,
    extra_col text,
    gploaded_time timestamp without time zone
)
WITH (appendonly=true) DISTRIBUTED BY (imsi); 

2. GPLOAD yaml file defined as follows

VERSION: 1.0.0.1
DATABASE: mydb
USER: rtiadmin
HOST: 172.1.1.1
PORT: 5432
GPLOAD:
   INPUT:
    - SOURCE:
         LOCAL_HOSTNAME:
            - loadhost
         PORT: 8100
         FILE:
          - /data/rti/stage/run/*.csv
    - COLUMNS:
          - imsi : text
          - subscriber_mccmnc : text
          - msisdn : text
          - imei : text
          - called_digits : text
          - start_datetime : text
          - end_datetime : text
          - first_cell_lac : integer
          - first_cell_idsac : integer
          - current_cell_lac : integer
          - current_cell_idsac : integer
          - dr_type : integer
          - status : text
          - ingest_time : bigint
          - processed_time : bigint
          - export_time : bigint
          - extra_col : text
    - FORMAT: text
    - HEADER: false
    - DELIMITER: ','
    - NULL_AS : ''
    - ERROR_LIMIT: 999999
    - ERROR_TABLE: rtiadmin.rtitrans_etl4_err
   OUTPUT:
    - TABLE: rtiadmin.rtitrans_etl4
    - MODE: INSERT
    - MAPPING:
           imsi : imsi
           subscriber_mccmnc : subscriber_mccmnc
           msisdn : msisdn
           imei : imei
           called_digits : called_digits
           start_datetime : substr(start_datetime, 1, 10)::int
           end_datetime : substr(end_datetime, 1, 10)::int
           first_cell_lac : first_cell_lac
           first_cell_idsac : first_cell_idsac
           current_cell_lac : current_cell_lac
           current_cell_idsac : current_cell_idsac
           dr_type : dr_type
           status : status
           ingest_time : ingest_time
           processed_time : processed_time
           export_time : export_time
           extra_col : extra_col
           gploaded_time : current_timestamp
   PRELOAD:
    - TRUNCATE : true 
    - REUSE_TABLES : true
   SQL:
    - AFTER : "insert into rtitrans select * from rtitrans_etl4" 

3. Call GPLOAD as follows

source $HOME/.bash_profile
gpload -f rtidata.yml


Note: We use the ENV variable as $PGPASSWORD which is used during the load if a password is required which was in this demo

Few things worth noting here.

REUSE_TABLES : This ensures the external tables created during the load are maintained and re-used on next load.

TRUNCATE: This clears the load table prior to load and we use this as we COPY the data once the load is finished into the main FACT table using the "AFTER"

Tuesday, 4 March 2014

Pivotal Cloud Foundry using App Direct "newrelic" Monitoring Service

PCF AWS marketplace provides app direct services and in this example I am going to use the "newrelic" monitoring service to monitor my spring based java application. It's really this simple.

1. Create a service as shown below.

[Tue Mar 04 17:19:34 papicella@:~/cfapps/spring-travel ] $ cf create-service newrelic standard dev-newrelic

2. Create a manifest.yml for my spring application which uses the new relic service above.

applications:
- name: pas-springtravel 
  memory: 1024M 
  instances: 1
  host: pas-springtravel 
  domain: cfapps.io 
  path: ./travel.war
  services:
  - dev-mysql
  - dev-newrelic

3. Push the application

[Tue Mar 04 17:19:34 papicella@:~/cfapps/spring-travel ] $ cf push -f manifest.yml 
Using manifest file manifest.yml

Creating app pas-springtravel in org papicella-org / space development as papicella@gopivotal.com...
OK

Using route pas-springtravel.cfapps.io
Binding pas-springtravel.cfapps.io to pas-springtravel...
OK

Uploading pas-springtravel...
Uploading from: travel.war
5.3M, 2748 files
OK
Binding service dev-mysql to pas-springtravel in org papicella-org / space development as papicella@gopivotal.com
OK
Binding service dev-newrelic to pas-springtravel in org papicella-org / space development as papicella@gopivotal.com
OK

Starting app pas-springtravel in org papicella-org / space development as papicella@gopivotal.com...
OK
-----> Downloaded app package (22M)
-----> Uploading droplet (67M)

0 of 1 instances running, 1 starting
0 of 1 instances running, 1 starting
0 of 1 instances running, 1 starting
0 of 1 instances running, 1 starting
0 of 1 instances running, 1 starting
1 of 1 instances running

App started

Showing health and status for app pas-springtravel in org papicella-org / space development as papicella@gopivotal.com...
OK

requested state: started
instances: 1/1
usage: 1G x 1 instances
urls: pas-springtravel.cfapps.io

     state     since                    cpu    memory         disk           
#0   running   2014-03-04 05:24:43 PM   0.0%   610.7M of 1G   155.9M of 1G 

4. Under the services listed on AWS click on "Manage" and here are some screen shots of what the newrelic monitoring service provides with just a simple BIND when we pushed the application.





Friday, 21 February 2014

Deploying Spring MVC application to Cloud Foundry from IntelliJ IDEA

I previously showed how to create a connection in IntelliJ IDEA to Cloud Foundry v2 in the post below.

http://theblasfrompas.blogspot.com.au/2014/02/intellij-idea-version-13-now-includes.html

With a Cloud Foundry CLOUD connection we can now PUSH our application directly from the IDE as shown below.

1. Create a run configuration for your project as shown below. We also specify the memory and number of instances on this page as part of the push  / deployment process.



2. Select the created run configuration and deploy the application, output as follows


3. View the deployed application on AWS hosted instance of Cloud Foundry



IntelliJ IDEA version 13 now includes CloudFoundry Connection

Just installed IntelliJ IDEA version 13 and found that it now includes a CloudFoundry connection type. You define it under IDE settings as shown below.


Will test deploying to the publicly hosted AWS Cloud Foundry using this connection at some stage.

Thursday, 20 February 2014

PCF (Pivotal Cloud Foundry) cf push multiple applications using manifest file

By creating a manifest as follows we can push multiple applications in one go as shown below.

1. manifest.yml

applications:
- name: pas-props
  memory: 256M
  instances: 1
  host: pas-props
  domain: cfapps.io
  path: ./props.war
- name: pas-httpsession
  memory: 256M
  instances: 1
  host: pas-httpsession
  domain: cfapps.io
  path: ./haclusterdemo.war

2. Push as follows

[Thu Feb 20 14:48:45 papicella@:~/vmware/pivotal/products/cloud-foundry/apps/other ] $ cf push -f manifest-twoapps.yml
Using manifest file manifest-twoapps.yml

Creating app pas-props in org papicella-org / space development as papicella@gopivotal.com...
OK

Using route pas-props.cfapps.io
Binding pas-props.cfapps.io to pas-props...
OK

Uploading pas-props...
Uploading from: props.war
2.7K, 5 files
OK

Starting app pas-props in org papicella-org / space development as papicella@gopivotal.com...
OK

1 of 1 instances running

App started

Showing health and status for app pas-props in org papicella-org / space development as papicella@gopivotal.com...
OK

requested state: started
instances: 1/1
usage: 256M x 1 instances
urls: pas-props.cfapps.io

     state     since                    cpu    memory           disk           
#0   running   2014-02-20 02:51:14 PM   0.0%   193.2M of 256M   110.7M of 1G   
Creating app pas-httpsession in org papicella-org / space development as papicella@gopivotal.com...
OK

Creating route pas-httpsession.cfapps.io...
OK

Binding pas-httpsession.cfapps.io to pas-httpsession...
OK

Uploading pas-httpsession...
Uploading from: haclusterdemo.war
130.2K, 10 files
OK

Starting app pas-httpsession in org papicella-org / space development as papicella@gopivotal.com...
OK

0 of 1 instances running, 1 starting
1 of 1 instances running

App started

Showing health and status for app pas-httpsession in org papicella-org / space development as papicella@gopivotal.com...
OK

requested state: started
instances: 1/1
usage: 256M x 1 instances
urls: pas-httpsession.cfapps.io

     state     since                    cpu    memory           disk           
#0   running   2014-02-20 02:53:53 PM   2.0%   214.3M of 256M   113.3M of 1G   

More Information

http://docs.cloudfoundry.org/devguide/deploy-apps/manifest.html

Tuesday, 18 February 2014

PCF (Pivotal Cloud Foundry) Using the new CLI to bind services from a cf push

I previously blogged how to PUSH an application into PCF , then bind services as shown below.



If you want to do this in one step simply create a manifest file as shown below. This example isn't the spring books application but shows you what your manifest.yml would look like with services included so you could then create one similar to that below.

1. Create a manifest.yml file as follows

manifest.yml

applications:
- name: pas-springtravel 
  memory: 512M 
  instances: 1
  host: pas-springtravel 
  domain: cfapps.io 
  path: ./travel.war
  services:
  - dev-mysql

2. Push the application and bind to the services at push time.

[Tue Feb 18 20:13:01 papicella@:~/vmware/pivotal/products/cloud-foundry/apps/spring-travel ] $ cf push -f manifest.yml 
Using manifest file manifest.yml

Creating app pas-springtravel in org papicella-org / space development as papicella@gopivotal.com...
OK

Creating route pas-springtravel.cfapps.io...
OK

Binding pas-springtravel.cfapps.io to pas-springtravel...
OK

Uploading pas-springtravel...
Uploading from: travel.war
5.3M, 2748 files
OK
Binding service dev-mysql to pas-springtravel in org papicella-org / space development as papicella@gopivotal.com
OK

Starting app pas-springtravel in org papicella-org / space development as papicella@gopivotal.com...
OK
-----> Downloaded app package (22M)
-----> Uploading droplet (62M)

0 of 1 instances running, 1 starting
0 of 1 instances running, 1 starting
1 of 1 instances running

App started

Showing health and status for app pas-springtravel in org papicella-org / space development as papicella@gopivotal.com...
OK

requested state: started
instances: 1/1
usage: 512M x 1 instances
urls: pas-springtravel.cfapps.io

     state     since                    cpu    memory           disk           
#0   running   2014-02-18 08:24:58 PM   0.5%   458.6M of 512M   149.9M of 1G   

3. The application using the MYSQL service


Monday, 17 February 2014

PCF (Pivotal Cloud Foundry) new command line client

I decided to install the new CF v6 CLI for PCF (Pivotal Cloud Foundry). Given this will be the new CLI for PCF here is how it differs to the old Ruby CLI. In this example I deploy / push a spring based application and bind it to existing services.

To download and install the new CLI v6 use the link below.

http://docs.cloudfoundry.com/docs/using/managing-apps/cf/go-cli.html

In this example we have the following 2 services that already exist and I am using the PUBLIC hosted AWS version of PCF at https://console.run.pivotal.io/

  • dev-mysql
  • dev-redis

1. First lets deploy our application and when we do we DON'T actually start it at this point.

[Sun Feb 16 22:23:22 papicella@:~/vmware/pivotal/products/cloud-foundry/apps/spring-books ] $ cf push pas-books -d cfapps.io -i 1 -m 512M -p springmvc31.war --no-start
Creating app pas-books in org papicella-org / space development as papicella@gopivotal.com...
OK

Creating route pas-books.cfapps.io...
OK

Binding pas-books.cfapps.io to pas-books...
OK

Uploading pas-books...
Uploading from: /Users/papicella/vmware/pivotal/products/cloud-foundry/apps/spring-books/springmvc31.war
511.2K, 44 files
OK

2. Bind the application to the 2 services above.

[Sun Feb 16 22:25:56 papicella@:~/vmware/pivotal/products/cloud-foundry/apps/spring-books ] $ cf bind-service pas-books dev-mysql
Binding service dev-mysql to app pas-books in org papicella-org / space development as papicella@gopivotal.com...
OK
TIP: Use 'cf push' to ensure your env variable changes take effect

[Sun Feb 16 22:25:56 papicella@:~/vmware/pivotal/products/cloud-foundry/apps/spring-books ] $ cf bind-service pas-books dev-redis
Binding service dev-redis to app pas-books in org papicella-org / space development as papicella@gopivotal.com...
OK
TIP: Use 'cf push' to ensure your env variable changes take effect

3. Push the application again and this time start it.

[Sun Feb 16 22:29:26 papicella@:~/vmware/pivotal/products/cloud-foundry/apps/spring-books ] $ cf push pas-books -p springmvc31.war
Updating app pas-books in org papicella-org / space development as papicella@gopivotal.com...
OK

Uploading pas-books...
Uploading from: /Users/papicella/vmware/pivotal/products/cloud-foundry/apps/spring-books/springmvc31.war
511.2K, 44 files
OK

Starting app pas-books in org papicella-org / space development as papicella@gopivotal.com...
-----> Downloaded app package (19M)
OK
-----> Java Buildpack source: system
-----> Downloading Open JDK 1.7.0_51 from http://download.pivotal.io.s3.amazonaws.com/openjdk/lucid/x86_64/openjdk-1.7.0_51.tar.gz 
(1.2s)
       Expanding Open JDK to .java-buildpack/open_jdk 
(1.2s)
-----> Downloading Spring Auto Reconfiguration 0.8.7 from http://download.pivotal.io.s3.amazonaws.com/auto-reconfiguration/auto-reconfiguration-0.8.7.jar 
(1.4s)
       Modifying /WEB-INF/web.xml for Auto Reconfiguration
-----> Downloading Tomcat 7.0.50 from http://download.pivotal.io.s3.amazonaws.com/tomcat/tomcat-7.0.50.tar.gz 
(0.3s)
       Expanding Tomcat to .java-buildpack/tomcat 
(0.1s)
-----> Downloading Buildpack Tomcat Support 1.1.1 from http://download.pivotal.io.s3.amazonaws.com/tomcat-buildpack-support/tomcat-buildpack-support-1.1.1.jar 
(0.0s)
-----> Uploading droplet (60M)

0 of 1 instances running, 1 starting
0 of 1 instances running, 1 starting
0 of 1 instances running, 1 starting
1 of 1 instances running

App started

Showing health and status for app pas-books in org papicella-org / space development as papicella@gopivotal.com...
OK

requested state: started
instances: 1/1
usage: 512M x 1 instances
urls: pas-books.cfapps.io

     state     since                    cpu    memory           disk           
#0   running   2014-02-16 10:31:13 PM   0.0%   494.9M of 512M   134.2M of 1G   


If all went well we should see something as follows.


And of course a running application