Search This Blog

Friday, 8 June 2012

Modelling SQLFire with a Locator VM in vFabric Application Director

I recently setup a SQLFire distributed system by modelling a blueprint from vFabric Application Director. In this simple setup I created a service for a locator as well as a SQLFire member. The screen shots below show what this looked like.

New Catalog Items

1. Create the locator service as follows

1.1. Define it as follows


1.2. Set properties as follows


1.3. Add INSTALL script as shown below
  
#!/bin/sh
# Import global conf
. $global_conf

set -e

#To support ubuntu java bins
export PATH=$PATH:/usr/java/jre-vmware/bin

mkdir -p $install_path
java -Dsqlfire.installer.directory="$install_path" -jar `basename "$SQLFire102_Installer"`
 
cd $install_path/*SQLFire*

mkdir locator
./bin/sqlf locator start -peer-discovery-address=$locator_ip -peer-discovery-port=$locator_port -client-bind-address=$locator_ip -client-port=$client_port -dir=locator

2. Create a SQLFire server service

2.1. Define it as follows




2.2. Set properties as follows


2.3 Add install script as shown below
  
#!/bin/sh
# Import global conf
. $global_conf

set -e

#To support ubuntu java bins
export PATH=$PATH:/usr/java/jre-vmware/bin

mkdir -p $install_path
java -Dsqlfire.installer.directory="$install_path" -jar `basename "$SQLFire102_Installer"`
 
cd $install_path/*SQLFire*

mkdir server1
./bin/sqlf server start -dir=server1 -client-port=$client_port -client-bind-address=$server_ip -locators=$locator_ip[$locator_port]

Blueprint using the Catalog Items above

The catalog items are simply added to the templates as needed. From the image below we have one locator VM and a cluster of SQLFire server node VM's as well. This enables us to create as many SQLFire member nodes as needed.


From the screen shot below it shows how we determine the locator IP address as well as the SQLFire server node IP address at runtime as we need that to be injected in for us before the lifecycle INSTALL phase is started.


Tuesday, 29 May 2012

SQLFire multi-site WAN replication

In then latest SQLFire 102 release we have introduced muti-site WAN deployments. The example below shows how quickly we can get this setup with simple SQL calls on both sites. In this very simple example we will have 2 distributed SQLFire sites being setup as a multi-site WAN deployment scenario.

The steps below switch from one site to the other so be careful to run the SQL on the correct site.

Note: This demo is designed to on a single laptop/desktop, hence the refernces to localhost

Site 1

1. Start a locator as follows for the first distributed SQLFire system

> sqlf locator start -peer-discovery-address=localhost -peer-discovery-port=10101 -locators=localhost:10101 -conserve-sockets=false -distributed-system-id=1 -remote-locators=localhost[20202] -client-bind-address=localhost -client-port=1527

2. start 2 SQLFire nodes as shown below.

> sqlf server start -server-groups=MYGROUP -locators=localhost[10101] -client-bind-address=localhost -client-port=1529 -dir=server1 &
sqlf server start -server-groups=MYGROUP -locators=localhost[10101] -client-bind-address=localhost -client-port=1530 -dir=server2 &

Site 2

3. Start a locator as follows for the second distributed SQLFire system

> sqlf locator start -peer-discovery-address=localhost -peer-discovery-port=20202 -locators=localhost:20202 -conserve-sockets=false -distributed-system-id=2 -remote-locators=localhost[10101] -client-bind-address=localhost -client-port=1528

4. start 2 SQLFire nodes as shown below.

> sqlf server start -server-groups=MYGROUP -locators=localhost[20202] -client-bind-address=localhost -client-port=1531 -dir=server1 &
sqlf server start -server-groups=MYGROUP -locators=localhost[20202] -client-bind-address=localhost -client-port=1532 -dir=server2 &

Site 1

Create sender/receiver as shown below.

5. Create sender as follows
  
create diskstore cluster1store;

CREATE GATEWAYSENDER cluster2sender
(
  REMOTEDSID 2
  ENABLEPERSISTENCE true
  DISKSTORENAME cluster1store 
)
SERVER GROUPS (MYGROUP); 

6. Create receiver as follows
  
CREATE GATEWAYRECEIVER test_receiver (startport 1550 endport 1561) 
server groups (MYGROUP);

Site 2

Create sender/receiver as well as the table we wish to replicate between sites.

7. Create sender as follows
  
create diskstore cluster2store;

CREATE GATEWAYSENDER cluster1sender
(
  REMOTEDSID 1
  ENABLEPERSISTENCE true
  DISKSTORENAME cluster2store 
)
SERVER GROUPS (MYGROUP);

8. Create receiver as follows
  
CREATE GATEWAYRECEIVER test_receiver (startport 1550 endport 1561) 
server groups (MYGROUP);

Site 1

Create the table we wish to replicate. This must be done on each site as DDL is not replicated, only DML is.

9. Create table as shown below
  
CREATE TABLE test_table 
(ID INT NOT NULL, NAME VARCHAR(10))
GATEWAYSENDER(cluster2sender) SERVER GROUPS (MYGROUP);

Site 2

Create the table we wish to replicate. This must be done on each site as DDL is not replicated, only DML is.

10. Create table as shown below.
  
CREATE TABLE test_table 
(ID INT NOT NULL, NAME VARCHAR(10))
GATEWAYSENDER(cluster1sender) SERVER GROUPS (MYGROUP);

Verifying the setup

With this now setup we simply need to connect to either distributed system and insert some data into our table as shown below.

11. Connect to the first distributed system and insert some data as shown below.
  
[Tue May 29 10:47:47 papicella@:~/sqlfire/vFabric_SQLFire_102/pasdemos/wan-demo/dist1 ] $ sqlf
sqlf version 10.4
sqlf> connect client 'localhost:1527';
sqlf> insert into test_table values (1, 'apples');
1 row inserted/updated/deleted
sqlf> commit;
sqlf> select * from test_table;
ID         |NAME      
----------------------
1          |apples    

1 row selected

12. Connect to the second distributed system to verify the data was replicated from test_table
  
[Tue May 29 10:46:31 papicella@:~/sqlfire/vFabric_SQLFire_102/pasdemos/wan-demo/dist2 ] $ sqlf
sqlf version 10.4
sqlf> connect client 'localhost:1528';
sqlf> select * from test_table;
ID         |NAME      
----------------------
1          |apples    

1 row selected

For more information on multi site WAN deployments with SQLFire see the documentation below.

http://pubs.vmware.com/vfabric5/index.jsp?topic=/com.vmware.vfabric.sqlfire.1.0/config_guide/topics/gateway-hubs/wan-steps.html


Monday, 21 May 2012

Simple example on how to caputure the query execution plan from SQLFire

The example below shows how we can capure the query plan for SQL running in SQLFire. Although any user can run an explain plan you must connect as a peer to show the query execution plan as the data is stored in the SYS.STATEMENTPLANS table.

1. Determine what indexes we currently have in the "APP" schema.
  
sqlf> show indexes in app;
TABLE_NAME          |COLUMN_NAME         |NON_U&|TYPE|ASC&|CARDINA&|PAGES   
----------------------------------------------------------------------------
DEPT                |DEPTNO              |0     |3   |A   |NULL    |NULL    
EMP                 |EMPNO               |0     |3   |A   |NULL    |NULL    
EMP                 |DEPTNO              |1     |3   |A   |NULL    |NULL    
EMP                 |JOB                 |1     |3   |A   |NULL    |NULL    

4 rows selected

2. Run a query explain for an individual SQL statement and display the execution plan. In this example below we connect as a peer client to perform the opertion BUT we only need to conect as a peer client to actually display the execution plan and we can explain our SQL as a regular client user.
  
sqlf> connect peer 'host-data=false;mcast-port=12333';
sqlf> explain select * from emp where job = 'CLERK';
MEMBER_PLAN                                                                                                                     
--------------------------------------------------------------------------------------------------------------------------------
ORIGINATOR 192-168-1-4.tpgi.com.au(2201)<v4>:11255/52510 BEGIN TIME 2012-05-21 20:02:03.659 END TIME 2012-05-21 20:02:03.694
DI&
Slowest Member Plan: 
member   192-168-1-4.tpgi.com.au(2064)<v0>:39756/52450 begin_execution  2012-05-21 20:02:03.664 end_execu&
Fastest Member Plan: 
member   192-168-1-4.tpgi.com.au(2065)<v1>:55134/52451 begin_execution  2012-05-21 20:02:03.664 end_execu&

3 rows selected
sqlf> select STMT_ID, STMT_TEXT from SYS.STATEMENTPLANS;
STMT_ID                             |STMT_TEXT                                                                                                                       
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
00000001-ffff-ffff-ffff-000300000016| select * from emp where job = 'CLERK'                                                                                          

1 row selected
sqlf> explain '00000001-ffff-ffff-ffff-000300000016';
stmt_id   00000001-ffff-ffff-ffff-000300000016 begin_execution  2012-05-21 20:02:03.659 end_execution  2012-05-21 20:02:03.694
QUERY-SCATTER execute_time 31696000 member_node 192-168-1-4.tpgi.com.au(2064)<v0>:39756/52450,192-168-1-4.tpgi.com.au(2065)<v1>:55134/52451
  QUERY-SEND execute_time 397000 member_node 192-168-1-4.tpgi.com.au(2064)<v0>:39756/52450
    QUERY-SEND execute_time 178000 member_node 192-168-1-4.tpgi.com.au(2065)<v1>:55134/52451
      RESULT-RECEIVE execute_time 150000 member_node 192-168-1-4.tpgi.com.au(2065)<v1>:55134/52451
        RESULT-RECEIVE execute_time 35000 member_node 192-168-1-4.tpgi.com.au(2064)<v0>:39756/52450
          SEQUENTIAL-ITERATION returned_rows 7 no_opens 1 execute_time 218000
            RESULT-HOLDER returned_rows 2 no_opens 1 execute_time 39000 member_node 192-168-1-4.tpgi.com.au(2065)<v1>:55134/52451
              RESULT-HOLDER returned_rows 5 no_opens 1 execute_time 33000 member_node 192-168-1-4.tpgi.com.au(2064)<v0>:39756/52450
                DISTRIBUTION-END returned_rows 7 execute_time 28521000
member   192-168-1-4.tpgi.com.au(2065)<v1>:55134/52451 begin_execution  2012-05-21 20:02:03.664 end_execution  2012-05-21 20:02:03.691
QUERY-RECEIVE execute_time 26501000 member_node 192-168-1-4.tpgi.com.au(2201)<v4>:11255/52510
  RESULT-SEND execute_time 35000 member_node 192-168-1-4.tpgi.com.au(2201)<v4>:11255/52510
    RESULT-HOLDER returned_rows 2 no_opens 1 execute_time 802000
      ROWIDSCAN returned_rows 2 no_opens 1 execute_time 62000
        INDEXSCAN returned_rows 2 no_opens 1 execute_time 6173000 scan_qualifiers None scanned_object EMP_JOB_IDX scan_type 
member   192-168-1-4.tpgi.com.au(2064)<v0>:39756/52450 begin_execution  2012-05-21 20:02:03.664 end_execution  2012-05-21 20:02:03.692
QUERY-RECEIVE execute_time 27574000 member_node 192-168-1-4.tpgi.com.au(2201)<v4>:11255/52510
  RESULT-SEND execute_time 25000 member_node 192-168-1-4.tpgi.com.au(2201)<v4>:11255/52510
    RESULT-HOLDER returned_rows 5 no_opens 1 execute_time 885000
      ROWIDSCAN returned_rows 5 no_opens 1 execute_time 108000
        INDEXSCAN returned_rows 5 no_opens 1 execute_time 6682000 scan_qualifiers None scanned_object EMP_JOB_IDX scan_type 
sqlf>

So the steps are as follows:

1. Run an explain for the SQL using the key word "EXPLAIN".
2. Connect as a peer
3. Show the query plan execution for the statement.

For more information on the query execution plan codes visit the link below.

http://pubs.vmware.com/vfabric5/index.jsp?topic=/com.vmware.vfabric.sqlfire.1.0/manage_guide/explain-codes.html



Wednesday, 9 May 2012

vfabric Application Director 3 Tier Spring Travel Blueprint

Here are the steps used to create a 3 tier application using Spring travel with a MYSQL database. This is based on the screen shots from the previous blog entry.

1. Log into "vFabric Application Director" home page
2. Click on "Manage Applications".
3. Locate "Spring Travel" and select the icon for "Copy this application version" under the "Actions" column
4. Select radio option "Save as new application" and enter details as follows.



5. Click ok.
6. Click on the symbol  IMG6 "Convert to node array" to ensure we create a cluster of these VM's.
7. Set the cluster size to 2.
8. Click on the "Memory" column and set each VM to 1024M as shown below

IMG7

9. Press the "Save" button at the top right hand corner of the blueprint editor.

Note: Ignore any warning and save the changes.

10. Under "Logical Templates -> OS Templates" drop "CentOS56 32 bit 1.0.0" onto the canvas.
11.  Name it "load_balancer" and set the memory to 512M as shown below.

IMG8


12. Save the blueprint
13. Under "Services -> Web Servers" drop "Apache 2.2.0"
14. Name it "Apache_LB" for the "Details" column
15. In the top right hand corner use the icon IMG9to create a relationship from "Apache_LB" to "SpringTravelApp" as shown below.

IMG10

16. Save the blueprint
17. Click on "Apache_LB" and select the "Properties" column
18. Edit the property "http_node_ips" and ensure the blueprint property is set as shown below. This will we setup our http.conf for Apache to bind to the 2 tc Server VM's using the correct ip address.

IMG11

19. Edit the property "jvm_routes" and edit it use the blueprint value "all(SpringTravel:tcServer:JVM_ROUTE)". You should have something as follows.

IMG12

20. Save the blueprint.
21. Under "Logical Templates -> OS Templates" drop "CentOS56 32 bit 1.0.0" onto the canvas.
22. Name it "MySQLTier" and change the "Memory" to 1024M.
23. Under "Services -> Database Servers" drop "MySQL 5.0.0" onto the template
24. Under the "Properties"column edit the "db_root_password" to "welcome1" as shown below.

IMG13

25. Save the blueprint
26. In the top right hand corner use the icon IMG9to create a relationship from "SpringTravelApp" to "Mysql".
27. Drop from "Application Components" a "SQL Script" onto "Mysql"
28. In the top right hand corner use the icon IMG9to create a relationship from "SQL_SCRIPT" to "Mysql" which should give you a blueprint as follows at this point.

IMG14

29. Select "SpringTravelApp" and add a property as shown below to ensure we bind to the "MySQLTier" ip address.

IMG15

30. Edit the "war" property and set it's "Blueprint Value" to "http://dl.dropbox.com/u/15829935/se-demos/app-director/mysql/mysql-swf-booking-mvc.war"
31. Click on the "Actions" column
32. Double click on the "script" column for the lifecycle stage "CONFIGURE" and add contents as shown below.


#!/bin/sh

env > /tmp/env.txt

sleep 10

export mysqlip="${dbip}"

echo "mysql ip is ${mysqlip}"

cd $installdir/working/springsource-tc-server-standard/instance1/webapps/mysql-swf-booking-mvc/WEB-INF/config
sed -e "s/localhost/${mysqlip}/g" data-access-config.xml > data-access-config-NEW.xml
mv data-access-config-NEW.xml data-access-config.xml

33. Double click on the "script" column for the lifecycle stage "START" and add contents as shown below.

#!/bin/sh

env > /tmp/env.txt

cd $installdir/working/springsource-tc-server-standard/instance1/bin

./tcruntime-ctl.sh stop

./tcruntime-ctl.sh start


34. Click on "SQL_SCRIPT" on the "MySQLTier"
35. Click on the "Actions" column
36. Double click on the "script" column for the lifecycle stage "INSTALL" and add contents as shown below

#!/bin/bash

mysql -h localhost -u root -pwelcome1 < create database travel;
CREATE USER 'travel'@'%' IDENTIFIED BY 'travel';
GRANT ALL ON travel.* TO 'travel'@'%';
!

37. Save the blueprint
38. Click on the icon in the top right hand corner IMG16 to deploy the blueprint
39. Enter a deployment profile name as "SpringTravelMysql"
40. Click Ok
41. Click on "Map Details" button
42. Click Next
43. Click Next
44. Click Next
45. Click Deploy

A successfull deployment will look as follows once complete.



Thursday, 26 April 2012

vFabric Application Director

VMware vFabric Application Director is a cloud-enabled application provisioning solution that simplifies how to create and standardize application deployment topologies across cloud services. It is designed for Application Provisioning in hybrid clouds, using an open and extensible architecture.

The screen shots below show a 3 tier application topology (blueprint) modeled in Application Director as follows. It is a 3 tier application using an Apache front end load balancer with 2 tc Server middle tier VM's and a single MYSQL server back end DB tier.

Application Blueprint Page


Deployment Summary Page



Deployment Details Page



Deployment Execution Plan



VM Console Page



Registered Cloud Provider Dialog



More Information

http://www.vmware.com/products/application-platform/vfabric-appdirector/overview.html

Thursday, 12 April 2012

vFabric Application Performance Manager

VMware vFabric Application Performance Manager sets a new standard for how businesses will manage applications in the cloud. With a focus on key indicators for application performance, such as throughput, latency, hit rate and error rates, along with deep code level diagnostics, the vFabric AppInsight dashboard provides IT and development teams with a common perspective so they can work together to fix problems quickly.

Below are some examples of what AppInsight dashboard gives you. In short the AppInsight probes run on the ESXi hosts. The BCI (Byte code Instrumentation) agents gives you the code level monitoring required from your java middleware VM's so you can for example see the SQL executed for any page request so developers can easily debug code.

Main App Insight Page


Spring Travel / Oracle Application Summary At a Glance



Spring Travel / Oracle Application Topology

Note: This topology builds itself out thanks to the Application Probes running on the ESXi hosts.


Spring Travel / Oracle Application Main Summary Page


Spring Travel / Oracle Transaction Sample Page

Note: Here we can see how the BCI agent gives us the code instrumentation so you pin point where latency/errors are occurring in the actual java code.



For more information on vFabric Application Performance Manager see the link below.

http://www.vmware.com/products/application-platform/vfabric-application-performance-manager/overview.html

Monday, 26 March 2012

Hyperic monitoring it's own Oracle repository

Below are some screen shots of what Hyperic 4.6.5 shows in regards to monitoring Oracle 11g RDBMS running on a VM running Redhat 5.7 guest OS. The database it's monitoring is the actual repository database being used by Hyperic itself. The screen shots below show the main dashboard with the Oracle 11g resource overall status as well as viewing a DB writer process for the running instance itself.


The plugin to monitor the Oracle instance is provided as one of the 80+ plugins available with Hyperic out of the box. Once the HQ agent discovers the Oracle instance we are left with simply providing the config to a DB user which has access to all data dictionary tables. The easiest way to do that is to ensure the user you connect wih has the following granted.

grant select any dictionary to {user};

To me given I own the VM / Oracle instance so I will use the system user as the user to query the data dictionary as shown below. Once setup your good to go.


For more information on vFabric Hyperic use the link below.

http://pubs.vmware.com/vfabric5/index.jsp?topic=/com.vmware.vfabric.hyperic.4.6/vFabric_Hyperic_4.6.html

Friday, 16 March 2012

Using derby style table functions to load data into SQLFire

Loading data into SQLFire can be done various ways including using Spring Batch with CSV files, Apache DDLUtils or direct JDBC connections pulling data into SQLFire. The approach below is yet another way. In this example we load the Oracle HR schema table "departments" into a SQLFire distributed system.

1. Create a java class with code that simple queries the table data and returns it as aJDBC ResultSet.
package vmware.au.se.sqlfire.derby;

import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;

import oracle.jdbc.OracleDriver;

public class DepartmentTable 
{

 public static String SQL = "select * from departments";
 
 public static ResultSet read() throws SQLException
 {
        Connection conn = getConnection();
        Statement  stmt = conn.createStatement(); 
        ResultSet  rset = stmt.executeQuery(SQL);
        
        return rset;
 }
 
 public static Connection getConnection() throws SQLException
 {
     String username = "hr";
     String password = "hr";
     String thinConn = "jdbc:oracle:thin:@172.16.101.70:1521/linux11gr2";
     DriverManager.registerDriver(new OracleDriver());
     Connection conn = DriverManager.getConnection(thinConn,username,password);
     conn.setAutoCommit(false);
     return conn;
 }
 
} 

2. Add the JAR file with the class above as well as the Oracle JDBC jar file to the system CLASSPATH as shown below.

export CUR_DIR=`pwd`
export CLASSPATH=$CUR_DIR/lib/ojdbc6.jar:$CUR_DIR/lib/derby-funct.jar

Note: This ensures when we start our SQLFire nodes they will have the classes avaiable on the classpath

3. Start the SQLFire servers as shown below.

> sqlf server start -server-groups=MYGROUP -locators=localhost[41111] -client-bind-address=localhost -client-port=1528 -dir=server1 -classpath=$CLASSPATH &

> sqlf server start -server-groups=MYGROUP -locators=localhost[41111] -client-bind-address=localhost -client-port=1529 -dir=server2 -classpath=$CLASSPATH &

4. Log into the distributed system using the CLI and run the SQL as follows to create the table in SQLFire which will store the same dataset from the Oracle "departments" table.
create diskstore STORE1;
 
call sys.set_eviction_heap_percentage_sg (85, 'MYGROUP');

drop table departments;

create table departments
(department_id int NOT NULL CONSTRAINT department_id_PK PRIMARY KEY,
 department_name varchar(40),
 manager_id int,
 location_id int)
partition by column (department_id)
SERVER GROUPS (MYGROUP)
persistent 'STORE1'
REDUNDANCY 1;

5. Log into the distributed system using the CLI and run the SQL below to create a function
CREATE FUNCTION externalDepartments
()
RETURNS TABLE
(
  DEPARTMENT_ID    INT,
  DEPARTMENT_NAME  VARCHAR( 40 ),
  MANAGER_ID       INT,
  LOCATION_ID      INT
)
LANGUAGE JAVA
PARAMETER STYLE DERBY_JDBC_RESULT_SET
READS SQL DATA
EXTERNAL NAME 'vmware.au.se.sqlfire.derby.DepartmentTable.read'; 

6. Log into the distributed system using the CLI and run the SQL below to insert data into the "departments" table in SQLFire using the function we created at #5.
insert into departments
select s.*
FROM TABLE (externalDepartments()) s;

7. Verify we now have our departments table in SQLFire with data.
[Fri Mar 16 08:55:40 papicella@:~/sqlfire/vFabric_SQLFire_101/pasdemos/oraclehr ] $ sqlf
sqlf version 10.4
sqlf> connect client 'localhost:1527';
sqlf> select * from departments;
DEPARTMENT&|DEPARTMENT_NAME                         |MANAGER_ID |LOCATION_ID
----------------------------------------------------------------------------
230        |IT Helpdesk                             |NULL       |1700       
120        |Treasury                                |NULL       |1700       
10         |Administration                          |200        |1700       
240        |Government Sales                        |NULL       |1700       
130        |Corporate Tax                           |NULL       |1700       
20         |Marketing                               |201        |1800       
250        |Retail Sales                            |NULL       |1700       
140        |Control And Credit                      |NULL       |1700       
30         |Purchasing                              |114        |1700       
260        |Recruiting                              |NULL       |1700       
150        |Shareholder Services                    |NULL       |1700       
40         |Human Resources                         |203        |2400       
270        |Payroll                                 |NULL       |1700       
160        |Benefits                                |NULL       |1700       
50         |Shipping                                |121        |1500       
170        |Manufacturing                           |NULL       |1700       
60         |IT                                      |103        |1400       
180        |Construction                            |NULL       |1700       
70         |Public Relations                        |204        |2700       
190        |Contracting                             |NULL       |1700       
80         |Sales                                   |145        |2500       
200        |Operations                              |NULL       |1700       
90         |Executive                               |100        |1700       
210        |IT Support                              |NULL       |1700       
100        |Finance                                 |108        |1700       
220        |NOC                                     |NULL       |1700       
110        |Accounting                              |205        |1700       

27 rows selected 

More info on derby-style functions can be found here.

http://db.apache.org/derby/docs/10.4/devguide/cdevspecialtabfuncs.html

Monday, 12 March 2012

vFabric hyperic application monitoring

vFabric Hyperic, a stand-alone component of vFabric Application Performance Manager, helps web operations teams monitor the application infrastructure for custom web applications across physical machines, a virtual infrastructure environment, or the cloud. By providing immediate notification of application performance degradation or unavailability, Hyperic enables system administrators ensure availability and reliability of critical business applications. With out-of-the-box monitoring of application metrics, app servers, web servers, databases, messaging servers, authentication systems, guest operating systems, virtual machines (VMs), vSphere ESX hosts, and more, you'll have single-pane visibility into your entire application stack regardless of where it is deployed.

Below are some screen shots showing the main dashboard page of hyperic along with it's integration into vSphere.


Friday, 17 February 2012

Using SQLFire sqlf CLI to connect to Oracle from a MAC OS-X Lion

Being on a MAC OS-X lion there is no client install for oracle to give me SQLPlus for example. BUT I can use sqlf command line client to connect to oracle to give me the ability to execute SQL much like SQLPLus. Handy indeed even if it isn't exactly SQLPlus it's good ienough for what I need when accessing my Oracle database from my MAC OS-X lion given thier is no oracle client for 11g on that platform.

It's done as shown below.

1. Add oracle JDBC driver to your classpath

export CLASSPATH=/Users/papicella/vmware/jdbcdrivers/11.2/ojdbc6.jar

2. Setup PATH to include SQLFire in your path as shown below.

export PATH=/Users/papicella/vmware/software/sqlfire/vFabric_SQLFire_101/bin:$PATH

3. Run some Oracle SQL as shown below using sql which shows how to load the driver, connect and run some SQL.

[Fri Feb 17 09:04:11 papicella@:~ ] $ sqlf
sqlf version 10.4
sqlf> driver 'oracle.jdbc.OracleDriver';
sqlf> connect 'jdbc:oracle:thin:scott/tiger@172.16.101.70:1521/linux11gr2';
sqlf> select to_char(sysdate, 'dd-MON-yyyy HH24:MI:SS') "Todays Date" from dual;
Todays Date                  
-----------------------------
17-FEB-2012 09:03:34         

1 row selected
sqlf> select * from dept;
DEPTNO                |DNAME         |LOC          
---------------------------------------------------
10                    |ACCOUNTING    |NEW YORK     
20                    |RESEARCH      |DALLAS       
30                    |SALES         |CHICAGO      
40                    |OPERATIONS    |BOSTON       

4 rows selected
sqlf> select * from v$version;
BANNER                                                                          
--------------------------------------------------------------------------------
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production    
PL/SQL Release 11.2.0.2.0 - Production                                          
CORE 11.2.0.2.0 Production                                                      
TNS for Linux: Version 11.2.0.2.0 - Production                                  
NLSRTL Version 11.2.0.2.0 - Production                                          

5 rows selected
sqlf>

Tuesday, 7 February 2012

How to offload data from an existing data store and loading it into SQLFire

A common requirement for using SQLFire is offloading data from an existing data store and loading it into SQLFire. This example show how you can load data into a SQLFire distributed system using Spring Batch from a CSV file and then how to present the data from a simple Spring MVC application. Finally the data is persisted to each SQLFire member to ensure we only need to load the data once and it will always be retained upon restarts of the distributed system.

The full demo can be viewed on the VMware vFabric SQLFire blog using the link below.

http://blogs.vmware.com/sqlfire/2012/02/sqlfire-demo-loading-the-afl-2012-fixture-data.html

Sunday, 5 February 2012

SQLFire locator equals Load Balancing + Failover

In this demo we show how not only does a locator provide load balancing among the SQLFire members from client connections but also high availability should a member crash or perhaps be brought down. This is illustrated below when using a JDBC Connection pool.

1. First we have started a locator and 2 SQLFire members as shown below.

Locator:
sqlf locator start -peer-discovery-address=localhost -peer-discovery-port=41111 -client-bind-address=localhost -client-port=1527

Members:

sqlf server start -server-groups=MYGROUP -locators=localhost[41111] -client-bind-address=localhost -client-port=1528 -dir=server1 &
sqlf server start -server-groups=MYGROUP -locators=localhost[41111] -client-bind-address=localhost -client-port=1529 -dir=server2 &

2. You can verify this by issuing a query as shown below , prior to running a JDBC Connection pool test.

Note: Notice how we connect to the locator client hostname/port rather then the individual SQLFire members.
[Sun Feb 05 21:38:50 papicella@:~/vmware/software/sqlfire/vFabric_SQLFire_101/pasdemos/simpledemo ] $ sqlf
sqlf version 10.4
sqlf> connect client 'localhost:1527';
sqlf> select substr(netservers, 1, 30) as "Netservers", kind from sys.members;
Netservers                    |KIND            
-----------------------------------------------
localhost/127.0.0.1[1529]     |datastore(norma&
localhost/127.0.0.1[1528]     |datastore(norma&
localhost/127.0.0.1[1527]     |locator(normal) 

3 rows selected

3. Here is our JDBC Connection Pool which shows that it's connecting to the locator itself to establish JDBC Conections.
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd">

  <bean id="sqlfireUcpDataSource" class="oracle.ucp.jdbc.PoolDataSourceFactory" factory-method="getPoolDataSource">
    <property name="URL" value="jdbc:sqlfire://localhost:1527/" />
    <property name="connectionFactoryClassName" value="com.vmware.sqlfire.jdbc.ClientDriver" />
    <property name="connectionPoolName" value="SQLFIRE_UCP_POOL" />
    <property name="minPoolSize" value="5" />
    <property name="maxPoolSize" value="20" />
    <property name="initialPoolSize" value="5" />
  </bean>
  
</beans>

4. At this point we create a test class which simply gets 5 connections and displays which SQLFire member we are connected to. So the process is as follows to test both load balancing and high availability when a member disappears, ensuring that existing connections fail over to the remaining SQLFire member.

- start JDBC Connection pool
- get 5 connections
- display each connection server side SQLFire member (It should load balance between our servers in this case we have 2 of them)
- Sleep for 10 seconds
- get 5 connections (verify that indeed we are only connected to the surviving member)

5. Output as follows.

Feb 5, 2012 9:59:26 PM org.springframework.context.support.AbstractApplicationContext prepareRefresh
INFO: Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@657d5d2a: startup date [Sun Feb 05 21:59:26 EST 2012]; root of context hierarchy
Feb 5, 2012 9:59:26 PM org.springframework.beans.factory.xml.XmlBeanDefinitionReader loadBeanDefinitions
INFO: Loading XML bean definitions from class path resource [application-config-SQLFIRE.xml]
Feb 5, 2012 9:59:26 PM org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletons
INFO: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@62da3a1e: defining beans [sqlfireUcpDataSource]; root of factory hierarchy
Feb 5, 2012 9:59:27 PM vmware.au.sqlfire.pool.TestSQLFireJDBCPool run
INFO: $Proxy0@1dfd868
Feb 5, 2012 9:59:27 PM vmware.au.sqlfire.pool.TestSQLFireJDBCPool run
INFO: Getting 5 connections from pool
Connection 0 [id=192-168-1-4.tpgi.com.au(1426):55091/50932, netserver=localhost/127.0.0.1[1528]]
Connection 1 [id=192-168-1-4.tpgi.com.au(1426):55091/50932, netserver=localhost/127.0.0.1[1528]]
Connection 2 [id=192-168-1-4.tpgi.com.au(1425):52022/50935, netserver=localhost/127.0.0.1[1529]]
Connection 3 [id=192-168-1-4.tpgi.com.au(1425):52022/50935, netserver=localhost/127.0.0.1[1529]]
Connection 4 [id=192-168-1-4.tpgi.com.au(1426):55091/50932, netserver=localhost/127.0.0.1[1528]]

Feb 5, 2012 9:59:27 PM vmware.au.sqlfire.pool.TestSQLFireJDBCPool run
INFO: Closing all connections from pool
Feb 5, 2012 9:59:27 PM vmware.au.sqlfire.pool.TestSQLFireJDBCPool run
INFO: ** Pool Details **
NumberOfAvailableConnections: 5
BorrowedConnectionsCount: 0


Sleeping for 20 seconds, shutdown a SQLFire server at this point...

6. Shutdown a server as shown below , before the existing program wakes up..

[Sun Feb 05 21:45:34 papicella@:~/vmware/software/sqlfire/vFabric_SQLFire_101/pasdemos/simpledemo ] $ sqlf server stop -dir=server1
The SQLFire Server has stopped.

7. Verify final output as follows

....

Feb 5, 2012 9:59:47 PM vmware.au.sqlfire.pool.TestSQLFireJDBCPool run
INFO: $Proxy0@580754fc
Feb 5, 2012 9:59:47 PM vmware.au.sqlfire.pool.TestSQLFireJDBCPool run
INFO: Getting 5 connections from pool
Connection 0 [id=192-168-1-4.tpgi.com.au(1425):52022/50935, netserver=localhost/127.0.0.1[1529]]
Connection 1 [id=192-168-1-4.tpgi.com.au(1425):52022/50935, netserver=localhost/127.0.0.1[1529]]
Connection 2 [id=192-168-1-4.tpgi.com.au(1425):52022/50935, netserver=localhost/127.0.0.1[1529]]
Connection 3 [id=192-168-1-4.tpgi.com.au(1425):52022/50935, netserver=localhost/127.0.0.1[1529]]
Connection 4 [id=192-168-1-4.tpgi.com.au(1425):52022/50935, netserver=localhost/127.0.0.1[1529]]

Feb 5, 2012 9:59:47 PM vmware.au.sqlfire.pool.TestSQLFireJDBCPool run
INFO: Closing all connections from pool
Feb 5, 2012 9:59:47 PM vmware.au.sqlfire.pool.TestSQLFireJDBCPool run
INFO: ** Pool Details **
NumberOfAvailableConnections: 5
BorrowedConnectionsCount: 0


As you can see from the output above we have switched over all existing connections in our JDBC pool to use the remaining SQLFire member only.

The test class used here is as follows.
package vmware.au.sqlfire.pool;

import java.sql.Connection;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.ArrayList;
import java.util.List;
import java.util.logging.Logger;

import oracle.ucp.jdbc.PoolDataSource;

import org.springframework.context.ApplicationContext;
import org.springframework.context.support.ClassPathXmlApplicationContext;

public class TestSQLFireJDBCPool 
{
 private Logger log = Logger.getLogger("com.vmware.sqlfire");
 private ApplicationContext context;
 private static final String BEAN_NAME = "sqlfireUcpDataSource";
 private PoolDataSource pds;
 
 public TestSQLFireJDBCPool() 
 {
  context = new ClassPathXmlApplicationContext("application-config-SQLFIRE.xml");
  pds = (PoolDataSource) context.getBean(BEAN_NAME);  
 }

 public void run()
 {
  Connection conn = null;
  List<Connection> connections = new ArrayList<Connection>();
  
  try 
  {
   // ensure pool is started by getting a connection object
   conn = pds.getConnection();
   log.info(conn.toString());
   conn.close();
   
   log.info("Getting 5 connections from pool");
   
   for (int i = 0; i < 5; i++)
   {
    connections.add(pds.getConnection());
   }

   for (int i = 0; i < 5; i++)
   {
    getInstanceDetails(connections.get(i), i);
   }
   
   log.info("Closing all connections from pool");
   
   for (Connection connection: connections)
   {
     connection.close(); 
   }
   
   log.info(displayPoolDetails());
   
 
  } 
  catch (SQLException e) 
  {
   // TODO Auto-generated catch block
   e.printStackTrace();
  } 
  finally
  {
   connections = null;
  }
 }

 public void getInstanceDetails (Connection conn, int i) throws SQLException
 {
  Statement stmt = null;
  PreparedStatement pstmt = null;
  ResultSet rset = null;
  String idSql = "select dsid() from sysibm.sysdummy1";
  String netSeverSql = "select netservers from sys.members where id = ?";
  
  stmt = conn.createStatement();
  rset = stmt.executeQuery(idSql);
  rset.next();
  
  String id = rset.getString(1);
  
  rset = null;
  pstmt = conn.prepareStatement(netSeverSql);
  pstmt.setString(1, id);
  rset = pstmt.executeQuery();
  rset.next();
  
  String netServer = rset.getString(1);
  
  System.out.println
    (String.format("Connection %s [id=%s, netserver=%s]", i, id, netServer));
  
  rset.close();
  stmt.close();
  pstmt.close();
  
 }
   
    public String displayPoolDetails () throws SQLException
    {
      StringBuffer sb = new StringBuffer();
      
      sb.append("** Pool Details **\n");
      sb.append("NumberOfAvailableConnections: " +
                         pds.getAvailableConnectionsCount());
      sb.append("\nBorrowedConnectionsCount: " +
                         pds.getBorrowedConnectionsCount());
      sb.append("\n");
      
      return sb.toString();
    }
    
 /**
  * @param args
  * @throws InterruptedException 
  */
 public static void main(String[] args) throws InterruptedException 
 {
  TestSQLFireJDBCPool test = new TestSQLFireJDBCPool();
  test.run();
        System.out.println("\nSleeping for 20 seconds, shutdown a SQLFire server at this point...\n");
  Thread.sleep(20000);
  test.run();
 }

}

Saturday, 28 January 2012

Automatic Client Failover within vFabric SQLFire

If your connected to a SQLFire member who unexpectedly disapears or is shutdown SQLFire will automatically ensure that client swicthes to another server within the distributed system without the need to reconnect as demonstrated using SQLFire command line client "sqlf".

Lets assume we have started 2 SQLFire members as follows

sqlf server start -server-groups=MYGROUP -dir=server1 -client-port=1527 -mcast-port=12333 &
sqlf server start -server-groups=MYGROUP -dir=server2 -client-port=1528 -mcast-port=12333 &

1. Lets connect to the first member as shown below
[Sat Jan 28 22:14:59 papicella@:~/vmware/software/sqlfire/vFabric_SQLFire_101/pasdemos/afl2011 ] $ sqlf
sqlf version 10.4
sqlf> connect client 'localhost:1527';
sqlf>

2. Now lets ensure we are indeed connected to the first member as shown below.
sqlf> show connections;
CONNECTION0* -  jdbc:sqlfire://localhost:1527/
* = current connection
sqlf>

3. Lets run a query against a table in the distributed system
sqlf> show connections;
sqlf> select team_id, name, long_name from teams;
TEAM_ID    |NAME           |LONG_NAME                     
----------------------------------------------------------
18         |Giants         |GWS GIANTS                    
1          |Tigers         |Richmond                      
16         |Bulldogs       |Western Bulldogs              
12         |Power          |Port Adelaide                 
5          |Pies           |Collingwood                   
17         |Suns           |Gold Coast Suns               
4          |Blues          |Carlton                       
2          |Crows          |Adelaide Crows                
10         |Roos           |North Melbourne               
13         |Saints         |St. Kilda                     
8          |Cats           |Geelong Cats                  
14         |Swans          |Sydney Swans                  
15         |Eagles         |West Coast Eagles             
11         |Demons         |Melbourne                     
7          |Dockers        |Fremantle                     
9          |Hawks          |Hawthorn                      
3          |Lions          |Brisbane Lions                
6          |Bombers        |Essendon                      

18 rows selected
sqlf> 

4. At this point we will shutdown the first SQLFire member who accepted the connection above using it's client port of 1527.

[Sat Jan 28 22:20:24 papicella@:~/vmware/software/sqlfire/vFabric_SQLFire_101/pasdemos/afl2011 ] $ sqlf server stop -dir=server1
The SQLFire Server has stopped.

5. Now lets re-run the query and verify we are still connected to the distributed system and can still display table data
sqlf> select team_id, name, long_name from teams;
TEAM_ID    |NAME           |LONG_NAME                     
----------------------------------------------------------
18         |Giants         |GWS GIANTS                    
1          |Tigers         |Richmond                      
16         |Bulldogs       |Western Bulldogs              
12         |Power          |Port Adelaide                 
5          |Pies           |Collingwood                   
17         |Suns           |Gold Coast Suns               
4          |Blues          |Carlton                       
2          |Crows          |Adelaide Crows                
10         |Roos           |North Melbourne               
13         |Saints         |St. Kilda                     
8          |Cats           |Geelong Cats                  
14         |Swans          |Sydney Swans                  
15         |Eagles         |West Coast Eagles             
11         |Demons         |Melbourne                     
7          |Dockers        |Fremantle                     
9          |Hawks          |Hawthorn                      
3          |Lions          |Brisbane Lions                
6          |Bombers        |Essendon                      

18 rows selected
sqlf> 

6. Finally lets ensure we indeed did switch to the remaining member which accepts client connections on 1528
sqlf> show connections;
CONNECTION0* -  jdbc:sqlfire://localhost:1528/
* = current connection

For more information on SQLFire see the link below.

http://www.vmware.com/products/application-platform/vfabric-sqlfire/overview.html

Thursday, 12 January 2012

Using a Connection Pool with SQLFire

A Java application can use the JDBC thin driver to access a single member of a SQLFire cluster and execute SQL statements. This makes it a perfect candidate to use a connection pool to maintain connections for clients rather then connect to it individually. Two that spring to mind are Apache DBCP or Oracle's UCP. In this example we show a configuration using Spring and Oracle UCP which creates a pool of SQLFire connections which clients can then use as required.

1. Create a new spring bean application config file with a UCP defined as follows.
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd">

  <bean id="sqlfireUcpDataSource" class="oracle.ucp.jdbc.PoolDataSourceFactory" factory-method="getPoolDataSource">
    <property name="URL" value="jdbc:sqlfire://localhost:1527/" />
    <property name="connectionFactoryClassName" value="com.vmware.sqlfire.jdbc.ClientDriver" />
    <property name="connectionPoolName" value="SQLFIRE_UCP_POOL" />
    <property name="minPoolSize" value="5" />
    <property name="maxPoolSize" value="20" />
    <property name="initialPoolSize" value="5" />
  </bean>
  
</beans>

2. Create a test class as follows to verify the pool as shown below.
package vmware.au.sqlfire.pool;

import java.sql.Connection;
import java.sql.SQLException;
import java.util.ArrayList;
import java.util.List;
import java.util.logging.Logger;

import oracle.ucp.jdbc.PoolDataSource;

import org.springframework.context.ApplicationContext;
import org.springframework.context.support.ClassPathXmlApplicationContext;

public class TestSQLFireJDBCPool 
{
 private Logger log = Logger.getLogger("com.vmware.sqlfire");
 private ApplicationContext context;
 private static final String BEAN_NAME = "sqlfireUcpDataSource";
 private PoolDataSource pds;
 
 public TestSQLFireJDBCPool() 
 {
  context = new ClassPathXmlApplicationContext("application-config-SQLFIRE.xml");
  pds = (PoolDataSource) context.getBean(BEAN_NAME);  
 }

 public void run()
 {
  Connection conn = null;
  List<Connection> connections = new ArrayList<Connection>();
  
  try 
  {
   // ensure pool is started by getting a connection object
   conn = pds.getConnection();
   log.info(conn.toString());
   conn.close();
   
   log.info("Getting 5 connections from pool");
   
   for (int i = 0; i < 5; i++)
   {
    connections.add(pds.getConnection());
   }
   
   log.info(displayPoolDetails());
   log.info("Closing all connections from pool");
   
   for (Connection connection: connections)
   {
     connection.close(); 
   }
   
   log.info(displayPoolDetails());
   
 
  } 
  catch (SQLException e) 
  {
   // TODO Auto-generated catch block
   e.printStackTrace();
  } 
  finally
  {
   connections = null;
  }
 }

    public String displayPoolDetails () throws SQLException
    {
      StringBuffer sb = new StringBuffer();
      
      sb.append("** Pool Details **\n");
      sb.append("NumberOfAvailableConnections: " +
                         pds.getAvailableConnectionsCount());
      sb.append("\nBorrowedConnectionsCount: " +
                         pds.getBorrowedConnectionsCount());
      sb.append("\n");
      
      return sb.toString();
    }
    
 /**
  * @param args
  */
 public static void main(String[] args) 
 {
  TestSQLFireJDBCPool test = new TestSQLFireJDBCPool();
  test.run();

 }

}

3. You will need to use the following JAR files such as Spring, Oracle UCP etc..


Output as follows:

Jan 12, 2012 10:11:54 PM org.springframework.context.support.AbstractApplicationContext prepareRefresh
INFO: Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@657d5d2a: startup date [Thu Jan 12 22:11:54 EST 2012]; root of context hierarchy
Jan 12, 2012 10:11:54 PM org.springframework.beans.factory.xml.XmlBeanDefinitionReader loadBeanDefinitions
INFO: Loading XML bean definitions from class path resource [application-config-SQLFIRE.xml]
Jan 12, 2012 10:11:55 PM org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletons
INFO: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@62da3a1e: defining beans [sqlfireUcpDataSource]; root of factory hierarchy
Jan 12, 2012 10:11:55 PM vmware.au.sqlfire.pool.TestSQLFireJDBCPool run
INFO: $Proxy0@1dc18a4c
Jan 12, 2012 10:11:55 PM vmware.au.sqlfire.pool.TestSQLFireJDBCPool run
INFO: Getting 5 connections from pool
Jan 12, 2012 10:11:55 PM vmware.au.sqlfire.pool.TestSQLFireJDBCPool run
INFO: ** Pool Details **
NumberOfAvailableConnections: 0
BorrowedConnectionsCount: 5

Jan 12, 2012 10:11:55 PM vmware.au.sqlfire.pool.TestSQLFireJDBCPool run
INFO: Closing all connections from pool
Jan 12, 2012 10:11:55 PM vmware.au.sqlfire.pool.TestSQLFireJDBCPool run
INFO: ** Pool Details **
NumberOfAvailableConnections: 5
BorrowedConnectionsCount: 0

Using DBVisualizer to connect to a SQLFire cluster

There are many database development visual tools out there which more often then not allow you to connect to the many different RDBMS databases including Oracle, MySQL, etc.. In this post here we use DBVisualizer to connect to SQLFire and show how easy you can view/manipulate your distributed SQLFire database from a visual tool.

Note: It's assumed you already have DbVisualizer installed and running.

First thing we need to do is add SQLFire driver JAR file to the list of drivers DBVisualizer can use.

1. Select Tools -> Driver Manager
2. Click the + symbol to define a new driver and enter details as shown below.



Select the com.vmware.sqlfire.jdbc.Driver class

Note: sqlfireclient.jar can be found in $SQLFIRE_HOME/lib directory

In order to use the client driver, you must specify a JDBC connection URL for your SQLFire distributed system. The basic URL format for the client driver is: jdbc:sqlfire://hostname:port/
where hostname and port correspond to the -client-bind-address and -client-port value of a SQLFire server or locator in your distributed system.

3. Close the window when done
4. Right click on "Connections" icon and select "Create Database Connection".
5. If promoted elect "Use Wizard"
6. Enter a connection name in this demo I enter -> pas-sqlfire
7. Select the database driver we created at step #2 which should be "SQLFire 1.0"
8. Click next
9. Enter in connection details as shown below.


If authentication is disabled, then you can specify any temporary username and password value into these fields.

Note: SQLFire uses the username specified in the JDBC connection as the schema name when you do not provide the schema name for a database object. SQLFire uses "APP" as the default schema. If your system does not enable authentication, you can specify "APP" for both the username and password to maintain consistency with the default schema behavior. 
 
 10. Click finish

The following shows a query we run to view the distributed database SQLFire members.


For more information on SQLFire see the link below.

http://www.vmware.com/products/application-platform/vfabric-sqlfire/overview.html