Monday, 14 October 2013

Pivotal SQLFire Rowloader for Greenplum

When you use SQLFire as a cache, you can configure a SQL data loader that is triggered to load data from a backend repository on a miss in SQLFire. When an incoming query request for a uniquely-identified row cannot be satisfied by the distributed cache, the loader is invoked to retrieve the data from an external source. SQLFire locks the associated row and prevents concurrent readers that are trying to fetch the same row from overloading the backend database.

The example below is using Greenplum as the main source database. The example is based on this MYSQL example

1. In this example we use a Greenplum table as follows, showing first 10 rows only, 500K of rows exist.

gpadmin=# SELECT * FROM apples.person limit 10;
 id |   name   
  2 | person2
  4 | person4
  6 | person6
  8 | person8
 10 | person10
 12 | person12
 14 | person14
 16 | person16
 18 | person18
 20 | person20
(10 rows)

Time: 11.947 ms

2. Add Postgress JDBC driver to the classpath of the SQLFire server nodes as well as the examples shipped with SQLFire in this case called "examples.jar". Example shown in the post above.

[Mon Oct 14 22:17:33 papicella@:~/sqlfire/final11build/vFabric_SQLFire_111_b42624/pasdemos/fire-rowloader/lib ] $ d
total 1248
-rw-r--r--@  1 papicella  staff  579785 13 Oct 20:01 postgresql-9.2-1002.jdbc4.jar
-rw-r--r--   1 papicella  staff   57203 13 Oct 20:05 examples.jar
drwxr-xr-x   4 papicella  staff     136 13 Oct 20:05 ./
drwxr-xr-x  16 papicella  staff     544 13 Oct 22:01 ../

3. From the table above that exists in a schema called "apples" so we must create the same table in SQLFire using the same schema name as shown below.
create schema apples;

set schema apples;

(id int, name varchar(200), primary key (id))

show tables in apples;

4. Create a row loader as shown below.
set schema apples;

call sys.attach_loader('APPLES', 'PERSON', 'examples.JDBCRowLoader', '|url=jdbc:postgresql://|query-string=SELECT * FROM apples.person WHERE id=?|user=pas|password=pas|min-connections=5|max-connections=100'); 

5. Run a test as shown below.
sqlf> set schema apples;
0 rows inserted/updated/deleted
sqlf> select * from person where id = 100;
ID         |NAME                                                                                                                            
100        |person100                                                                                                                       

1 row selected
sqlf> select * from person;
ID         |NAME                                                                                                                            
100        |person100                                                                                                                       

1 row selected  

6. Finally the log file will show the connections being established , in this case 5 as a Minimum.
[info 2013/10/14 21:47:16.600 EST  <Pooled Waiting Message Processor 2> tid=0x52] (tid=11 msgId=1) JDBCRowLoader initialized.

[info 2013/10/14 21:47:16.600 EST  <Pooled Waiting Message Processor 2> tid=0x52] (tid=11 msgId=2)    user: pas

[info 2013/10/14 21:47:16.600 EST  <Pooled Waiting Message Processor 2> tid=0x52] (tid=11 msgId=3)    password: xxx

[info 2013/10/14 21:47:16.600 EST  <Pooled Waiting Message Processor 2> tid=0x52] (tid=11 msgId=4)    url: jdbc:postgresql://

[info 2013/10/14 21:47:16.601 EST  <Pooled Waiting Message Processor 2> tid=0x52] (tid=11 msgId=5)    min-connections: 5

[info 2013/10/14 21:47:16.601 EST  <Pooled Waiting Message Processor 2> tid=0x52] (tid=11 msgId=6)    max-connections: 100

[info 2013/10/14 21:47:16.601 EST  <Pooled Waiting Message Processor 2> tid=0x52] (tid=11 msgId=7)    query-string: SELECT * FROM apples.person WHERE id=?

[info 2013/10/14 21:47:16.604 EST  <Pooled Waiting Message Processor 2> tid=0x52] AbstractSqlfReplayableMessage: Successfully executed message with fields: SqlfSetLoaderMessage@6127ffd7(processorId=53; processorType=77;posDup=false; replayKey=201; schema = APPLES; table = PERSON; implementation = examples.JDBCRowLoader; initInfoStr = |url=jdbc:postgresql://|query-string=SELECT * FROM apples.person WHERE id=?|user=pas|password=pas|min-connections=5|max-connections=100)

[info 2013/10/14 21:48:55.513 EST  <Pooled Waiting Message Processor 3> tid=0x53] Initializing region _B__APPLES_PERSON_100

[info 2013/10/14 21:48:55.530 EST  <PartitionedRegion Message Processor2> tid=0x56] (tid=12 msgId=8) JDBCRowLoader invoked to fetch from schema <APPLES> on table <PERSON>.

[info 2013/10/14 21:48:55.530 EST  <PartitionedRegion Message Processor2> tid=0x56] (tid=12 msgId=9)  primary key element 0: 100

[info 2013/10/14 21:48:55.566 EST  <pool-1-thread-6> tid=0x5c] (tid=13 msgId=10)  Successful connection to target database: jdbc:postgresql://

[info 2013/10/14 21:48:55.566 EST  <pool-1-thread-1> tid=0x57] (tid=15 msgId=12)  Successful connection to target database: jdbc:postgresql://

[info 2013/10/14 21:48:55.566 EST  <pool-1-thread-2> tid=0x58] (tid=18 msgId=15)  Successful connection to target database: jdbc:postgresql://

[info 2013/10/14 21:48:55.566 EST  <pool-1-thread-5> tid=0x5b] (tid=14 msgId=11)  Successful connection to target database: jdbc:postgresql://

[info 2013/10/14 21:48:55.566 EST  <pool-1-thread-3> tid=0x59] (tid=16 msgId=13)  Successful connection to target database: jdbc:postgresql://

[info 2013/10/14 21:48:55.566 EST  <pool-1-thread-4> tid=0x5a] (tid=17 msgId=14)  Successful connection to target database: jdbc:postgresql://

[info 2013/10/14 21:48:55.791 EST  <PartitionedRegion Message Processor2> tid=0x56] (tid=12 msgId=16) Executing query SELECT * FROM apples.person WHERE id=100

[info 2013/10/14 21:48:55.829 EST  <PartitionedRegion Message Processor2> tid=0x56] (tid=12 msgId=17) Query succeeded 

For more information refer to the link below.

Friday, 4 October 2013

Spring MVC Rest - Content Negotiation using JSON/XML

In this example we will show how we can determine which  data format to return  writing just a single controller method. In this example we either return XML or JSON and the system knows whether to convert to XML or JSON because of content negotiation. This example is based on the example provided by Paul on the link below.

I have taken what Paul described into a working example of my own using Content Negotiation with Spring MVC Rest support.

The project I created looks as follows

The complete code for this example exists on GitHub as follows

1. Create the required pom.xml dependancy elements as shown below.

<project xmlns="" xmlns:xsi="" xsi:schemaLocation="">

2. Create a Person class as follows


import javax.xml.bind.annotation.XmlRootElement;

public class Person implements Serializable

 private int id;
 private String firstName;
 private String lastName;
 public Person() 
  // TODO Auto-generated constructor stub

 public Person(int id, String firstName, String lastName) {
  super(); = id;
  this.firstName = firstName;
  this.lastName = lastName;

 public int getId() {
  return id;

 public void setId(int id) { = id;

 public String getFirstName() {
  return firstName;

 public void setFirstName(String firstName) {
  this.firstName = firstName;

 public String getLastName() {
  return lastName;

 public void setLastName(String lastName) {
  this.lastName = lastName;

 public String toString() {
  return "Person [id=" + id + ", firstName=" + firstName + ", lastName="
    + lastName + "]";


3. Create a People class which provides a list of Person objects, we do this so we can return XML as a List doesn't work directly with JAXB

import java.util.List;

import javax.xml.bind.annotation.XmlElement;
import javax.xml.bind.annotation.XmlRootElement;

public class People 
  private List<Person> people;
  protected People() {}   // Keep JAXB happy
  public People(List<Person> people)
     this.people = people;
    public List<Person> getPeople() 
     return people;


4. Create a controller as follows, there is a request mapping for HTML as well here with the same path, which is the default, in order not to duplicate code we call the JSON/XML method from the HTML method itself.

import java.util.Arrays;
import java.util.List;

import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import org.springframework.http.HttpStatus;
import org.springframework.stereotype.Controller;
import org.springframework.ui.Model;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.ResponseBody;
import org.springframework.web.bind.annotation.ResponseStatus;


public class PersonController 

 private static List<Person> personList;

  personList = 
    Arrays.asList(new Person[] 
      { new Person(1, "Pas", "Apicella"),
           new Person(2, "Lucia", "Apicella"),
           new Person(3, "Lucas", "Apicella"),
           new Person(4, "Siena", "Apicella")

              method = RequestMethod.GET, 
              produces={"application/xml", "application/json"})
    public @ResponseBody People listWithJSON() 
     return new People(personList);

    // View-based method
    @RequestMapping(value = "/people", method = RequestMethod.GET)
    public String listWithView(Model model, HttpServletResponse response, HttpServletRequest request) 
     // Call RESTful method to avoid repeating code
     model.addAttribute("peopleList", listWithJSON().getPeople());

     // Return the view to use for rendering the response
     return "people";

5. Create a Spring XML file as follows

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns=""

 <!-- Activates various annotations to be detected in bean classes -->
 <context:annotation-config />
 <!-- Scans the classpath for annotated components that will be auto-registered as Spring beans.
  For example @Controller and @Service. Make sure to set the correct base-package-->
 <context:component-scan base-package="" />
    <!-- Configures the annotation-driven Spring MVC Controller programming model.
  Note that, with Spring 3.0, this tag works in Servlet MVC only!  -->
 <bean id="cnManager"
  <property name="favorPathExtension" value="true"/>
  <property name="ignoreAcceptHeader" value="true" />
  <property name="defaultContentType" value="text/html" />
  <property name="useJaf" value="false"/>

  <property name="mediaTypes">
   <entry key="html" value="text/html" />
   <entry key="json" value="application/json" />
   <entry key="xml" value="application/xml" />

 <mvc:annotation-driven content-negotiation-manager="cnManager"/>

 <bean class="" />


6. Create a class to enable pretty print for JSON data

import org.springframework.beans.BeansException;
import org.springframework.beans.factory.config.BeanPostProcessor;
import org.springframework.http.converter.HttpMessageConverter;
import org.springframework.http.converter.json.MappingJackson2HttpMessageConverter;
import org.springframework.http.converter.json.MappingJacksonHttpMessageConverter;
* The HTTP Message converters are created automatically by Spring. To perform
* additional configuration we use a bean post-processor.
public class MvcConfiguringPostProcessor implements BeanPostProcessor {

 * Enable pretty print on any bean of type
 * {@link MappingJacksonHttpMessageConverter} or
 * {@link MappingJackson2HttpMessageConverter}.
 public Object postProcessBeforeInitialization(Object bean, String name) throws BeansException 
  if (bean instanceof HttpMessageConverter<?>)
  if (bean instanceof MappingJacksonHttpMessageConverter) { 
   ((MappingJacksonHttpMessageConverter) bean).setPrettyPrint(true);
  else if (bean instanceof MappingJackson2HttpMessageConverter) {
   ((MappingJackson2HttpMessageConverter) bean).setPrettyPrint(true);   
  return bean;

 public Object postProcessAfterInitialization(Object bean, String beanName) throws BeansException 
  // Nothing to do
  return bean;


7. Finally create a HTML view page

<%@ page language="java" contentType="text/html; charset=UTF-8"
<%@ taglib uri="" prefix="c"%>
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>Spring MVC Rest Demo</title>

<h2>Spring MVC Rest Demo</h2>

<table border="1">
  <th>First Name</th>
  <th>Last Name</th>
 <c:forEach var="row" varStatus="loop" items="${peopleList}">

<p />

Created by Pas Apicella


8. Run the application and access the 3 supported views with request mapping path of "/people"

HTML - http://localhost:8080/springmvc-rest/apples/people

JSON - http://localhost:8080/springmvc-rest/apples/people.json

XML - http://localhost:8080/springmvc-rest/apples/people.xml

Wednesday, 2 October 2013

Spring Data GemFire GemfireTemplate

As with many other high-level abstractions provided by the Spring projects, Spring Data GemFire provides a template that simplifies GemFire data access. The class provides several one-line methods, for common region operations but also the ability to execute code against the native GemFire API without having to deal with GemFire checked exceptions for example through the GemfireCallback. The template class requires a GemFire Region instance and once configured is thread-safe and should be reused across multiple classes:


In this example we are using and existing GemFire cluster which has DEPT / EMP objects already existing in the distributed system as shown below.

gfsh>query --query="select * from /departments";

Result     : true
startCount : 0
endCount   : 20
Rows       : 4

deptno | name
------ | ----------
30     | SALES
20     | RESEARCH


gfsh>query --query="select * from /employees";

Result     : true
startCount : 0
endCount   : 20
Rows       : 13

empno | deptno |   name   | job
----- | ------ | -------- | ---------
7380  | 40     | BLACK    | CLERK
7373  | 40     | SIENA    | CLERK
7377  | 20     | ADAM     | CLERK
7370  | 10     | APPLES   | MANAGER
7381  | 40     | BROWN    | SALESMAN
7379  | 10     | FRANK    | CLERK
7375  | 30     | ROB      | CLERK
7371  | 10     | APICELLA | SALESMAN
7374  | 10     | LUCAS    | SALESMAN
7378  | 20     | SALLY    | MANAGER
7372  | 30     | LUCIA    | PRESIDENT
7376  | 20     | ADRIAN   | CLERK
7369  | 20     | SMITH    | CLERK


1. client.xml (GemFire Client cache XML file)
<!DOCTYPE client-cache PUBLIC 
"-//GemStone Systems, Inc.//GemFire Declarative Caching 7//EN" 
    <parameter name="classes">
 <!-- No cache storage in the client region because of the PROXY client region shortcut setting. -->

    <region name="departments">
      <region-attributes refid="PROXY" pool-name="gfPool" />
    <region name="employees">
  <region-attributes refid="PROXY" pool-name="gfPool" />

2. application-context.xml
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns=""

     properties-ref="props" />
 <gfe:pool id="gfPool" max-connections="10">
  <gfe:locator host="localhost" port="10334"/>
 <gfe:lookup-region id="departments" name="departments" cache-ref="client-cache"/>
    <gfe:lookup-region id="employees" name="employees" cache-ref="client-cache"/>
    <gfe-data:repositories base-package=""  />
    <util:properties id="props" location=""/>
Here we simply import the client.xml file and then define the regions we wish to access from GemFire.


import java.util.Collection;

import org.springframework.context.ConfigurableApplicationContext;


import com.gemstone.gemfire.cache.Region;
import com.gemstone.gemfire.cache.query.SelectResults;

public class TestGemFireTemplate 

 private ConfigurableApplicationContext ctx = null;
 public TestGemFireTemplate() 
  ctx = new ClassPathXmlApplicationContext("application-context.xml");

 public void run ()
  GemfireTemplate empTemplate = new GemfireTemplate((Region) ctx.getBean("employees"));
  System.out.println("-> template.query() test \n ");
  SelectResults<?> results = empTemplate.query("deptno = 40");
  Collection<Employee> emps = (Collection<Employee>) results.asList();
  for (Employee e: emps)
  System.out.println("\n-> template.get(key) test \n ");
  Employee emp = empTemplate.get("7373");
  System.out.println("\n-> template.find() test \n ");
  SelectResults<Employee> clerkEmpResults = 
    empTemplate.find("SELECT * from /employees WHERE job=$1", "CLERK");
  Collection<Employee> clerkEmps = (Collection<Employee>) clerkEmpResults.asList();
  for (Employee e: clerkEmps)
 public static void main(String[] args) 
  // TODO Auto-generated method stub
  TestGemFireTemplate test = new TestGemFireTemplate();
  System.out.println("\nStarting Spring Data GemFire Template Test.... \n");;
  System.out.println("\nAll done.... ");

Output as shown below.

Starting Spring Data GemFire Template Test.... 

-> template.query() test 

Employee [empno=7380, name=BLACK, job=CLERK, deptno=40]
Employee [empno=7381, name=BROWN, job=SALESMAN, deptno=40]
Employee [empno=7373, name=SIENA, job=CLERK, deptno=40]

-> template.get(key) test 

Employee [empno=7373, name=SIENA, job=CLERK, deptno=40]

-> template.find() test 

Employee [empno=7369, name=SMITH, job=CLERK, deptno=20]
Employee [empno=7375, name=ROB, job=CLERK, deptno=30]
Employee [empno=7376, name=ADRIAN, job=CLERK, deptno=20]
Employee [empno=7380, name=BLACK, job=CLERK, deptno=40]
Employee [empno=7377, name=ADAM, job=CLERK, deptno=20]
Employee [empno=7379, name=FRANK, job=CLERK, deptno=10]
Employee [empno=7373, name=SIENA, job=CLERK, deptno=40]

All done....