Spring XD Pivotal Gemfire Sink Demo

Spring XD is a unified, distributed, and extensible system for data ingestion, real time analytics, batch processing, and data export. The project's goal is to simplify the development of big data applications.

There are 2 implementation of the gemfire sink: gemfire-server and gemfire-json-server. They are identical except the latter converts JSON string payloads to a JSON document format proprietary to GemFire and provides JSON field access and query capabilities. If you are not using JSON, the gemfire-server module will write the payload using java serialization to the configured region.

In this example below we show how we connect to an existing GemFire 7.0.2 cluster using a locator to add some JSON trade symbols to an existing region in the cluster.

1. Start a GemFire cluster with with an existing region as shown below. The following cache.xml is for "server1" of the cluster and "server2" of the cluster. They are identical configs , just using different ports

server1 cache.xml

  
<?xml version="1.0"?>
<!DOCTYPE cache PUBLIC
    "-//GemStone Systems, Inc.//GemFire Declarative Caching 7.0//EN"
    "http://www.gemstone.com/dtd/cache7_0.dtd">

<cache>
    <cache-server bind-address="localhost" port="40404" hostname-for-clients="localhost"/>

    <region name="springxd-region">
        <region-attributes data-policy="partition">
           <partition-attributes redundant-copies="1" total-num-buckets="113"/>
           <eviction-attributes>
                <lru-heap-percentage action="overflow-to-disk"/>
           </eviction-attributes>
        </region-attributes>
    </region>

    <resource-manager critical-heap-percentage="75" eviction-heap-percentage="65"/>

</cache>

server2 cache.xml

  
<?xml version="1.0"?>
<!DOCTYPE cache PUBLIC
    "-//GemStone Systems, Inc.//GemFire Declarative Caching 7.0//EN"
    "http://www.gemstone.com/dtd/cache7_0.dtd">

<cache>
    <cache-server bind-address="localhost" port="40405" hostname-for-clients="localhost"/>

    <region name="springxd-region">
        <region-attributes data-policy="partition">
           <partition-attributes redundant-copies="1" total-num-buckets="113"/>
           <eviction-attributes>
                <lru-heap-percentage action="overflow-to-disk"/>
           </eviction-attributes>
        </region-attributes>
    </region>

    <resource-manager critical-heap-percentage="75" eviction-heap-percentage="65"/>

</cache> 

2. Verify using GFSH you have 2 members , a locator and a region as follows

  
$ gfsh
    _________________________     __
   / _____/ ______/ ______/ /____/ /
  / /  __/ /___  /_____  / _____  /
 / /__/ / ____/  _____/ / /    / /
/______/_/      /______/_/    /_/    v7.0.2.10

Monitor and Manage GemFire
gfsh>connect --locator=localhost[10334];
Connecting to Locator at [host=localhost, port=10334] ..
Connecting to Manager at [host=10.98.94.88, port=1099] ..
Successfully connected to: [host=10.98.94.88, port=1099]

gfsh>list members;
  Name   | Id
-------- | ---------------------------------------
server1  | 10.98.94.88(server1:10161)<v1>:15610
server2  | 10.98.94.88(server2:10164)<v2>:39300
locator1 | localhost(locator1:10159:locator):42885

gfsh>list regions;
List of regions
---------------
springxd-region

3. Start single node SpringXD server

  
[Wed Oct 08 14:51:06 papicella@:~/vmware/software/spring/spring-xd/spring-xd-1.0.1.RELEASE ] $ xd-singlenode

 _____                           __   _______
/  ___|          (-)             \ \ / /  _  \
\ `--. _ __  _ __ _ _ __   __ _   \ V /| | | |
 `--. \ '_ \| '__| | '_ \ / _` |  / ^ \| | | |
/\__/ / |_) | |  | | | | | (_| | / / \ \ |/ /
\____/| .__/|_|  |_|_| |_|\__, | \/   \/___/
      | |                  __/ |
      |_|                 |___/
1.0.1.RELEASE                    eXtreme Data


Started : SingleNodeApplication
Documentation: https://github.com/spring-projects/spring-xd/wiki

.... 

4. Start SpringXD shell

  
$ xd-shell
 _____                           __   _______
/  ___|          (-)             \ \ / /  _  \
\ `--. _ __  _ __ _ _ __   __ _   \ V /| | | 
 `--. \ '_ \| '__| | '_ \ / _` |  / ^ \| | | |
/\__/ / |_) | |  | | | | | (_| | / / \ \ |/ /
\____/| .__/|_|  |_|_| |_|\__, | \/   \/___/
      | |                  __/ |
      |_|                 |___/
eXtreme Data
1.0.1.RELEASE | Admin Server Target: http://localhost:9393
Welcome to the Spring XD shell. For assistance hit TAB or type "help".
xd:>

5. Create a stream as follows

  
xd:>stream create --name gemfiredemo --definition "http --port=9090 | gemfire-json-server --host=localhost --port=10334 --useLocator=true --regionName=springxd-region --keyExpression=payload.getField('symbol')" --deploy
Created and deployed new stream 'gemfiredemo'

6. Post some entries via HTTP which will be inserted into the GemFire Region

  
xd:>http post --target http://localhost:9090 --data {"symbol":"ORCL","price":38}
> POST (text/plain;Charset=UTF-8) http://localhost:9090 {"symbol":"ORCL","price":38}
> 200 OK

xd:>http post --target http://localhost:9090 --data {"symbol":"VMW","price":94}
> POST (text/plain;Charset=UTF-8) http://localhost:9090 {"symbol":"VMW","price":94}
> 200 OK  

7. Verify via GFSH that data has been inserted into the GemFire region. JSON data stored in GemFire regions is done using PDX.

  
gfsh>query --query="select * from /springxd-region";

Result     : true
startCount : 0
endCount   : 20
Rows       : 2

symbol | price
------ | -----
ORCL   | 38
VMW    | 94

NEXT_STEP_NAME : END

More Information

SpringXD
http://projects.spring.io/spring-xd/

GemFire Sinks
http://docs.spring.io/spring-xd/docs/1.0.1.RELEASE/reference/html/#gemfire-server