Spring XD is a unified, distributed, and extensible system for data
ingestion, real time analytics, batch processing, and data export. The
project's goal is to simplify the development of big data applications.
There are 2 implementation of the gemfire sink:
gemfire-server and
gemfire-json-server.
They are identical except the latter converts JSON string payloads to a
JSON document format proprietary to GemFire and provides JSON field
access and query capabilities. If you are not using JSON, the
gemfire-server module will write the payload using java serialization to
the configured region.
In this example below we show how we connect to an existing GemFire 7.0.2 cluster using a locator to add some JSON trade symbols to an existing region in the cluster.
1. Start a GemFire cluster with with an existing region as shown below. The following cache.xml is for "server1" of the cluster and "server2" of the cluster. They are identical configs , just using different ports
server1 cache.xml
<?xml version="1.0"?>
<!DOCTYPE cache PUBLIC
"-//GemStone Systems, Inc.//GemFire Declarative Caching 7.0//EN"
"http://www.gemstone.com/dtd/cache7_0.dtd">
<cache>
<cache-server bind-address="localhost" port="40404" hostname-for-clients="localhost"/>
<region name="springxd-region">
<region-attributes data-policy="partition">
<partition-attributes redundant-copies="1" total-num-buckets="113"/>
<eviction-attributes>
<lru-heap-percentage action="overflow-to-disk"/>
</eviction-attributes>
</region-attributes>
</region>
<resource-manager critical-heap-percentage="75" eviction-heap-percentage="65"/>
</cache>
server2 cache.xml
<?xml version="1.0"?>
<!DOCTYPE cache PUBLIC
"-//GemStone Systems, Inc.//GemFire Declarative Caching 7.0//EN"
"http://www.gemstone.com/dtd/cache7_0.dtd">
<cache>
<cache-server bind-address="localhost" port="40405" hostname-for-clients="localhost"/>
<region name="springxd-region">
<region-attributes data-policy="partition">
<partition-attributes redundant-copies="1" total-num-buckets="113"/>
<eviction-attributes>
<lru-heap-percentage action="overflow-to-disk"/>
</eviction-attributes>
</region-attributes>
</region>
<resource-manager critical-heap-percentage="75" eviction-heap-percentage="65"/>
</cache>
2. Verify using GFSH you have 2 members , a locator and a region as follows
$ gfsh
_________________________ __
/ _____/ ______/ ______/ /____/ /
/ / __/ /___ /_____ / _____ /
/ /__/ / ____/ _____/ / / / /
/______/_/ /______/_/ /_/ v7.0.2.10
Monitor and Manage GemFire
gfsh>connect --locator=localhost[10334];
Connecting to Locator at [host=localhost, port=10334] ..
Connecting to Manager at [host=10.98.94.88, port=1099] ..
Successfully connected to: [host=10.98.94.88, port=1099]
gfsh>list members;
Name | Id
-------- | ---------------------------------------
server1 | 10.98.94.88(server1:10161)<v1>:15610
server2 | 10.98.94.88(server2:10164)<v2>:39300
locator1 | localhost(locator1:10159:locator):42885
gfsh>list regions;
List of regions
---------------
springxd-region
3. Start single node SpringXD server
[Wed Oct 08 14:51:06 papicella@:~/vmware/software/spring/spring-xd/spring-xd-1.0.1.RELEASE ] $ xd-singlenode
_____ __ _______
/ ___| (-) \ \ / / _ \
\ `--. _ __ _ __ _ _ __ __ _ \ V /| | | |
`--. \ '_ \| '__| | '_ \ / _` | / ^ \| | | |
/\__/ / |_) | | | | | | | (_| | / / \ \ |/ /
\____/| .__/|_| |_|_| |_|\__, | \/ \/___/
| | __/ |
|_| |___/
1.0.1.RELEASE eXtreme Data
Started : SingleNodeApplication
Documentation: https://github.com/spring-projects/spring-xd/wiki
....
4. Start SpringXD shell
$ xd-shell
_____ __ _______
/ ___| (-) \ \ / / _ \
\ `--. _ __ _ __ _ _ __ __ _ \ V /| | |
`--. \ '_ \| '__| | '_ \ / _` | / ^ \| | | |
/\__/ / |_) | | | | | | | (_| | / / \ \ |/ /
\____/| .__/|_| |_|_| |_|\__, | \/ \/___/
| | __/ |
|_| |___/
eXtreme Data
1.0.1.RELEASE | Admin Server Target: http://localhost:9393
Welcome to the Spring XD shell. For assistance hit TAB or type "help".
xd:>
5. Create a stream as follows
xd:>stream create --name gemfiredemo --definition "http --port=9090 | gemfire-json-server --host=localhost --port=10334 --useLocator=true --regionName=springxd-region --keyExpression=payload.getField('symbol')" --deploy
Created and deployed new stream 'gemfiredemo'
6. Post some entries via HTTP which will be inserted into the GemFire Region
xd:>http post --target http://localhost:9090 --data {"symbol":"ORCL","price":38}
> POST (text/plain;Charset=UTF-8) http://localhost:9090 {"symbol":"ORCL","price":38}
> 200 OK
xd:>http post --target http://localhost:9090 --data {"symbol":"VMW","price":94}
> POST (text/plain;Charset=UTF-8) http://localhost:9090 {"symbol":"VMW","price":94}
> 200 OK
7. Verify via GFSH that data has been inserted into the GemFire region. JSON data stored in GemFire regions is done using PDX.
gfsh>query --query="select * from /springxd-region";
Result : true
startCount : 0
endCount : 20
Rows : 2
symbol | price
------ | -----
ORCL | 38
VMW | 94
NEXT_STEP_NAME : END
More Information
SpringXD
http://projects.spring.io/spring-xd/
GemFire Sinks
http://docs.spring.io/spring-xd/docs/1.0.1.RELEASE/reference/html/#gemfire-server
