SpringXD FILE/TAIL input ingestion / JDBC Sink output to GemFireXD

The following examples shows what we can do out of the box with SpringXD TAIL input source and JDBC sink as the output source. This can all be done without writing and code.

Assumes you have SpringXD installed and GemFireXD installed. See the getting stared guide which explains how to setup SpringXD. As for a database output for JDBC you can use any RDBMS here so you could use MYSQL , ORACLE etc if you don't have GemFireXD installed.

Note: XD_BASE = /Users/papicella/vmware/software/spring/spring-xd-1.0.0.M4

1. Configure connection to RDBMS in "$XD_BASE/xd/config/jdbc.properties".

driverClass=com.vmware.sqlfire.jdbc.ClientDriver
url=jdbc:sqlfire://10.32.240.113:1527
username=APP
password=APP

2. Add jdbc jar file to $XD_BASE/xd/lib directory, in this case GemFireXD which uses "sqlfireclient.jar"

3. Start SpringXD single node using "./$XD_BASE/xd/bin/xd-singlenode"

./$XD_BASE/xd/bin/xd-singlenode
.....

11:29:00,417  INFO main container.XDContainer:175 - started container: 0
11:29:00,424  INFO main launcher.LocalContainerLauncher:95 - 
 _____                           __   _______
/  ___|          (-)             \ \ / /  _  \
\ `--. _ __  _ __ _ _ __   __ _   \ V /| | | |
 `--. \ '_ \| '__| | '_ \ / _` |  / ^ \| | | |
/\__/ / |_) | |  | | | | | (_| | / / \ \ |/ /
\____/| .__/|_|  |_|_| |_|\__, | \/   \/___/
      | |                  __/ |
      |_|                 |___/
1.0.0.M4                         eXtreme Data

Using local mode JMX is disabled for XD components
XD Configuration:
XD_HOME=/Users/papicella/vmware/software/spring/spring-xd-1.0.0.M4/xd
XD_TRANSPORT=local
XD_STORE=memory
XD_ANALYTICS=memory
XD_HADOOP_DISTRO=hadoop12

started container : 85631@localhost
Documentation: https://github.com/SpringSource/spring-xd/wiki

4. Create a empty file that the stream will read from using TAIL as the input

> touch filetest

eg:

[Fri Jan 10 11:29:06 papicella@:/tmp/xd/input ] $ ls -la /tmp/xd/input/filetest 
-rw-r--r--  1 papicella  wheel  0 10 Jan 11:29 /tmp/xd/input/filetest

5. Log into SpringXD shell as shown below using "$XD_BASE/shell/bin/xd-shell"

[Fri Jan 10 10:46:39 papicella@:~/vmware/software/spring/spring-xd-1.0.0.M4/shell/bin ] $ ./xd-shell

 _____                           __   _______

/  ___|          (-)             \ \ / /  _  \

\ `--. _ __  _ __ _ _ __   __ _   \ V /| | | |

 `--. \ '_ \| '__| | '_ \ / _` |  / ^ \| | | |

/\__/ / |_) | |  | | | | | (_| | / / \ \ |/ /

\____/| .__/|_|  |_|_| |_|\__, | \/   \/___/

      | |                  __/ |

      |_|                 |___/

eXtreme Data

1.0.0.M4 | Admin Server Target: http://localhost:9393

Welcome to the Spring XD shell. For assistance hit TAB or type "help".

6. Create a STREAM as shown below.

xd:> stream create --name filetest --definition "tail --name=/tmp/xd/input/filetest |  jdbc --columns=id,name"

Created new stream 'filetest'

7. Create table in database as follows.

  
sqlf> create table filetest (id int, name varchar(20));

8. Create file as follows called "testfile" we will copy this to stream file soon.

[Fri Jan 10 11:38:44 papicella@:~/vmware/software/spring/spring-xd-1.0.0.M4 ] $ cat filetest

{"id":"1","name":"pas"}

{"id":"2","name":"lucia"}

{"id":"3","name":"lucas"}

{"id":"4","name":"siena"}

7. Copy file "filetest" above to /tmp/xd/input/filetest

9. Check base table in RDBMS for data inserts.

  
sqlf> select * from filetest;
ID         |NAME                
--------------------------------
4          |siena               
1          |pas                 
3          |lucas               
2          |lucia               

4 rows selected

More Information

For more information on SpringXD or GemFireXD see the links below.

http://projects.spring.io/spring-xd/ - SpringXD

http://gopivotal.com/products/pivotal-hd - GemFireXD Beta