Get Tweets and store them in NoSQL DB in real time using Bluemix (Part 1)

Imagine we want to get Tweets in real-time talking about some specific topic and we want to store them in a database to later run some analysis. There are many ways to do that but using IBM Bluemix it is extremely easy, we do not have to write a single line of code or leave our computer switched on capturing tweets…because the application is all running on the cloud.  For doing this we are going to use a tool integrated on IBM Bluemix called Node-Red.

Node-RED is a tool for wiring together hardware devices, APIs and online services in new and interesting ways. It is a creation of IBM Emerging Technology and it is simply…amazing! 

cloudantTweets

Here you have the steps to do it:

1. Log in into IBM Bluemix

2. Click on “Create an App” and select the Boirlerplate “Node-Red Starter“. This will create a SDK for Node.js and a Cloudant NoSQL service automatically and will be set up in seconds. You only need to give a name to your application.

nodered

3. Your application will appear in the Dashboard. Click on it and click on the link to access your app. You will have the launch page of Node-Red. Click on “Go to your Node-RED flow editor”

4. You will have the flow editor. Here is as easy as drag and drop the components, connect them and create the flow of your live data. You change the properties of each of the nodes, and you have a debugger on the right to see the live data that is coming to your application.

bluemix, nodered

5. We will select the “Twitter” node from the social category. To get the tweets you have to promt access to the Twitter API, that’s easy and done automatically in the properties. You can also select what to search for.  In my case I am getting tweets containing the word “bigdata”.

bigdata

6. Last step is to select the output. In this case we will store the tweets in Cloudant NoSQL database that we created previously. We select the name of the Cloudant instance in our Bluemix, and the database name (by default is called “nodered”).

flow

7. Now all the tweets with the word “bigdata” will be stored in my Cloudant database. I can do some more accurate filtering in Node-RED, like filtering by language, user, etc. In this tutorial we kept it simple. If I open my Cloudant Administratio console, I can see all the tweets stored:

screentsqdd

In next post we will start doing basic analysis of this data using tools like SPSS and R.

Share this:

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong> <pre lang="" line="" escaped="">