Built a real-time analytics dashboard to visualize the trending hashtags and @mentions at a given location by using real time streaming twitter API to get data.
| Area | Technology |
|---|---|
| Front-End | HTML5, Bootstrap, CSS3, Socket.IO, highcharts.js |
| Back-End | Express, Node.js |
| Cluster Computing Framework | Apache Spark (python) |
| Message Broker | Apache kafka |
- Extract data from Twitter's streaming API and put it into Kakfa topic.
- Spark is listening to this topic, it will read the data from topic, analyze it is using spark streaming and put top 10 trending hashtags and @mentions into another kafka topic.
- Spark Streaming creates DStream whenever it read the data from kafka and analyze it by performing operation like map, filter, updateStateByKey, countByValues and forEachRDD on the RDD and top 10 hashtags and mentions are obtained from RDD using SparkSQL.
- Node.js will pick up the this data from kafka topic on server side and emit it to the socket.
- Socket will push data to user's dashboard which is rendered using highcharts.js in realtime.
- The dashboard is refreshed every 60 secs.
