Updated: Aug 25
Twitter is all about what is happening and what people are talking about right now. Existing analytics help unlock the business value of discussions on Twitter, but most of these tools don’t provide a live view about what’s happening or what actions you should take in real time. In today’s post, I’m excited to partner with Travis Collins, the founder of datapm, to share how Timeplus and DataPM can help you develop a real-time Twitter marketing app within a few minutes, without a single line of code. This makes it super easy to create real-time social listening apps to understand how customers are talking about your company or products, and plan your next social marketing campaign with trending hot topics.
Seeing is believing. You can check this short video:
Or you can try live demonstration systems:
How it works
At a high level, there are just three simple steps:
Load the real-time twitter data to Timeplus, via datapm command-line interface (CLI)
Build Streaming-SQL to understand the hot hashtags or how others think of your product/company
Visualize the real-time insights via charts/dashboards, or take real-time actions via calling Twitter API
Let’s dive in.
In our example, we will be using the command line interface of datapm, which is available for Windows/Linux/Mac, to easily load real-time tweets and send them to Timeplus.
Note: datapm command line is also a great tool to load other data to Timeplus, such as PostgreSQL, Redshift, local files, Google Sheets, and Bitcoin live prices.
Load real-time tweets to Timeplus via datapm
Twitter provides a real-time API to programmatically retrieve and analyze Twitter data. Please follow the docs to get access to the v2 Twitter API.
You will also need to download datapm and install it locally. https://datapm.io/downloads
In the recent versions of datapm, the Twitter source and Timeplus sink are added to make the integration extremely easy.
Make sure the datapm version is 0.31.23 or above.
datapm --version 0.31.23
To load the real-time tweet, simply run
datapm fetch twitter/sample
This will get roughly 1% random sample of publicly available Tweets in real-time.
Once you have the Twitter API token, you can load data via datapm with a single command:
datapm fetch twitter/sample datapm client version 0.31.23 on darwin-x64 Source Selection ✔ Found package twitter/sample ℹ 1 schemas with more than 2k records tweets Schema Options ✔ Exclude any attributes from tweets? › No ✔ Rename attributes from tweets? › No Inspecting twitter Twitter Credentials ✔ Twitter API Bearer Token? … ****************************************************************************************************** ✔ Authentication succeeded ℹ Connecting to twitter.com
Then you can send the real-time data to Timeplus. Simply join the Beta program of Timeplus Cloud and to receive an API token for your Beta tenant.
Choose Timeplus as the sink:
Sink Connector ? Sink Connector? › Big Query Console (Standard Out) Decodable Kafka Local File MongoDB MySQL PostgreSQL Redshift ❯ Timeplus
Then set the hostname for the Timeplus beta tenant:
Timeplus Connection ? Repository? › ❯ datapm.beta.timeplus.com New Repository
Set the API token and type the stream name. If the data stream doesn’t exist, it will be created automatically with the proper data schema. In this example, let’s name the stream as “tweets”:
Timeplus Configuration ✔ Stream for tweets records? … tweets ✔ Created Timeplus Stream tweets Checking State of tweetStream ✔ New records may be available Reading tweetStream ✔ text has integer and string values. How should this conflict be handled? › Cast all values as string ⠏ Stream tweetStream 301 records received - 28.6 per second
That’s it! Then go to the Timeplus Cloud web UI and click the ‘tweets’ stream. A SQL is generated:
SELECT * FROM tweets
You will see the live tweets immediately.
To get the tweets relevant to your product or company, load data with `datapm fetch twitter/stream`:
datapm fetch twitter/stream Source Selection ✔ Found package twitter/stream ℹ 1 schemas with more than 124 records tweets Schema Options ✔ Exclude any attributes from tweets? › No ✔ Rename attributes from tweets? › No Inspecting twitter Twitter Credentials ✔ Credentials? › AAAAA...AlIT3 ✔ Authentication succeeded ℹ Connecting to twitter.com ✔ Select Twitter Stream Type › Search/Filter Stream ✔ Twitter Search Query? … #timeplus Sink Connector ✔ Sink Connector? › Timeplus ✔ Found the connector named Timeplus Timeplus Connection ✔ Repository? › datapm.beta.timeplus.com ✔ Connection successful Timeplus Credentials ✔ Credentials? › *** ✔ Authentication succeeded Timeplus Configuration ✔ Stream for tweets records? … tweets_timeplus ✔ Created Timeplus Stream tweets_timeplus Checking State of tweetStream ✔ New records may be available Reading tweetStream ⠙ Stream tweetStream 1 records received - 0.0 per second
Then post a tweet with the specific keywords. Within 10-15 seconds, this tweet will be available in the search results:
Analyze real-time tweets with streaming SQL
Once the real-time data arrives in Timeplus, you can run all kinds of analysis to gain real-time insight. Please check our Top 10 Streaming SQL Patterns for details.
To understand the most popular hashtags in the real-time tweets, you can extract the hashtag from the tweets (if any). The Timeplus UI will show the top 3 tags automatically in the live view.
To determine the sentiment of the tweets (whether the tweets about your product are positive or negative), you can use online services or NLP (natural language processing) programming libraries to do the accurate analysis. Just for demonstration purposes, in this example, we can use SQL to do keyword-based analysis.
Now you have real-time insights. What’s next?
As we demonstrate in many other case studies, you can visualize the real-time insights by building a live dashboard with streamlit or other BI tools, or you can take real-time actions automatically via webhook or AWS Lambda. For example, you can create an “auto-reply” process, when anyone tweets about your product publicly on Twitter, you can add a reply to manage customers and create ServiceNow tickets or support tickets automatically to track the issue. Or even as simple as “like” and “retweet” the tweets if anyone shares a positive product review on twitter. This would greatly improve customer satisfaction and bring customers closer to your social media account and community.
At Timeplus, we are pushing the limits to build real-time applications much faster and much easier than ever before. We are excited to partner with DataPM to together deliver on this vision. Our blog shows that you don’t need coding to get real-time Twitter data into Timeplus and gain real-time insights. All you have to do is to join the Timeplus Cloud beta, download the datapm tool, and connect things together via API tokens. Fast, powerful, intuitive.