• Jove Zhong

Real-time Twitter Marketing with Timeplus and datapm

Twitter is all about what is happening and what people are talking about right now. Existing analytics help unlock the business value of discussions on Twitter, but most of these tools don’t provide a live view about what’s happening or what actions you should take in real time. In today’s post, I’m excited to partner with Travis Collins, the founder of datapm, to share how Timeplus and DataPM can help you develop a real-time Twitter marketing app within a few minutes, without a single line of code. This makes it super easy to create real-time social listening apps to understand how customers are talking about your company or products, and plan your next social marketing campaign with trending hot topics.


Seeing is believing. You can check this short video:

Or you can try live demonstration systems:


https://share.streamlit.io/timeplus-io/streamlit_apps/main/hot_tags.py

show what people are talking about right now on Twitter

https://share.streamlit.io/timeplus-io/streamlit_apps/main/live_tweets.py

You can send a new tweet with the #timeplus tag, such as “#timeplus is awesome!” then within 15 seconds, your tweet will be analyzed and you can see how others are thinking of the product.

 

How it works

At a high level, there are just three simple steps:

  1. Load the real-time twitter data to Timeplus, via datapm command-line interface (CLI)

  2. Build Streaming-SQL to understand the hot hashtags or how others think of your product/company

  3. Visualize the real-time insights via charts/dashboards, or take real-time actions via calling Twitter API


Let’s dive in.


Introducing datapm


datapm (Data Package Manager) is an open source data exchange platform.


You can publish your own public or private packages and even host your own data registry for private B2B exchanges. Check out datapm.io for more information.


In our example, we will be using the command line interface of datapm, which is available for Windows/Linux/Mac, to easily load real-time tweets and send them to Timeplus.


Note: datapm command line is also a great tool to load other data to Timeplus, such as PostgreSQL, Redshift, local files, Google Sheets, and Bitcoin live prices.


Load real-time tweets to Timeplus via datapm


Twitter provides a real-time API to programmatically retrieve and analyze Twitter data. Please follow the docs to get access to the v2 Twitter API.


You will also need to download datapm and install it locally. https://datapm.io/downloads


In the recent versions of datapm, the Twitter source and Timeplus sink are added to make the integration extremely easy.


Make sure the datapm version is 0.31.23 or above.

datapm --version
0.31.23

To load the real-time tweet, simply run

datapm fetch twitter/sample

This will get roughly 1% random sample of publicly available Tweets in real-time.

Once you have the Twitter API token, you can load data via datapm with a single command:

datapm fetch twitter/sample

datapm client version 0.31.23 on darwin-x64

Source Selection
✔ Found package twitter/sample
ℹ 1 schemas with more than 2k  records

tweets Schema Options
✔ Exclude any attributes from tweets? › No
✔ Rename attributes from tweets? › No

Inspecting twitter

Twitter Credentials
✔ Twitter API Bearer Token?******************************************************************************************************
✔ Authentication succeeded
ℹ Connecting to twitter.com

Then you can send the real-time data to Timeplus. Simply join the Beta program of Timeplus Cloud and to receive an API token for your Beta tenant.


Choose Timeplus as the sink:

Sink Connector
? Sink Connector? ›
    Big Query
    Console (Standard Out)
    Decodable
    Kafka
    Local File
    MongoDB
    MySQL
    PostgreSQL
    Redshift
❯   Timeplus

Then set the hostname for the Timeplus beta tenant:

Timeplus Connection
? Repository? ›
❯   datapm.beta.timeplus.com
    New Repository

Set the API token and type the stream name. If the data stream doesn’t exist, it will be created automatically with the proper data schema. In this example, let’s name the stream as “tweets”:

Timeplus Configuration
✔ Stream for tweets records? … tweets
✔ Created Timeplus Stream tweets

Checking State of tweetStream
✔ New records may be available

Reading tweetStream
✔ text has integer and string values. How should this conflict be handled? › Cast all values as string
⠏ Stream tweetStream
   301 records received - 28.6 per second

That’s it! Then go to the Timeplus Cloud web UI and click the ‘tweets’ stream. A SQL is generated:

SELECT * FROM tweets 

You will see the live tweets immediately.

To get the tweets relevant to your product or company, load data with `datapm fetch twitter/stream`:

datapm fetch twitter/stream

Source Selection
✔ Found package twitter/stream
ℹ 1 schemas with more than 124  records

tweets Schema Options
✔ Exclude any attributes from tweets? › No
✔ Rename attributes from tweets? › No

Inspecting twitter

Twitter Credentials
✔ Credentials?AAAAA...AlIT3
✔ Authentication succeeded
ℹ Connecting to twitter.com
✔ Select Twitter Stream Type › Search/Filter Stream
✔ Twitter Search Query? … #timeplus

Sink Connector
✔ Sink Connector? › Timeplus
✔ Found the connector named Timeplus

Timeplus Connection
✔ Repository? › datapm.beta.timeplus.com
✔ Connection successful

Timeplus Credentials
✔ Credentials?***
✔ Authentication succeeded

Timeplus Configuration
✔ Stream for tweets records? … tweets_timeplus
✔ Created Timeplus Stream tweets_timeplus

Checking State of tweetStream
✔ New records may be available

Reading tweetStream
⠙ Stream tweetStream
   1 records received - 0.0 per second

Then post a tweet with the specific keywords. Within 10-15 seconds, this tweet will be available in the search results:


Analyze real-time tweets with streaming SQL


Once the real-time data arrives in Timeplus, you can run all kinds of analysis to gain real-time insight. Please check our Top 10 Streaming SQL Patterns for details.


To understand the most popular hashtags in the real-time tweets, you can extract the hashtag from the tweets (if any). The Timeplus UI will show the top 3 tags automatically in the live view.

To determine the sentiment of the tweets (whether the tweets about your product are positive or negative), you can use online services or NLP (natural language processing) programming libraries to do the accurate analysis. Just for demonstration purposes, in this example, we can use SQL to do keyword-based analysis.


Now you have real-time insights. What’s next?


As we demonstrate in many other case studies, you can visualize the real-time insights by building a live dashboard with streamlit or other BI tools, or you can take real-time actions automatically via webhook or AWS Lambda. For example, you can create an “auto-reply” process, when anyone tweets about your product publicly on Twitter, you can add a reply to manage customers and create ServiceNow tickets or support tickets automatically to track the issue. Or even as simple as “like” and “retweet” the tweets if anyone shares a positive product review on twitter. This would greatly improve customer satisfaction and bring customers closer to your social media account and community.


Conclusion


At Timeplus, we are pushing the limits to build real-time applications much faster and much easier than ever before. We are excited to partner with DataPM to together deliver on this vision. Our blog shows that you don’t need coding to get real-time Twitter data into Timeplus and gain real-time insights. All you have to do is to join the Timeplus Cloud beta, download the datapm tool, and connect things together via API tokens. Fast, powerful, intuitive.


Join our Timeplus community or sign up for our private beta to learn more. We look forward to hearing your feedback!




27 views

Recent Posts

See All