Social Science – Get Twitter data in Power BI

Ever wanted to do some analytics on tweets with a certain hashtag (#) or from a certain user (@)? Don’t look any further, there is a great tool called Microsoft Power BI. In this blog post, I’ll be explaining how to get twitter data into your Power BI. Let’s go!

Creating an account

First, we need to authorize ourselves on Twitter. This can be done by creating a Developer Account. Click on Sign Up in the upper right corner and create an account. Then go to apps.twitter.com and click Create New App. Now you need to fill in some details about your application. It does not really matter what you fill in for Name and Description, but your Name needs to be globally unique. For Website you can use your company website. Callback URL is not necessary for this project, so you don’t have to fill it in. Don’t forget to agree with the Twitter Developer Agreement at the end and simply confirm your application by clicking Create your Twitter application.

When you land on the application page, click the tab Keys and Access Tokens and then click on the bottom button Create my access token. Keep this page open because you will need the Consumer Key & Access Token later on.

Most work is finished now, so let’s load some data! Open Power BI and click Get Data and choose R Script. Yes, we will be using some R coding, but don’t worry! It’s very basic and my comments will explain everything.

You will see a window where you can input your R Script. Paste following code and click OK to confirm.

[sql]

1 #Check if all libraries are available.
2 list.of.packages <- c("twitteR","stringi")
3 new.packages <- list.of.packages[!(list.of.packages %in% installed.packages()[,"Package"])]
4 if(length(new.packages)) install.packages(new.packages)
5
6 #Load Twitter Library
7 library(twitteR)
8
9 #Fill in these details with your personal consumer en access keys that you received on apps.twitter.com
10 consumer_key <- "xxxxx"
11 consumer_secret <- "xxxxx"
12 access_token <- "xxxxx"
13 access_secret <- "xxxxx"
14
15 #Initial authentication to twitter
16 setup_twitter_oauth(consumer_key, consumer_secret, access_token, access_secret)
17
18 #What are we looking for on twitter?
19 search_term <- "#powerbi"
20 #How many tweets do you want to receive
21 number_of_tweets <- 1000
22
23 #Search on Twitter
24 tweets <- searchTwitter(search_term, number_of_tweets )
25
26 #convert received data to something PowerBi understands
27 df_tweets <- twListToDF(tweets)
28
29 #column cleanup
30 df_tweets <- df_tweets[,c("text","favorited","favoriteCount","created","retweetCount","isRetweet","retweeted")]
31 df_tweets$text <- stri_encode(df_tweets$text, "", "UTF-8")

[/sql]

First we will install the ‘twitteR’ library. Its a package created by Jeff Gentry. The package that he has developed will do all the hard work, so we don’t need to bother about it anymore. Second step is to fill in all the keys that we received from Twitter. After that the code will do some authorization with Twitter, so we are allowed to receive data. Now its our turn to tell the code what we are looking for and how many tweets we want to receive back. Next is the real search on Twitter and converting the received data in something readable for Power BI. Final part is cleaning up some columns and changing the code page of the tweets. Click OK to confirm the code. This might take a while depending on how many tweets you want to be returned. A navigator window will open. Select the data set df_tweets and click OK.

All the next steps are up to you! The data is now available in Power BI. Enjoy querying Twitter!