Twitter API Setup

To collect Twitter data, we’re going to work with the Twitter API and twarc, a Python package for collecting Twitter data through the Twitter API. To access the Twitter API, we first need to:

1. Apply for a Twitter developer account

2. Create a Twitter application

The developer account will allow us to create an application, which will eventually get us a series of API keys and tokens, which we can then use to access Twitter data.

According to Twitter, the reason for this somewhat drawn-out application process is to “prevent abuse of the Twitter platform” and “better understand and serve our developer community.”

After getting our API keys, we then need to

3. Install twarc

4. Configure/set up twarc

5. Download the twarc repository

The following instructions will guide you through each part of this 5-step process.

1. Apply for a Twitter Developer Account

1. First navigate to Twitter’s “Apply for access” web page and select “Apply for a developer account.”

../../_images/apply-for-access1.png

2. If you already have a Twitter account, you’ll be asked to log in with your account username and password. If you don’t already have a Twitter account, you’ll be asked to sign up for one.

3. After you’ve logged in, Twitter will ask: “What is your primary reason for using Twitter developer tools?” Because we’re collecting data for our class “Introduction to Cultural Analytics,” you can select “Doing academic research” or “Student.”

../../_images/developer-primary-reason1.png

4. The next page of the application will ask: “This is you, right?” Confirm that your Twitter username and email are correct, select your country of residence (United States), and come up with a name for your application. It doesn’t matter which name you choose. I’d suggest using your first name.

5. Then Twitter will ask: “How will you use the Twitter API or Twitter data?”

../../_images/twitter-use-explanation1.png

**In English, please describe how you plan to use Twitter data and/or APIs. For students and teachers, please include the name of the school, the name of the instructor and the course number (i

Are you planning to analyze Twitter data?

Will your app use Tweet, Retweet, like, follow, or Direct Message functionality?

Do you plan to display Tweets or aggregate data about Twitter content outside of Twitter?

Will your product, service or analysis make Twitter content or derived information available to a government entity? In general, schools, colleges, and universities do not fall under this category.

6. Review your application and accept Twitter’s Developer Agreement. Before accepting the agreement, however, briefly read through it and remember to note down one thing that surprises you about the terms in for HW 6.

../../_images/developer-agreement1.png

7. Verify your email.

../../_images/developer-success1.png

2. Create an Application

Once you confirm your email address, you’ll be taken to your developer account home page, where you can now create an app.

../../_images/app-home-page1.png

1. Select “Create an app.”

../../_images/create-app1.png

2. Now you have to fill out another application! Don’t fret. We need to create an app to collect Twitter data, but many of the questions here are geared toward other kinds of Twitter apps, and we can skip them.

Below are suggested responses:

App name (required)

(Choose a name. Again, it doesn’t matter which name you choose. First name will suffice.)

Application description (required)

This app will be used to collect, curate, and analyze datasets related to culture and the humanities.

Website URL (required)

https://melaniewalsh.github.io/Intro-Cultural-Analytics

You can skip Callback URLs, Terms of Service URL, Privacy Policy URL, Organization name, and Organization website URL.

Tell us how this app will be used (required)

I will be using this app for a class called “Introduction to Cultural Analytics” (INFO 1350), which is being taught at Cornell University by Prof. Melanie Walsh. We will be collecting, curating, and analyzing datasets related to culture and the humanities. I plan to collect tweets about different subjects, to contextualize them, and to possibly share them as tweet IDs.

Then select “Create.”

3. You’ve successfully created an app! Now select the “Keys and tokens” tab of your application home page. You will be able to see your unique “Consumer API key” and “Consumer API secret key.” You will also need to generate your own “Access token & access token secret.”

../../_images/keys-and-tokens1.png

Write down your consumer key, consumer secret, access token, and access token secret in a safe location. Treat these keys and tokens like you would a password. (For more information about how and why to protect your keys and tokens, see “Securing keys and access tokens”.) You will need to use your consumer key, consumer secret, access token, and access token secret to access Twitter’s API through twarc.

3. Install twarc

Now we’re going to install the Python and command line tool twarc, which you can find hosted on this GitHub page with installation and usage instructions: https://github.com/DocNow/twarc

To install twarc, simply run the following on the command line:

!pip install twarc
!twarc version
twarc v1.7.3

4. Configure/Set up Twarc

Now that twarc is installed on our computers, we need to set it up so that we can collect Twitter data with this tool. We need to submit and save our Twitter API keys into twarc.

There are two options for configuring twarc.

Option 1 — Configure Twarc From the Command Line

To configure twarc, open up your Terminal or PowerShell and copy and paste twarc configure into your command line.

!twarc configure

Twarc will prompt you to copy and paste in your Twitter consumer key and Twitter consumer secret. Then it will ask you to visit a URL to authorize access to the Twitter account that is associated with your API keys.

../../_images/Twitter-authorize1.png

Once you click “Authorize App”, you will be redirected to the URL that is associated with your API keys (likely our course website). You need to carefully inspect this URL because it actually contains the PIN that you need for the last step of twarc configuration. The URL will look something like this:

https://melaniewalsh.github.io/Intro-Cultural-Analytics/oauthtoken=YOUR-UNIQUE-TOKEN&oauthverifier=THIS-IS-THE-DISPLAYED-PIN-YOU-NEED

You need to copy and paste the part after oauthverifier= into the prompt at the command line:

After you have authorized the application please enter the displayed PIN:

If the PIN works, then you will get a happy successs message. Make sure to copy and paste this message into HW 6.

The credentials for mellymeldubs have been saved to your configuration file at /Users/melaniewalsh/.twarc

✨ ✨ ✨  Happy twarcing! ✨ ✨ ✨

Option 2 — Configure Twarc in This Notebook

Copy your API keys and paste them into the quotation marks below. Also type in your Twitter handle without the @ symbol.

twitter_handle = ""
consumer_key= ""
consumer_secret = ""
access_token = ""
access_token_secret= ""

Then run the two cells below:

configuration = f"""[{twitter_handle}]
consumer_key={consumer_key}
consumer_secret = {consumer_secret}
access_token = {access_token}
access_token_secret= {access_token_secret}
"""
import os
config_filename = os.path.join(os.path.expanduser("~"), ".twarc")
with open(config_filename, "w") as file_object:
    file_object.write(configuration)

To test whether twarc has been properly configured, run a sample search and see if Twitter data gets returned:

!twarc search "something incredibly obscure"
{"created_at": "Wed Mar 04 16:09:46 +0000 2020", "id": 1235235999770439681, "id_str": "1235235999770439681", "full_text": "@ARX8x No problem! \nOne more small thing: you could also add Installer 5 on the list, if it has URL scheme support... and that would be all the available package managers out there, unless I'm missing something incredibly obscure or outdated", "truncated": false, "display_text_range": [7, 241], "entities": {"hashtags": [], "symbols": [], "user_mentions": [{"screen_name": "ARX8x", "name": "ARX8x", "id": 2305006776, "id_str": "2305006776", "indices": [0, 6]}], "urls": []}, "metadata": {"iso_language_code": "en", "result_type": "recent"}, "source": "<a href=\"http://twitter.com\" rel=\"nofollow\">Twitter Web Client</a>", "in_reply_to_status_id": 1235193727393492992, "in_reply_to_status_id_str": "1235193727393492992", "in_reply_to_user_id": 2305006776, "in_reply_to_user_id_str": "2305006776", "in_reply_to_screen_name": "ARX8x", "user": {"id": 1441729536, "id_str": "1441729536", "name": "Simply Seb", "screen_name": "YoshiFan13", "location": "Romania", "description": "Developer | That guy who made Windows 7 Sparta Remix (6M+ views) | Fandoms: Danger Mouse @Danger_Seb, Club Penguin, Guitar Hero | I love @TrashCosWynaut \u2764\ufe0f", "url": "https://t.co/w3vIr9MsDv", "entities": {"url": {"urls": [{"url": "https://t.co/w3vIr9MsDv", "expanded_url": "http://yoshifan.me", "display_url": "yoshifan.me", "indices": [0, 23]}]}, "description": {"urls": []}}, "protected": false, "followers_count": 3922, "friends_count": 766, "listed_count": 24, "created_at": "Sun May 19 17:24:04 +0000 2013", "favourites_count": 20324, "utc_offset": null, "time_zone": null, "geo_enabled": true, "verified": false, "statuses_count": 55589, "lang": null, "contributors_enabled": false, "is_translator": false, "is_translation_enabled": false, "profile_background_color": "373940", "profile_background_image_url": "http://abs.twimg.com/images/themes/theme1/bg.png", "profile_background_image_url_https": "https://abs.twimg.com/images/themes/theme1/bg.png", "profile_background_tile": true, "profile_image_url": "http://pbs.twimg.com/profile_images/716118950602543104/ip0zmHkl_normal.jpg", "profile_image_url_https": "https://pbs.twimg.com/profile_images/716118950602543104/ip0zmHkl_normal.jpg", "profile_banner_url": "https://pbs.twimg.com/profile_banners/1441729536/1511096815", "profile_link_color": "373940", "profile_sidebar_border_color": "000000", "profile_sidebar_fill_color": "DDEEF6", "profile_text_color": "333333", "profile_use_background_image": true, "has_extended_profile": true, "default_profile": false, "default_profile_image": false, "following": false, "follow_request_sent": false, "notifications": false, "translator_type": "regular"}, "geo": null, "coordinates": null, "place": null, "contributors": null, "is_quote_status": false, "retweet_count": 0, "favorite_count": 0, "favorited": false, "retweeted": false, "lang": "en"}

5. Download the Twarc Repository

Finally, we also need to download the twarc repository from GitHub, because there are a few things in it that aren’t included in the version of twarc that’s installed through pip. To download the repository, run:

!git clone https://github.com/DocNow/twarc.git

If Git isn’t working for some reason, you can also download the repository as a zip file: https://github.com/DocNow/twarc/archive/master.zip