{"id":248,"date":"2017-03-27T00:00:39","date_gmt":"2017-03-26T22:00:39","guid":{"rendered":"http:\/\/www.mikaelbrunila.fi\/?p=248"},"modified":"2017-04-22T18:07:20","modified_gmt":"2017-04-22T16:07:20","slug":"scraping-extracting-mapping-geodata-twitter","status":"publish","type":"post","link":"http:\/\/www.mikaelbrunila.fi\/2017\/03\/27\/scraping-extracting-mapping-geodata-twitter\/","title":{"rendered":"Scraping, extracting and mapping geodata from Twitter"},"content":{"rendered":"

&lt;\/iframe&gt;&lt;\/p&gt;\n&lt;p&gt;&lt;em&gt;This post gives a brief and general introduction to scraping Twitter with Python, extracting geodata from this data and mapping it using Fusion Tables and CartoDB. I talked\u00a0about this as well as some Facebook scraping I did with R in the &lt;a href=\"http:\/\/westermarck.fi\/sosiologipaivat\/working-groups\/\"&gt;working group on the Digitalization of Societies and Methods&lt;\/a&gt; at the Annual Westermarck Conference in Tampere on Thursday.&lt;\/em&gt;&lt;\/p&gt;\n&lt;p&gt;Twitter is a tricky platform for any\u00a0programmer who wants to extract geodata. Scrape tweets from any given hashtag and you&amp;#8217;ll only get a marginal amount of actual, pinpointed geodata with a longitude and a latitude. This has\u00a0at least my experience when scraping Twitter.&lt;\/p&gt;\n&lt;p&gt;What is one to do then, if what you are looking for is geodata? I came across this problem when I was scraping a hashtag for &lt;a href=\"https:\/\/roarmag.org\/essays\/spain-evictions-resistance-pah\/\"&gt;an international day of action &lt;\/a&gt;against the private equity (or vulture) fund Blackstone. The events of the day were organized by a coalition of housing groups across several countries and continents and had an appointed hashtag, &lt;a href=\"https:\/\/twitter.com\/hashtag\/stopblackstone\"&gt;#StopBlackstone&lt;\/a&gt;. I was following the action on the ground in Barcelona, where I was doing fieldwork with the compa\u00f1erxs of &lt;a href=\"https:\/\/en.wikipedia.org\/wiki\/Plataforma_de_Afectados_por_la_Hipoteca\"&gt;la PAH&lt;\/a&gt; for the project &lt;a href=\"http:\/\/www.helsinki.fi\/cradle\/lpsm_project.htm\"&gt;Learning in Productive Social Movements&lt;\/a&gt;.&lt;\/p&gt;\n&lt;p&gt;La PAH is a huge movement and it was clear from the outset that they would come up with a big mobilization. But what would the day look like in other places? I didn&amp;#8217;t see\u00a0participant numbers from different places reported afterwards, but on the could deduce from pictures that most actions were small. But how about other types of engagement? How did the designated hashtag take off? Without falling into the old hyperbole of &amp;#8216;Twitter revolutions&amp;#8217; I think it&amp;#8217;s still safe to say that social media engagement is a real part of how many movements build power. Yes, it is still only one metric among others when measuring the power of a movement, but it is still a metric. As such it can at least tell us &lt;em&gt;something,&lt;\/em&gt; even if it doesn&amp;#8217;t tell us nearly\u00a0&lt;em&gt;everything&lt;\/em&gt;.&lt;\/p&gt;\n&lt;p&gt;What I wanted to find out was what amount of unique users were engaging with the hashtag #StopBlackstone in different places. On the one hand, this\u00a0is admittedly a blunt approach, which captures only a fraction of the\u00a0properties of the network the formed around the hashtag. On the other hand, a map showing all the unique users that engaged with that hashtag would probably (and in the end did) produce a good approximation of the impact the action had in different places.&lt;\/p&gt;\n&lt;p&gt;But where to start? For each tweet you send, Twitter includes two types of location data in the\u00a0data that makes up the tweet:&lt;\/p&gt;\n&lt;ol&gt;\n&lt;li&gt;Exact location data for &lt;a href=\"https:\/\/dev.twitter.com\/overview\/api\/tweets\"&gt;the tweet.&lt;\/a&gt;&lt;\/li&gt;\n&lt;li&gt;General location data for &lt;a href=\"https:\/\/dev.twitter.com\/overview\/api\/users\"&gt;the user&lt;\/a&gt;.&lt;\/li&gt;\n&lt;\/ol&gt;\n&lt;p&gt;Most people have turned off sharing of geodata for individual tweets. This means that there is no information on where they tweeted that exact tweet from. For instance, in the batch of 8310 tweets I scraped during one day of activities on the #StopBlackstone hashtag, only two\u00a0users included\u00a0the coordinates that the tweet was sent from. Five included\u00a0the place they were tweeting from. Not much data to work with!&lt;\/p&gt;\n&lt;p&gt;What most users do provide though, is information on their own general location. For the purposes of my research this was enough: it wasn&amp;#8217;t really important to know where one individual was tweeting from but to get an approximation of the geographical distribution of people tweeting.&lt;\/p&gt;\n&lt;p&gt;What I did then, was a script to loop through the diverse geodata in each scraped tweet. I would first check if the tweet was provided with exact data on where it was sent from. If not, I would move on to the general location data provided by the user and saving that instead. The resulting list of JSON objects included locations\u00a0for &lt;strong&gt;709 users out of 1245&lt;\/strong&gt;, who had produced\u00a0&lt;strong&gt;5184 out of the total 8310 of tweets&lt;\/strong&gt; I scraped during that day.&lt;\/p&gt;\n&lt;p&gt;To geocode and map this data I tried two different approaches. First, I passed\u00a0the data through Google Refine to convert it from JSON to CSV. Then I uploaded it to Google Fusion Tables to geocode the data and produce this map:&lt;\/p&gt;\n&lt;p&gt;&lt;iframe src=\"https:\/\/fusiontables.google.com\/embedviz?q=select+col3+from+1a0pv7jnJu7igDTCiSWDYl7WpS5XKFpTH8nCKMroB&amp;amp;viz=MAP&amp;amp;h=false&amp;amp;lat=42.01709532298182&amp;amp;lng=-31.90821875000006&amp;amp;t=1&amp;amp;z=3&amp;amp;l=col3&amp;amp;y=2&amp;amp;tmplt=2&amp;amp;hml=GEOCODABLE\" width=\"500\" height=\"300\" frameborder=\"no\" scrolling=\"no\"&gt;&lt;\/iframe&gt;&lt;\/p&gt;\n&lt;p&gt;Because Fusion Tables has very few features for visualizing data points, I used\u00a0 &lt;a href=\"https:\/\/en.wikipedia.org\/wiki\/CartoDB\"&gt;CartoDB&lt;\/a&gt; in my second approach to produce something more informative and pretty. This time I used &lt;a href=\"http:\/\/openrefine.org\/\"&gt;OpenRefine&lt;\/a&gt; to both convert the data to CSV &lt;em&gt;and&lt;\/em&gt; geocode it.\u00a0Then the data was ready for CartoDB, that I used to produce the map at the beginning of this post. This map doesn&amp;#8217;t only show the geographical distribution of users, but also the amount of tweets (and retweets) that each user produced during the day as the size of the data point. In both maps you can find out more about a particular user by clicking on a data point.&lt;\/p&gt;\n&lt;p&gt;&lt;a href=\"https:\/\/news.vice.com\/article\/activists-gather-worldwide-to-protest-against-housing-vulture-fund\"&gt;On the ground&lt;\/a&gt;, the #StopBlackstone action\u00a0had at least 200 participants in Barcelona, with far smaller \u00a0turnouts for the demonstrations in &lt;a href=\"https:\/\/www.thejournal.ie\/blackstone-irish-housing-network-demonstration-2388933-Oct2015\/\"&gt;Dublin&lt;\/a&gt;, &lt;a href=\"http:\/\/www.tenantstogether.org\/updates\/blackstoneinvitation-homes-tenants-fight-back\"&gt;Atlanta, Chicago, New York, Seattle&lt;\/a&gt;, London and Japan. This is clearly reflected in the data on the map, where most locations match with actual places. There are\u00a0also many outliers, of which\u00a0most seem to be due to the elusive locations some people have in their Twitter description (&amp;#8220;Peace is a continuous act&amp;#8221;, &amp;#8220;Donde haya injusticia&amp;#8221;, &amp;#8220;el mundo&amp;#8221;, &amp;#8220;Union Sovietica&amp;#8221;, Naci\u00f3n Humana Universal&amp;#8221; etc.).&lt;\/p&gt;\n&lt;p&gt;In this post I will walk through the different steps of scraping tweets,\u00a0extracting the relevant location data from them, geocoding this data and then mapping it. Not because this would necessarily be the\u00a0best approach to my chosen problem, but because this was something I did in 2015 when I had just started working with Python, the Twitter API and geocoding, and I learnt a lot while doing it.&lt;\/p&gt;\n&lt;h6&gt;1. Scraping Twitter with\u00a0Python&lt;\/h6&gt;\n&lt;p&gt;Twitter is the chosen medium to experiment on for a lot people who want to learn data science. The API of the platform is well documented and clear. Most programming languages have good libraries for interfacing with it. And, finally, Twitter data is much more straightforward to process than, for instance, Facebook data. That&amp;#8217;s also why I started working with Twitter data in 2015.&lt;\/p&gt;\n&lt;p&gt;To get my data, I used the following elegant solution based on the Python Twitter library Tweepy and a script\u00a0&lt;a href=\"https:\/\/marcobonzanini.com\/2015\/03\/02\/mining-twitter-data-with-python-part-1\/\"&gt;by Marco Bonzanini&lt;\/a&gt;, with some added code for saving the tweets into a JSON file:&lt;\/p&gt;\n&lt;pre class=\"EnlighterJSRAW\" data-enlighter-language=\"python\"&gt;import tweepy\r\nfrom tweepy import Stream\r\nfrom tweepy.stream import StreamListener \r\nfrom tweepy import OAuthHandler\r\nimport json\r\n\r\nconsumer_key = \"YOUR CONSUMER KEY\"\r\nconsumer_secret = \"YOUR CONSUMER SECRET\"\r\naccess_token = \"YOUR ACCESS TOKEN\"\r\naccess_secret = \"YOUR ACCESS SECRET\"\r\n \r\nauth = OAuthHandler(consumer_key, consumer_secret)\r\nauth.set_access_token(access_token, access_secret)\r\n \r\napi = tweepy.API(auth)\r\n\r\n@classmethod\r\ndef parse(cls, api, raw):\r\n status = cls.first_parse(api, raw)\r\n setattr(status, 'json', json.dumps(raw))\r\n return status\r\n \r\n# Status() is the data model for a tweet\r\ntweepy.models.Status.first_parse = tweepy.models.Status.parse\r\ntweepy.models.Status.parse = parse\r\n\r\nclass MyListener(StreamListener):\r\n \r\n def on_data(self, data):\r\n try:\r\n with open('FILENAME.json', 'a') as f:\r\n f.write(data)\r\n return True\r\n except BaseException as e:\r\n print(\"Error on_data: %s\" % str(e))\r\n return True\r\n \r\n def on_error(self, status):\r\n print(status)\r\n return True\r\n \r\n#Set the hashtag to be searched\r\ntwitter_stream = Stream(auth, MyListener())\r\ntwitter_stream.filter(track=['#HASHTAG_TO_SEARCH'])\r\n&lt;\/pre&gt;\n&lt;p&gt;If you install the Tweepy library and run this code in Terminal with an active hashtag, tweets will start pouring into your JSON file. Twitter imposes some limits on how many tweets you can access per second, but that only poses a problem for very active hashtags. These\u00a0can produce several megabytes of data in less than a minute. To interrupt the code, just press ctrl+C.&lt;\/p&gt;\n&lt;h6&gt;\u00a02. Getting location data out of tweets&lt;\/h6&gt;\n&lt;p&gt;In the #StopBlackstone data I scraped with the code above in October 2015, the first scraped tweet is my own retweet of this tweet:&lt;\/p&gt;\n&lt;blockquote class=\"twitter-tweet\" data-width=\"525\"&gt;\n&lt;p lang=\"en\" dir=\"ltr\"&gt;Tomorrow at 1pm in &lt;a href=\"https:\/\/twitter.com\/hashtag\/NYC?src=/hash\"&gt;#NYC&lt;\/a&gt; | Our Homes are NOT Commodities! &lt;a href=\"https:\/\/twitter.com\/hashtag\/StopBlackstone?src=/hash\"&gt;#StopBlackstone&lt;\/a&gt; &lt;a href=\"https:\/\/t.co\/3foWJ6XsE4\"&gt;https:\/\/t.co\/3foWJ6XsE4&lt;\/a&gt; &lt;a href=\"http:\/\/t.co\/BYBmLlQ8WG\"&gt;pic.twitter.com\/BYBmLlQ8WG&lt;\/a&gt;&lt;\/p&gt;\n&lt;p&gt;&amp;mdash; Occupy Wall Street (@OccupyWallStNYC) &lt;a href=\"https:\/\/twitter.com\/OccupyWallStNYC\/status\/653973425724227585\"&gt;October 13, 2015&lt;\/a&gt;&lt;\/p&gt;&lt;\/blockquote&gt;\n&lt;p&gt;&lt;script async src=\"\/\/platform.twitter.com\/widgets.js\" charset=\"utf-8\"&gt;&lt;\/script&gt;&lt;\/p&gt;\n&lt;p&gt;Each tweet I scraped is saved as a JSON object and includes massive amounts of data. I&amp;#8217;ve made an online copy of the retweet of this tweet as it was when I scraped\u00a0JSON object above &lt;a href=\"http:\/\/jsoneditoronline.org\/?id=0f285ce44490929f43a085d5445e9d58\"&gt;here&lt;\/a&gt;.&lt;\/p&gt;\n&lt;p&gt;Out of the multitude of information stored in the tweet, I want to access location data for the tweet itself, and if that isn&amp;#8217;t available, the user. I wrote the following script to do this:&lt;\/p&gt;\n&lt;pre class=\"EnlighterJSRAW\" data-enlighter-language=\"python\"&gt;import json\r\n\r\n# Tweets are stored in in file \"fname\". In the file used for this script, \r\n# each tweet was stored on one line\r\n\r\nfname = 'Blackstone_STREAM.json'\r\n\r\nwith open(fname, 'r') as f:\r\n \r\n #Create dictionary to later be stored as JSON. All data will be included\r\n # in the list 'data'\r\n users_with_geodata = {\r\n \"data\": []\r\n }\r\n all_users = []\r\n total_tweets = 0\r\n geo_tweets = 0\r\n for line in f:\r\n tweet = json.loads(line)\r\n if tweet['user']['id']:\r\n total_tweets += 1 \r\n user_id = tweet['user']['id']\r\n if user_id not in all_users:\r\n all_users.append(user_id)\r\n \r\n #Give users some data to find them by. User_id listed separately \r\n # to make iterating this data later easier\r\n user_data = {\r\n \"user_id\" : tweet['user']['id'],\r\n \"features\" : {\r\n \"name\" : tweet['user']['name'],\r\n \"id\": tweet['user']['id'],\r\n \"screen_name\": tweet['user']['screen_name'],\r\n \"tweets\" : 1,\r\n \"location\": tweet['user']['location'],\r\n }\r\n }\r\n\r\n #Iterate through different types of geodata to get the variable primary_geo\r\n if tweet['coordinates']:\r\n user_data[\"features\"][\"primary_geo\"] = str(tweet['coordinates'][tweet['coordinates'].keys()[1]][1]) + \", \" + str(tweet['coordinates'][tweet['coordinates'].keys()[1]][0])\r\n user_data[\"features\"][\"geo_type\"] = \"Tweet coordinates\"\r\n elif tweet['place']:\r\n user_data[\"features\"][\"primary_geo\"] = tweet['place']['full_name'] + \", \" + tweet['place']['country']\r\n user_data[\"features\"][\"geo_type\"] = \"Tweet place\"\r\n else:\r\n user_data[\"features\"][\"primary_geo\"] = tweet['user']['location']\r\n user_data[\"features\"][\"geo_type\"] = \"User location\"\r\n\r\n #Add only tweets with some geo data to .json. Comment this if you want to include all tweets.\r\n if user_data[\"features\"][\"primary_geo\"]:\r\n users_with_geodata['data'].append(user_data)\r\n geo_tweets += 1\r\n \r\n #If user already listed, increase their tweet count\r\n elif user_id in all_users:\r\n for user in users_with_geodata[\"data\"]:\r\n if user_id == user[\"user_id\"]:\r\n user[\"features\"][\"tweets\"] += 1\r\n \r\n #Count the total amount of tweets for those users that had geodata \r\n for user in users_with_geodata[\"data\"]:\r\n geo_tweets = geo_tweets + user[\"features\"][\"tweets\"]\r\n\r\n #Get some aggregated numbers on the data\r\n print \"The file included \" + str(len(all_users)) + \" unique users who tweeted with or without geo data\"\r\n print \"The file included \" + str(len(users_with_geodata['data'])) + \" unique users who tweeted with geo data, including 'location'\"\r\n print \"The users with geo data tweeted \" + str(geo_tweets) + \" out of the total \" + str(total_tweets) + \" of tweets.\"\r\n\r\n# Save data to JSON file\r\nwith open('Blackstone_users_geo.json', 'w') as fout:\r\n fout.write(json.dumps(users_with_geodata, indent=4))\r\n\r\n\r\n\r\n&lt;\/pre&gt;\n&lt;p&gt;Running the script in Terminal produces the file &amp;#8220;Blackstone_users_geo.json&amp;#8221; along with the following dump:&lt;\/p&gt;\n&lt;p style=\"padding-left: 30px;\"&gt;&lt;span class=\"s1\"&gt;The file included 1245 unique users who tweeted with or without geo data.&lt;br \/&gt;\n&lt;\/span&gt;&lt;span class=\"s1\"&gt;The file included 709 unique users who tweeted with geo data, including &amp;#8216;location&amp;#8217;.&lt;br \/&gt;\n&lt;\/span&gt;&lt;span class=\"s1\"&gt;The users with geo data tweeted 5184 out of the total 8310 of tweets.&lt;\/span&gt;&lt;\/p&gt;\n&lt;p&gt;What the script actually does, is this: It iterates over all the tweets I scraped, extracting the user who authored the tweet or retweet. For each user, the script checks if they provided specific geodata on where they tweeted from. The vast majority didn&amp;#8217;t. Then the script moves on to location data that users provide voluntarily in their profile description. In most cases this includes some actual information on where this person is based, thus giving a clue on how engagement with the hashtag was distributed geographically.&lt;\/p&gt;\n&lt;p&gt;The scripts creates a variable, &amp;#8220;primary_geo&amp;#8221;, which saves the best possible geo data found. In addition, there are loops for figuring out how many tweets people made in total and how many users\u00a0provided some kind of geodata.&lt;\/p&gt;\n&lt;h6&gt;3.\u00a0Converting JSON to CSV with OpenRefine&lt;\/h6&gt;\n&lt;p&gt;Before geocoding and mapping the data, I needed\u00a0to\u00a0change the format of the data.\u00a0Twitter serves tweets as JSON objects and this is also how I process them in my script. But neither CartoDB or Fusion Tables work well with JSON.&lt;\/p&gt;\n&lt;p&gt;Instead of writing a script for converting JSON to CSV (or doing it in the original script) I just passed the data to &lt;a href=\"http:\/\/openrefine.org\/\"&gt;OpenRefine&lt;\/a&gt;,\u00a0converted it to a table and downloaded it as a CSV file.&lt;\/p&gt;\n&lt;h6&gt;4a. Mapping geocoded data with Google Fusion Tables&lt;\/h6&gt;\n&lt;p&gt;Next I wanted to\u00a0&lt;a href=\"https:\/\/en.wikipedia.org\/wiki\/Geocoding\"&gt;geocode&lt;\/a&gt; this data, so that I can plot it on a map. The purpose is, in other words, to transform location data into a longitude and latitude marking it&amp;#8217;s position on planet Earth.&lt;\/p&gt;\n&lt;p&gt;I usually use CartoDB out of personal preference, but due to their data limits I initially found Google Fusion Tables to be a better fit. To my great surprise, I noticed that Fusion Tables automatically geocodes any column in a table that is designated as &amp;#8216;location&amp;#8217;. So, after uploading my data to Fusion Tables, designating my &amp;#8220;primary geo&amp;#8221; column as a &amp;#8220;location&amp;#8221; column and creating a map the Fusion Table map earlier in the post was born.&lt;\/p&gt;\n&lt;p&gt;&lt;strong&gt;4b. Geocoding with Google Refine, mapping with CartoDB&lt;\/strong&gt;&lt;\/p&gt;\n&lt;p&gt;I give Fusion Tables\u00a0points for simplicity and clarity. But if you want to customize your map, it&amp;#8217;s not the best possible service. I quickly got frustrated when I wanted to make the data points reflect the amount of tweets each user had created. For this, CartoDB proved much better.&lt;\/p&gt;\n&lt;p&gt;The problem with CartoDB is, in my experience, that their geocoding quotas are pretty strict. This means that you can only get a limited amount of data geocoded through their service, much less than the 709 places I needed to geocode.&lt;\/p&gt;\n&lt;p&gt;So before uploading my data to CartoDB, I needed to geocode it somewhere. For this I again used OpenRefine, following\u00a0the excellent tutorial at the\u00a0&lt;a href=\"https:\/\/opensas.wordpress.com\/2013\/06\/30\/using-openrefine-to-geocode-your-data-using-google-and-openstreetmap-api\/\"&gt;Opensas blog&lt;\/a&gt;. There were small differences to the geocoding done by Fusion Tables. My own location at the time was &amp;#8220;Helsinki | Barcelona&amp;#8221;. OpenRefine interpreted this as &amp;#8220;Barcelona&amp;#8221; and Fusion Tables as &amp;#8220;Helsinki&amp;#8221;. Both approaches\u00a0use\u00a0the Google Maps API, so I&amp;#8217;m not sure why the difference. Anyway, the resulting map can be seen at the beginning of the post. I used the CartoDB web app to do small edits in the style of the map and borrowed a &lt;a href=\"https:\/\/www.mapbox.com\/help\/cartodb\/\"&gt;basemap from Mapbox&lt;\/a&gt;.&lt;\/p&gt;\n&lt;h6&gt;Conclusions&lt;\/h6&gt;\n&lt;p&gt;Producing the maps for this semi-tutorial gave me some interesting information both on geodata on Twitter as well as the relationship between local activist networks and\u00a0actions by international coalitions.&lt;\/p&gt;\n&lt;p&gt;Regarding geodata and Twitter, it seems like one can produce informative displays of the geographical distribution of different networks even if the available data is quite inexact. Many users provide their location with enough accuracy to geocode it and map it. This data is static rather than dynamic, as it doesn&amp;#8217;t provide information on where people tweet, but in large amounts it provides a good overview of where a specific hashtag has gained traction.&lt;\/p&gt;\n&lt;p&gt;Looking at the maps produced here, we notice that even though the action was supposedly\u00a0international, \u00a0Spanish Twitter really dominated the #StopBlackstone hashtag. Even many data points in the US are actually Spanish (like &amp;#8220;Valencia&amp;#8221; that both Fusion Tables and Google Refine coded into Florida, even though it&amp;#8217;s an autonomous province in Spain) and many random locations that seem to be in Spain, were coded into Latin America. The correct way to use the map would then to take a first look to get an approximation of were users\u00a0were clustered. After that, you can click at the data points in a cluster to get a better understanding of why users were clustered there and assess the validity of the geodata.&lt;\/p&gt;\n&lt;p&gt;Regarding the relationships between different levels of activity in networked movements: Combined with the fact that most pictures from the actions indicate a much larger mobilization at the Spanish offices of the Blackstone subsidiary Anticipa in Barcelona than anywhere else, it seems like the Spanish companerxs really dominated the action. My own gut feeling about this is that international actions usually only work if they are nurtured and amplified by networks that are more present in the everyday lives of people. \u00a0La PAH has built its power on countless assemblies, stopped evictions and occupations of banks and this is what they brought to the international mobilization.&lt;\/p&gt;\n","protected":false},"excerpt":{"rendered":"&lt;p&gt;This post gives a brief and general introduction to scraping Twitter with Python, extracting geodata from this data and mapping it using Fusion Tables and CartoDB. I talked\u00a0about this as well as some Facebook scraping I did with R in the working group on the Digitalization of Societies and Methods at the Annual Westermarck Conference &amp;hellip; &lt;\/p&gt;\n&lt;p class=\"link-more\"&gt;&lt;a href=\"http:\/\/www.mikaelbrunila.fi\/2017\/03\/27\/scraping-extracting-mapping-geodata-twitter\/\" class=\"more-link\"&gt;Continue reading&lt;span class=\"screen-reader-text\"&gt; &amp;#8220;Scraping, extracting and mapping geodata from Twitter&amp;#8221;&lt;\/span&gt;&lt;\/a&gt;&lt;\/p&gt;\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[13],"tags":[],"jetpack_featured_media_url":"","yoast_head":"\n&lt;title&gt;Scraping, extracting and mapping geodata from Twitter - Mikael Brunila&lt;\/title&gt;\n&lt;meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/&gt;\n&lt;link rel=\"canonical\" href=\"http:\/\/www.mikaelbrunila.fi\/2017\/03\/27\/scraping-extracting-mapping-geodata-twitter\/\" \/&gt;\n&lt;meta property=\"og:locale\" content=\"en_US\" \/&gt;\n&lt;meta property=\"og:type\" content=\"article\" \/&gt;\n&lt;meta property=\"og:title\" content=\"Scraping, extracting and mapping geodata from Twitter - Mikael Brunila\" \/&gt;\n&lt;meta property=\"og:description\" content=\"This post gives a brief and general introduction to scraping Twitter with Python, extracting geodata from this data and mapping it using Fusion Tables and CartoDB. I talked\u00a0about this as well as some Facebook scraping I did with R in the working group on the Digitalization of Societies and Methods at the Annual Westermarck Conference &amp;hellip; Continue reading &amp;quot;Scraping, extracting and mapping geodata from Twitter&amp;quot;\" \/&gt;\n&lt;meta property=\"og:url\" content=\"http:\/\/www.mikaelbrunila.fi\/2017\/03\/27\/scraping-extracting-mapping-geodata-twitter\/\" \/&gt;\n&lt;meta property=\"og:site_name\" content=\"Mikael Brunila\" \/&gt;\n&lt;meta property=\"article:published_time\" content=\"2017-03-26T22:00:39+00:00\" \/&gt;\n&lt;meta property=\"article:modified_time\" content=\"2017-04-22T16:07:20+00:00\" \/&gt;\n&lt;meta name=\"twitter:card\" content=\"summary\" \/&gt;\n&lt;meta name=\"twitter:title\" content=\"Scraping, extracting and mapping geodata from Twitter\" \/&gt;\n&lt;meta name=\"twitter:description\" content=\"This post gives a brief and general introduction to scraping Twitter with Python, extracting geodata from this data and mapping it using Fusion Tables and CartoDB.\" \/&gt;\n&lt;meta name=\"twitter:image\" content=\"http:\/\/www.mikaelbrunila.fi\/wp-content\/uploads\/2017\/03\/blackstone_twitter.jpg\" \/&gt;\n&lt;meta name=\"twitter:label1\" content=\"Written by\"&gt;\n\t&lt;meta name=\"twitter:data1\" content=\"mikael\"&gt;\n\t&lt;meta name=\"twitter:label2\" content=\"Est. reading time\"&gt;\n\t&lt;meta name=\"twitter:data2\" content=\"11 minutes\"&gt;\n&lt;script type=\"application\/ld+json\" class=\"yoast-schema-graph\"&gt;{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebSite\",\"@id\":\"http:\/\/www.mikaelbrunila.fi\/#website\",\"url\":\"http:\/\/www.mikaelbrunila.fi\/\",\"name\":\"Mikael Brunila\",\"description\":\"Geographer. Data Scientist. Researcher.\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":\"http:\/\/www.mikaelbrunila.fi\/?s={search_term_string}\",\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"http:\/\/www.mikaelbrunila.fi\/2017\/03\/27\/scraping-extracting-mapping-geodata-twitter\/#webpage\",\"url\":\"http:\/\/www.mikaelbrunila.fi\/2017\/03\/27\/scraping-extracting-mapping-geodata-twitter\/\",\"name\":\"Scraping, extracting and mapping geodata from Twitter - Mikael Brunila\",\"isPartOf\":{\"@id\":\"http:\/\/www.mikaelbrunila.fi\/#website\"},\"datePublished\":\"2017-03-26T22:00:39+00:00\",\"dateModified\":\"2017-04-22T16:07:20+00:00\",\"author\":{\"@id\":\"http:\/\/www.mikaelbrunila.fi\/#\/schema\/person\/61a7c4c2fa5f0a6105ee25488536fe94\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"http:\/\/www.mikaelbrunila.fi\/2017\/03\/27\/scraping-extracting-mapping-geodata-twitter\/\"]}]},{\"@type\":\"Person\",\"@id\":\"http:\/\/www.mikaelbrunila.fi\/#\/schema\/person\/61a7c4c2fa5f0a6105ee25488536fe94\",\"name\":\"mikael\",\"image\":{\"@type\":\"ImageObject\",\"@id\":\"http:\/\/www.mikaelbrunila.fi\/#personlogo\",\"inLanguage\":\"en-US\",\"url\":\"http:\/\/0.gravatar.com\/avatar\/3c5e50e6f330a74dfc590a4a3265ea04?s=96&amp;d=mm&amp;r=g\",\"caption\":\"mikael\"}}]}&lt;\/script&gt;\n","_links":{"self":[{"href":"http:\/\/www.mikaelbrunila.fi\/wp-json\/wp\/v2\/posts\/248"}],"collection":[{"href":"http:\/\/www.mikaelbrunila.fi\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.mikaelbrunila.fi\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.mikaelbrunila.fi\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/www.mikaelbrunila.fi\/wp-json\/wp\/v2\/comments?post=248"}],"version-history":[{"count":33,"href":"http:\/\/www.mikaelbrunila.fi\/wp-json\/wp\/v2\/posts\/248\/revisions"}],"predecessor-version":[{"id":282,"href":"http:\/\/www.mikaelbrunila.fi\/wp-json\/wp\/v2\/posts\/248\/revisions\/282"}],"wp:attachment":[{"href":"http:\/\/www.mikaelbrunila.fi\/wp-json\/wp\/v2\/media?parent=248"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.mikaelbrunila.fi\/wp-json\/wp\/v2\/categories?post=248"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.mikaelbrunila.fi\/wp-json\/wp\/v2\/tags?post=248"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}