Extend Historical Data For Academics
Academics should receive increased access (beyond seven days) to historical data. The seven-day limitation severely limits academic inquiry and is a primary driver behind extravagant costs for those using Twitter data in their research.
Kurt Wirth
shared this idea
12 comments
-
[Deleted User]
commented
Good
-
Stu Shulman
commented
Perhaps restore sifter.texifter.com as an option for self serve historical data access with valid academic credentials.
-
Damilola Adegoke
commented
We need more access to historical data.
-
Anders Olof Larsson
commented
Historical data access, for instance to trace the development of specific hashtags or specific accounts, is highly important. As most of us are on a budget, I would suggest a highly subsidised or even free access fee. Plenty of commercial entities offer their products and services free for academics and students, such as Tableau.
-
Kurt Wirth
commented
That's great, Adam. Good info, but this is very expensive, no?
-
Adam Tornes
commented
Thank you for all of the feedback.
To address a point of confusion because multiple people in this thread commented asking for such: (ignoring any viability and affordability considerations) beyond our Premium 30-Day Search API, we do indeed also have a Premium Full-archive Search API available for use by _anyone_ https://developer.twitter.com/en/docs/tweets/search/quick-start/premium-full-archive
-
Schin सचिन गर्ग Garg
commented
I am using Twitter data to understand how Indians use Twitter and require historical data (user timelines) going back quite some time - neither the 7 day nor the 30 day actually suffices. I am in the process of writing a grant for this and require some estimates of the budgetary requirements. Sadly, no one from Twitter has responded to my tweets and I cannot find a way to simply ask folks how much does this access cost? Anyone, anyone here who can help me with this?
-
Nandan Rao
commented
Agreed. Currently, if I have the foresight, I can set up a machine to stream live tweets, on the free tier, about the topic of interest and then do the research after I've gathered all the data. Unfortunately, often the best ideas involve past data. It's very frustrating that if I had had the foresight, I could have the data for free, but if I think about it after-the-fact, the costs to obtain historical data for long periods (more than a month) are often prohibitive.
This creates an incentive for organizations to gather streaming data "just in case" and then use that data to collaborate with other researchers. Several projects I know of use this approach because it is too pricey to gather historical data directly from Twitter, and if someone has happened to gather it live, even if it isn't perfect, it will be a more realistic option for research data.
Similarly, it would be great if there were a way to work with the data first and pay for it later. If I can do some initial analysis on the data and things look promising, I can raise money via grants and am happy for that money to support the system that gathered the data (Twitter). Or even better, it would be great if Twitter could arrange directly from large academic funding organizations that if useful research is created with Twitter data, they get support, provided Twitter provides easy and free access to the data for researchers. Thus, academics get free and unfettered access to data, Twitter gets compensated as long as its data is insightful and useful, and the funding organizations are paying for results.
-
Igor Brigadir
commented
Yes, ideally, the fullarchive premium endpoint should be the one made available to researchers.
-
Kurt Wirth
commented
Unfortunately, the Premium limit of 30 days still wouldn't be enough for most researchers, I believe.
-
Igor Brigadir
commented
I agree with providing historical data, but I think it's better to offer cheaper / extended access to Premium instead, rather than extending the time range of the 7 day Standard Search.
-
I need to access more historical data the current APIs make available.
