Text Mining Databases: Publicly available databases

HathiTrust

• HathiTrust
• "A partnership of academic & research institutions, offering a collection of millions of titles digitized from libraries around the world."
• "HathiTrust Digital Library is a digital preservation repository and highly functional access platform. It provides long-term preservation and access services for public domain and in copyright content from a variety of sources, including Google, the Internet Archive, Microsoft, and in-house partner institution initiatives." "The HathiTrust Research Center (HTRC) facilitates non-profit and educational uses of the HathiTrust Digital Library by enabling computational analysis of public domain works and (on limited terms) in-copyright works from its collection."
• Data Availability and APIs ♦ HathiTrust Research Center ♦ HTRC Extracted Features Dataset
• ?See "New Metadata Recipes for Old Cookbooks: Creating and Analyzing a Digital Collection Using the HathiTrust Research Center Portal", Code4Lib Journal 37, 18 July 2017.

Internet Archive

• Internet Archive
• "A non-profit library of millions of free books, movies, software, music, websites, and more."
• The Internet Archive "runs several services including the Archive.org search engine, OpenLibrary, and the Wayback Machine. In alignment with its mission, the Archive encourages developers to add media to archive.org, as well as to consume and re-purpose media and its metadata, for the great good of our community and beyond."
• Developers  ♦  Internet Archive Developers
• The Internet Archive Metadata API ♦  See Data Mining the Internet Archive Collection ♦  openlibrary.org

News

• List of news media APIs

And more...

• arXiv ♦ arXiv API
• Consortium of European Social Science Data Archives (CESSDA), "a one-stop shop for searching and finding European social science data. The data described may be quantitative, qualitative or mixed-modes data, cross-sectional or longitudinal, recently collected or historical data."
• Constellate (JSTOR). For free version, see Constellate tab.
• Data Is Plural, Jeremy Singer-Vine. "Useful/curious datasets."
• Dataverse Project, Harvard. "An open source web application to share, preserve, cite, explore, and analyze research data".
• Directory of Open Access Journals (DOAJ) ♦  FAQ: API   ♦  FAQ: Metadata
• Documenting the American South ♦ DocSouth Data
• Europeana ♦  Introduction to the Europeana APIs
• The General Index, from public.resource.org. ♦ "The General Index consists of 3 tables derived from 107,233,728 journal articles. A table of n-grams, ranging from unigrams to 5-grams, is extracted using SpaCy. Each of the 355,279,820,087 rows of the n-gram table consists of an n-gram coupled with a journal article id. A second table is constructed using Yake and consists of 19,740,906,314 rows, each with a keywords and an article id. A third table associates an article id with metadata." ♦ About, Nature.
• Hansard archive, U.K. Parliament. ♦ U.K. Parliamentary debates from 1803. "Scanned from hard copy sources. They are in XML format and held within a zip file."
• Journal of Open Humanities Data, "features peer reviewed publications describing humanities research objects or techniques with high potential for reuse. Humanities subjects of interest to JOHD include, but are not limited to Art History, Classics, History, Library Science, Linguistics, Literature, Media Studies, Modern Languages, Music and musicology, Philosophy, Religious Studies, etc."
• kaggle Datasets. "Join over 14M+ machine learners to share, stress test, and stay up-to-date on all the latest ML techniques and technologies. Discover a huge repository of community-published models, data & code for your next project."
• National Center for Biotechnology Information "advances science and health by providing access to biomedical and genomic information".
• Papers With Code: Datasets, machine learning datasets.
• Project Gutenberg ♦  Information About Robot Access to our Pages
• Public Library of Science (PLOS) ♦ PLOS API
• Projects Using the TEI (Text Encoding Initiative) encoding scheme. Text Encoding Initiative Consortium.

• The Social Media Archive at ICPSR (SOMAR). "The Social Media Archive at ICPSR (SOMAR) is a collection of public and restricted data from various social media platforms organized and stored for research and analysis purposes. With their data available to the community, SOMAR aims to help researchers and community members better understand social media behavior and trends. In addition, the data can inform the development of new technologies and services." "Much of SOMAR's data will be available through approved restricted data applications, and the data will be accessed through a virtual data enclave." Source

Twitter

• The Twitter's API may be unreliable.
• We may be able to help you gather limited historical Twitter data using Martin Hawksey's Twitter Archiving Google Sheet (TAGS).

Facebook and Instagram

• Because Facebook and Instagram do not provide public APIs, we are unable to collect text from them.

congress.gov (Library of Congress)

• congress.gov
• Provides access to U.S. federal legislative information.
• How To Download Your Congress.gov Search Results
• congress.gov API  ♦  GitHub repository
• See also govinfo.gov (this page)

govinfo (Government Publishing Office)

• govinfo.gov
• "Provides free public access to official publications from all three branches of the Federal Government."
• Developer Hub  ♦  Bulk Data Repository
• Statute Compilations
• "Tools available for software developers and data users include XML for bulk download, a link service, design documentation, and more available on GPO's GitHub page."
• Most data in govinfo are not copyrighted.
• Format: Text, PDF, XML
• See also congress.gov (this page)

Library of Congress

• LC for Robots
• Library of Congress digital collections.

regulations.gov

• regulations.gov, from partner agencies participating in the eRulemaking Program
• "When Congress passes laws, federal agencies implement those laws through regulations. These regulations vary in subject, but include everything from ensuring water is safe to drink to setting health care standards. Regulations.gov is the place where users can find and comment on regulations. The APIs allow for users to find creative ways to present regulatory data."
• Regulations.gov API

PubMed (U.S. National Library of Medicine)

• PubMed, U.S. National Library of Medicine. ♦  Developer Resources
• The PubMed Central (PMC) Article Datasets. "PubMed Central® (PMC) is a free full-text archive of biomedical and life sciences journal article at the U.S. National Institutes of Health's National Library of Medicine (NIH/NLM). The PubMed Central (PMC) Article Datasets include full-text articles archived in PMC and made available under license terms that allow for text mining and other types of secondary analysis and reuse."

And more from the legislative branch ...

• Legislative Documents in XML at the United States House of Representatives
• House of Representatives Document Repository
• House of Representatives Floor Proceedings (XML)
• House Roll Call Votes
• Office of the Law Revision Counsel, U.S. Code
• Senate.gov XML Sources
• Chronicling America ♦  API

And more from the U.S. federal government...

• Office of the Historian, U.S. Department of State. Includes Foreign Relations of the United States (FRUS); Principal Officers & Chiefs of Mission; Foreign Travels of the President and Secretary of State; Visits by Foreign Leaders and Heads of State; Administrative Timeline of the Department of State; Milestones in the History of U.S. Foreign Relations; Tweets; etc.
• Data.gov, "the United States government's open data website. It provides access to datasets published by agencies across the federal government".
• Federal Register, Office of the Federal Register ♦  Reader Aids for developers
• Consumer Complaint Database, Consumer Financial Protection Bureau  ♦  Downloads, API
• FRASER, Federal Reserve Bank of St. Louis. "FRASER is a digital library of U.S. economic, financial, and banking history—particularly the history of the Federal Reserve System." ♦ FRASER® API

Government information from other sources

• the @unitedstates project.
  "a shared commons of data and tools for the United States [...] Featuring work from people with the Sunlight Foundation, GovTrack.us, the New York Times, the Electronic Frontier Foundation, and the Internet. "

For your reference

United States Legislative Markup (USLM) XML schema

• Caselaw Access Project ♦ Usage & access. From Library Innovation Lab, Harvard University. "CAP includes all official, book-published United States case law — every volume designated as an official report of decisions by a court within the United States."
• CourtListener ♦ CourtListener APIs and Bulk Legal Data
• North Carolina Jim Crow Laws, On The Books: Jim Crow and Algorithms of Resistance. University of North Carolina.