.. _Corpora:

#################
Corpora
#################

This page contains several corpora relevant to political science research, categorized by country and key source, a link for where to find them and a note if they are not free. We are working with many on these to develop Texti. 

Parties and elections
^^^^^^^^^^^^^^^^^^^^^

.. list-table::
   :header-rows: 1

   * - Item
     - Country
     - Description
     - Access
     - Link
   * - Manifesto Project
     - 51 inc. OECD
     - All political manifestos from the first democratic election onwards.
     - API; stata, spss, csv, xslx
     - `Here <https://visuals.manifesto-project.wzb.eu/mpdb-shiny/cmp_dashboard_corpus_doc/>`_
   * - Speeches
     - UK
     - Speeches from party leaders from 1895 to today
     - HTML on site
     - `Here <http://www.britishpoliticalspeech.org/speech-archive.htm>`_
   * - Regional manifesto
     - Spain
     - 1980 to 2019, all regional parties
     - Download
     - `Here <http://www.regionalmanifestosproject.com/ingles/download-dataset>`_
   * - Regional manifesto
     - Wales and Scotland
     - 1999 and 2016
     - Download
     - `Here <http://www.regionalmanifestosproject.com/ingles/download-dataset>`_
   * - Regional manifesto
     - Italy
     - Fragmented depending on region
     - Download
     - `Here <http://www.regionalmanifestosproject.com/ingles/download-dataset>`_


Parliament Activity
^^^^^^^^^^^^^^^^^^^

.. list-table::
   :header-rows: 1

   * - Item
     - Country
     - Description
     - Access
     - Link
   * - Parliamentary Questions Answered
     - UK
     - 278428 questions; csv
     - API
     - `Here <http://explore.data.parliament.uk/?endpoint=answeredquestions#download-list>`_
   * - EP Plenary
     - European Union
     - 1997 to 2019
     - HTTP resolvable URIs
     - `Here <https://linkedpolitics.project.cwi.nl/web/html/home.html>`_
   * - Parliament Debates
     - France
     - Debates of the 15th legislature
     - HTTP resolvable URIs; XML
     - `Here <http://data.assemblee-nationale.fr/travaux-parlementaires/debats>`_
   * - Lords Written Questions
     - UK
     - 52004 questions
     - API; csv
     - `Here <http://explore.data.parliament.uk/?endpoint=lordswrittenquestions#download-list>`_
   * - Commons Written Questions
     - UK
     - 275929 questions
     - API; csv
     - `Here <http://explore.data.parliament.uk/?endpoint=commonswrittenquestions#download-list>`_
   * - Questions to the  Government
     - France
     - Since 2017
     - HTTP resolvable URIs
     - `Here <http://data.assemblee-nationale.fr/travaux-parlementaires/questions/questions-au-gouvernement>`_
   * - Questions to the  Government - without debates
     - France
     - Since 2017
     - HTTP resolvable URIs
     - `Here <http://data.assemblee-nationale.fr/travaux-parlementaires/questions/questions-orales-sans-debat>`_
   * - Written quesions to  the Government
     - France
     - Since 2017
     - HTTP resolvable URIs
     - `Here <http://data.assemblee-nationale.fr/travaux-parlementaires/questions/questions-ecrites>`_
   * - Parliamentary Debates  on Europe
     - France
     - 2002 to 2012
     - HTTP resolvable URIs
     - `Here <https://www.ortolang.fr/market/item/fr-parl/v1>`_
   * - Parliamentary speeches
     - Austria, Czech Republic, Germany, Denmark, Netherlands, NZ, Spain, Sweden, UK, Ireland
     - 21 to 32 years of data
     - API on DataVerse; full-text vectors in rds
     - `Here <https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/L4OAKN>`_
   * - Parliament Rules
     - UK
     - 1811 to 2019
     - Download
     - `Here <https://parlrulesdata.org>`_
   * - Parliament Rules
     - Ireland
     - 1922 to 2020
     - Download
     - `Here <https://parlrulesdata.org>`_
   * - Debates and Replies to Questions
     - Ireland
     - All
     - API
     - `Here <https://api.oireachtas.ie>`_
   * - Senate "Dossiers Legislatifs"
     - France
     - Documents discussed since 1977
     - Download
     - `Here <https://data.senat.fr/dosleg/>`_
   * - Amendments by the Senate
     - France
     - Amendments since 2001
     - Download
     - `Here <Amendements>`_
   * - Lords Bill Amendments
     - UK
     - 11727 Amendments
     - API
     - `Here <http://explore.data.parliament.uk/?endpoint=lordsbillamendments#download-list>`_
   * - Questions to the Government (Senate)
     - France
     - Since 1978
     - Download
     - `Here <https://data.senat.fr/la-base-questions/>`_
   * - Research Briefings
     - UK
     - 9739 briefings
     - API, csv with 500 records limit
     - `Here <http://explore.data.parliament.uk/?endpoint=researchbriefings#download-list>`_
   * - Proceedings
     - European union
     - 1996-2011
     - Download, xml
     - `Here <https://www.statmt.org/europarl/>`_


Legislative Documents
^^^^^^^^^^^^^^^^^^^^^

.. list-table::
   :header-rows: 1

   * - Item
     - Country
     - Description
     - Access
     - Link
   * - All legislation
     - European Union
     - Summaries of EU legislation (full corpus exists but wrong license)
     - HTML on site (can email Dimiter Toshkov for ``Python`` script)
     - `Here <https://eur-lex.europa.eu/browse/summaries.html>`_
   * - Trade agreements
     - European Union
     - All free trade agreements
     - List of linked PDFs
     - `Here <https://trade.ec.europa.eu/tradehelp/free-trade-agreements>`_
   * - Bills
     - UK
     - All bills since 2007
     - API
     - `Here <http://explore.data.parliament.uk/?endpoint=bills>`_
   * - All Legal Texts
     - France
     - Constitution, laws and decrees, court rulings, treaties (in French and translated)
     - Downloadable + beta API
     - `Here <https://www.legifrance.gouv.fr>`_
   * - Legislation
     - Wales
     - All Bills, Acts, Marshalled lists
     - XML export
     - `Here <https://senedd.wales/en/help/our-information/Pages/Open-data.aspx>`_
   * - The Record of Proceedings
     - Wales
     - All proceedings
     - XML export
     - `Here <https://senedd.wales/en/help/our-information/Pages/Open-data.aspx>`_
   * - International Environment Agency
     - World
     - Most environmental treaties and agreements
     - List of .txt on the website
     - `Here <https://iea.uoregon.edu/text-index>`_
   * - Bills and Acts
     - Ireland
     - All
     - API
     - `Here <https://api.oireachtas.ie>`_
   * - All trade agreements
     - All
     - All
     - Download
     - `Here <https://github.com/mappingtreaties/tota>`_



Identity and Culture
^^^^^^^^^^^^^^^^^^^^

.. list-table::
   :header-rows: 1

   * - Item
     - Country
     - Description
     - Access
     - Link
   * - National Anthems
     - World
     - 194 countries
     - Download
     - `Here <https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/PZG8TH>`_



Presidential & Governmental Activity
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. list-table::
   :header-rows: 1

   * - Item
     - Country
     - Description
     - Access
     - Link
   * - Political speeches
     - UK
     - 8000+ political speeches on British Politics
     - HTML
     - `Here <http://www.ukpol.co.uk>`_
   * - Official correspondence
     - UK
     - All official correspondence of PMs
     - API
     - `Here <https://www.nationalarchives.gov.uk/help-with-your-research/research-guides/prime-ministers-office-records/>`_
   * - PM transcripts
     - Australia
     - Ministerial transcripts from 1940s to date
     - API; xml
     - `Here <https://pmtranscripts.pmc.gov.au/developers>`_
   * - Speeches
     - EU
     - All ECB President / VP speeches
     - Download; csv
     - `Here <https://www.ecb.europa.eu/press/key/html/downloads.en.html>`_
   * - Speeches
     - Germany
     - 6,685 speeches by 71 officials, spanning a time from 1984 to 2017
     - Download, xml
     - `Here <http://adrien.barbaresi.eu/corpora/speeches/>`_
   * - Speeches
     - EU
     - 18,403 speeches from EU leaders from 2007 to 2015
     - API from DataVerse; csv raw speeches, and term-document matrices in R
     - `Here <https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/XPCVEI>`_
   * - State of the Nation
     - South Africa
     - 1990 to 2018
     - Download from Kaggle; txt per speech
     - `Here <https://www.kaggle.com/allank/state-of-the-nation-1990-2017>`_


Participative democracy
^^^^^^^^^^^^^^^^^^^^^^^

.. list-table::
   :header-rows: 1

   * - Item
     - Country
     - Description
     - Access
     - Link
   * - Public consultations
     - France
     - Recent public consultations
     - HTTP-resolvable URIs
     - `Here <http://data.assemblee-nationale.fr/autres/consultations-citoyennes>`_
   * - E-petitions
     - UK
     - All official e-petitions
     - API; JSON, xml, csv, HTML
     - `Here <https://explore.data.parliament.uk/?learnmore=e-Petitions>`_



News and Media
^^^^^^^^^^^^^^

.. list-table::
   :header-rows: 1

   * - Item
     - Country
     - Description
     - Access
     - Link
   * - EUvsDisinfo
     - Europe
     - Debunked news articles by European External Action Services
     - API; HTML
     - `Here <https://euvsdisinfo.eu/disinformation-cases/>`_
   * - New York Times
     - All
     - Archive metadata, books, comments, reviews, most popular articles
     - API; JSON
     - e.g. `Here <https://developer.nytimes.com/apis>`_
   * - Public debates over European integration
     - Austria, Britain, France, Germany, Sweden, and Switzerland
     - 1970s to 2012 from newspapers
     - csv, dta
     - `Here <https://poldem.eui.eu/download/>`_
   * - Public debates over globalization issues
     - Austria, Britain, France, Germany, the Netherlands, and Switzerland
     - 2004-2006 from newspapers
     - csv, dta
     - `Here <https://poldem.eui.eu/download/>`_
   * - Archive of Political emais
     - Australie, Canada, France, Germany, Ireland, Italy, NZ, UK, USA
     - 348,680 emails
     - HTML
     - `Here <http://politicalemails.org>`_
   * - News articles
     - Not specified
     - 9+ million articles and metadata for each
     - CSV split in 1GB zip files, download from GitHub
     - `Here <https://github.com/several27/FakeNewsCorpus>`_
   * - Poliwoops
     - Many countries including USA, UK and most European countries
     - Deleted tweets by public officials and politicians
     - API; JSON
     - `Here <https://www.politwoops.co.uk/page/api>`_


Messy list of promising websites
--------------------------------

Websites that might be goldmines but would require some time to explore. 


* European Language Resource Coordincation

  * A lot of legal / official documents translated and sometimes already processed. E.g. IP case law, audits, a lot of legal texts from EU countries (not sure how useful they really are, but it is a *lot* of them, there might be some interesting ones)
  * https://elrc-share.eu

* 
  Clarin


  * List of 24 parliamentary corpora, not all easy access
  * https://www.clarin.eu/resource-families/parliamentary-corpora

* 
  EveryCRSReport.com 


  * Reports from the Congressional Research Service — essentially the national legislature’s think-tank. 
  * https://www.everycrsreport.com/

* Supreme court transcripts

  * https://www.oyez.org/

Complementary text data
-----------------------

Texts that are not necessarily directly relevant to political science research but are used for context / complement. E.g. annotate etc.


* Wikipedia or other "ground truth" sources
* Network data
* Dictionaries: e.g. sentiment or emotions to use automated dictionary methods with one click

----

US Political Science focus
^^^^^^^^^^^^^^^^^^^^^^^^^^

.. list-table::
   :header-rows: 1

   * - Item
     - Country
     - Description
     - Access
     - Link
   * - General Social Survey
     - US
     - General Social Survey (GSS) monitors societal change in the US
     - Download: for SPSS, STATA
     - `Here <http://gss.norc.org/get-the-data>`_
   * - The Supreme Court Database
     - US
     - Case Centered Data - Total Rows : 13,533
     - Download: CSV, DTA (STATA), POR (SPSS), RDATA, XLSX
     - `Here <http://scdb.wustl.edu/data.php>`_
   * - The Supreme Court Database
     - US
     - Justice Centered Data - Total Rows : 121,224
     - Download: CSV, DTA (STATA), POR (SPSS), RDATA, XLSX
     - `Here <http://scdb.wustl.edu/data.php>`_
   * - Congressional speech data
     - US
     - Congressional-speech corpus includes labels for whether the speaker supported or opposed, by-name references between speakers, and the scores that our agreement/disagreement classifier(s), debate and related extracted information. (9.8 Mb, tar.gz format)
     - Download: compressed tar.gz, multiple types including CSV
     - `Here <http://www.cs.cornell.edu/home/llee/data/convote.html>`_
   * - ANES
     - US
     - Electoral behavior, political participation, and public opinion studies - Time Series Studies , Pilot Studies, Special Studies
     - Download
     - `Here <https://electionstudies.org/data-center/>`_
   * - CorPS
     - US
     - CORPS is a corpus of political speeches tagged with specific audience reactions, such as APPLAUSE or LAUGHTER.
     - Request from marco.guerini[at]trentorise.eu and strappa[at]fbk.eu
     - `Here <https://hlt-nlp.fbk.eu/corps>`_
   * - Congressional Record for the 43rd-114th Congresses
     - US
     - Parsed Speeches and Phrase Counts
     - Download: zip of organized txt files
     - `Here <https://data.stanford.edu/congress_text>`_
   * - GDELT
     - US
     - All events from broadcast, print, and web news from nearly every corner of every country in over 100 languages
     - Download: CSV 
     - `Here <https://www.gdeltproject.org/data.html#rawdatafiles>`_
   * - The American Presidency Project
     - US
     - Presidential documents, papers, press, orders, memoranda etc
     - HTML
     - `Here <https://www.presidency.ucsb.edu/documents>`_
   * - Full text corpus data
     - US
     - 10 large corpora of English: iWeb, COCA, COHA, NOW, Coronavirus, GloWbE, TV Corpus, Movie Corpus, Soap Corpus, Wikipedia
     - Purchase raw data in 3 formats
     - `Here <https://www.corpusdata.org/>`_
   * - GovInfo
     - US
     - Congressional Bills; Bill Status; Bill Summaries; Commerce Business Daily; Code of Federal Regulations (Annual Edition); Electronic Code of Federal Regulations; Federal Register; United States Government Manual; House Rules and Manual; Privacy Act Issuances; Public Papers of the Presidents of the United States; Supreme Court Decisions 1937-1975 (FLITE)
     - Download: XML
     - `Here <https://www.govinfo.gov/bulkdata>`_
   * - DIME PLUS
     - US
     - Database on Ideology, Money in Politics, and Elections: Public version 2.0
     - Download: compressed CSV
     - `Here <https://data.stanford.edu/dime>`_
   * - Replication data for: Tracing the Flow of Policy Ideas in Legislatures: A Text Reuse Approach
     - US
     - Replication Data
     - Download: compressed archive 
     - `Here <https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/27584>`_
   * - CONGRESSIONAL & FEDERAL - Government Web Harvests
     - US
     - The National Archives and Records Administration (NARA) web harvests (i.e. capture) of Federal Agency public web sites since 2004
     - Web harvests
     - `Here <https://www.webharvest.gov/>`_
   * - Congress.gov - Bill Status
     - US
     - Bill Status data includes all data from the existing Bill Summaries data se
     - XML bulk data; API
     - `Here <https://www.congress.gov/about/data>`_