Open data is the idea that some data should be freely available to everyone to use and republish as they wish, without restrictions from copyright, patents or other mechanisms of control. The goals of the open data movement are similar to those of other "open" movements such as open source, open hardware, open content and open access. The philosophy behind open data has been long established (for example in the Mertonian tradition of science), but the term "open data" itself is recent, gaining popularity with the rise of the Internet and World Wide Web and, especially, with the launch of open-data government initiatives such as Data.gov and Data.gov.uk.
The concept of open data is not new; but a formalized definition is relatively new—the primary such formalization being that in the Open Definition which can be summarized in the statement that "A piece of data is open if anyone is free to use, reuse, and redistribute it – subject only, at most, to the requirement to attribute and/or share-alike."
Open data may include non-textual material such as maps, genomes, connectomes, chemical compounds, mathematical and scientific formulae, medical data and practice, bioscience and biodiversity. Problems often arise because these are commercially valuable or can be aggregated into works of value. Access to, or re-use of, the data is controlled by organisations, both public and private. Control may be through access restrictions, licenses, copyright, patents and charges for access or re-use. Advocates of open data argue that these restrictions are against the communal good and that these data should be made available without restriction or fee. In addition, it is important that the data are re-usable without requiring further permission, though the types of re-use (such as the creation of derivative works) may be controlled by a license.
A typical depiction of the need for open data:
Numerous scientists have pointed out the irony that right at the historical moment when we have the technologies to permit worldwide availability and distributed process of scientific data, broadening collaboration and accelerating the pace and depth of discovery ... we are busy locking up that data and preventing the use of correspondingly advanced technologies on knowledge.
Creators of data often do not consider the need to state the conditions of ownership, licensing and re-use; instead presuming that not asserting copyright puts the data into the public domain. For example, many scientists do not regard the published data arising from their work to be theirs to control and consider the act of publication in a journal to be an implicit release of data into the commons. However, the lack of a license makes it difficult to determine the status of a data set and may restrict the use of data offered in an "Open" spirit. Because of this uncertainty it is also possible for public or private organizations to aggregate said data, protect it with copyright and then resell it.
The issue of indigenous knowledge (IK) poses a great challenge in terms of capturing, storage and distribution. Many societies in third-world countries lack the technicality processes of managing the IK.
At his presentation at the XML 2005 conference, Connolly displayed these two quotations regarding open data:
"I want my data back." (Jon Bosak circa 1997)
"I've long believed that customers of any application own the data they enter into it." (This quote refers to Veen's own heart-rate data.)
Arguments for and against
The debate on Open Data is still evolving. The best open government applications seek to empower citizens, to help small businesses, or to create value in some other positive, constructive way. Opening government data is only a way-point on the road to improving education, improving government, and building tools to solve other real world problems. While many arguments have been made categorically, the following discussion of arguments for and against open data highlights that these arguments often depend highly on the type of data and its potential uses.
Arguments made on behalf of Open Data include the following:
"Data belong to the human race". Typical examples are genomes, data on organisms, medical science, environmental data following the Aarhus Convention
Public money was used to fund the work and so it should be universally available.
It was created by or at a government institution (this is common in US National Laboratories and government agencies)
Facts cannot legally be copyrighted.
Sponsors of research do not get full value unless the resulting data are freely available.
Restrictions on data re-use create an anticommons.
Data are required for the smooth process of running communal human activities and are an important enabler of socio-economic development (health care, education, economic productivity, etc.).
In scientific research, the rate of discovery is accelerated by better access to data.
Making data open helps combat "data rot" and ensure that scientific research data are preserved over time.
It is generally held that factual data cannot be copyrighted. However, publishers frequently add copyright statements (often forbidding re-use) to scientific data accompanying publications. It may be unclear whether the factual data embedded in full text are part of the copyright.
While the human abstraction of facts from paper publications is normally accepted as legal there is often an implied restriction on the machine extraction by robots.
Unlike Open Access, where groups of publishers have stated their concerns, Open Data is normally challenged by individual institutions. Their arguments have been discussed less in public discourse and there are fewer quotes to rely on at this time.
Arguments against making all data available as Open Data include the following:
Government funding may not be used to duplicate or challenge the activities of the private sector (e.g. PubChem).
Governments have to be accountable for the efficient use of taxpayer's money: If public funds are used to aggregate the data and if the data will bring commercial (private) benefits to only a small number of users, the users should reimburse governments for the cost of providing the data.
The revenue earned by publishing data permits non-profit organisations to fund other activities (e.g. learned society publishing supports the society).
The government gives specific legitimacy for certain organisations to recover costs (NIST in US, Ordnance Survey in UK).
Privacy concerns may require that access to data is limited to specific users or to sub-sets of the data.
Collecting, 'cleaning', managing and disseminating data are typically labour- and/or cost-intensive processes – whoever provides these services should receive fair remuneration for providing those services.
Sponsors do not get full value unless their data is used appropriately – sometimes this requires quality management, dissemination and branding efforts that can best be achieved by charging fees to users.
Often, targeted end-users cannot use the data without additional processing (analysis, apps etc.) – if anyone has access to the data, none may have an incentive to invest in the processing required to make data useful (typical examples include biological, medical, and environmental data).
The goals of the Open Data movement are similar to those of other "Open" movements.
Open access is concerned with making scholarly publications freely available on the internet. In some cases, these articles include open datasets as well.
Open content is concerned with making resources aimed at a human audience (such as prose, photos, or videos) freely available.
Open knowledge. Open Knowledge International argues for Openness in a range of issues including, but not limited to, those of Open Data. It covers (a) scientific, historical, geographic or otherwise (b) Content such as music, films, books (c) Government and other administrative information. Open data is included within the scope of the Open Knowledge Definition, which is alluded to in Science Commons' Protocol for Implementing Open Access Data.
Open notebook science refers to the application of the Open Data concept to as much of the scientific process as possible, including failed experiments and raw experimental data.
Open source (software) is concerned with the licenses under which computer programs can be distributed and is not normally concerned primarily with data.
Open educational resources are freely accessible, openly licensed documents and media that are useful for teaching, learning, and assessing as well as for research purposes.
Open research/Open science/Open science data (Linked open science) means an approach to open and interconnect scientific assets like data, methods and tools with Linked Data techniques to enable transparent, reproducible and transdisciplinary research.
Funders' mandates
Several funding bodies which mandate Open Access also mandate Open Data. A good expression of requirements (truncated in places) is given by the Canadian Institutes of Health Research (CIHR):
to deposit bioinformatics, atomic and molecular coordinate data, experimental data into the appropriate public database immediately upon publication of research results.
to retain original data sets for a minimum of five years after the grant. This applies to all data, whether published or not.
Note the fundamental requirement to be able to replicate the experiment.
Other bodies active in promoting the deposition of data as well as fulltext include the Wellcome Trust. An academic paper published in 2013 advocated that Horizon 2020 (the science funding mechanism of the EU, due to launch in 2014) should mandate that funded projects hand in their databases as "deliverables" at the end of the project, so that they can be checked for third party usability then shared.
Several mechanisms restrict access to or reuse of data. They include:
making data available for a charge.
compilation in databases or websites to which only registered members or customers can have access.
use of a proprietary or closed technology or encryption which creates a barrier for access.
copyright forbidding (or obfuscating) re-use of the data, including the use of "no derivatives" requirements.
patent forbidding re-use of the data (for example the 3-dimensional coordinates of some experimental protein structures have been patented).
restriction of robots to websites, with preference to certain search engines.
aggregating factual data into "databases" which may be covered by "database rights" or "database directives" (e.g. Directive on the legal protection of databases).
time-limited access to resources such as e-journals (which on traditional print were available to the purchaser indefinitely).
"webstacles", or the provision of single data points as opposed to tabular queries or bulk downloads of data sets.
political, commercial or legal pressure on the activity of organisations providing Open Data (for example the American Chemical Society lobbied the US Congress to limit funding to the National Institutes of Health for its Open PubChem data).
ArcGIS Open Data
Bloomberg L.P. via Financial Instrument Global Identifier also see: Bloomberg Open Symbology
Blue Obelisk
Center for Government Excellence, Johns Hopkins University
CiteSeer
Common Crawl
cTuning.org
Datos Peru
Development Gateway
Ecodesk
ENGAGE platform
Factual
Freebase
Free Our Data
GOPI: Geospatial Open Data Portal 4 India
GEO: Group on Earth Observations
Global Open Data in Agriculture and Nutrition- GODAN
Hack de Overheid
Health Data Consortium
Information Retrieval Facility
Institute for Development of Freedom of Information (IDFI), Georgia
International Development Research Centre
Junar
Knoema
LIBER
LinkedScience.org
Newsjelly
OMG standard
Open & Agile Smart Cities (OASC)
OpenCorporates
OpenDataSoft
The Open Data Innovation Gathering
Open Data Institute
Open Data Nottingham
Opendata.ch
Open Energy Modelling Initiative
The Open Institute
Open Knowledge International (was Open Knowledge Foundation)
Open Power System Data
Open State Foundation
Quandl
Renewable Energy & Energy Efficiency Partnership
Rijksmuseum Amsterdam
Scholarly Publishing and Academic Resources Coalition
Socrata
Sunlight Foundation
Talis
ThinkData Works with the Namara Platform http://namara.io/#/
University of Babylon
w3.org
Wikidata
