The Good Judgment Project (GJP) is a project "harnessing the wisdom of the crowd to forecast world events". It was co-created by Philip E. Tetlock (author of Superforecasting and of Expert Political Judgment: How Good Is It? How Can We Know?), decision scientist Barbara Mellers, and Don Moore. It was a participant in the Aggregative Contingent Estimation (ACE) program of the Intelligence Advanced Research Projects Activity (IARPA) in the United States. Predictions are scored using Brier scores. The top forecasters in GJP are "reportedly 30% better than intelligence officers with access to actual classified information."
Contents
History
The Good Judgment Project began in July 2011 in collaboration with IARPA-ACE. The first contest began in September 2011.
GJP was one of many entrants in the IARPA-ACE tournament, and has repeatedly emerged as the winner in the tournament.
Starting with the summer of 2013, GJP contestants had access to the Integrated Conflict Early Warning System.
People
The co-leaders of the GJP include Philip Tetlock, Barbara Mellers and Don Moore. The website lists a total of about 30 team members, including the co-leaders as well as David Budescu, Lyle Ungar, Jonathan Baron, and prediction-markets entrepreneur Emile Servan-Schreiber. The advisory board included Daniel Kahneman, Robert Jervis, J. Scott Armstrong, Michael Mauboussin, Carl Spetzler and Justin Wolfers. The study employed several thousand people as volunteer forecasters. Using personality-trait tests, training methods and strategies the researchers at GJP were able to select forecasting participants with less cognitive bias than the average person; as the forecasting contest continued the researchers were able to further down select these individuals in groups of so-called superforecasters. The last season of the GJP enlisted a total of 260 superforecasters.
Research
A significant amount of research has been conducted based on the Good Judgment Project by the people involved with it. The results show that harnessing a blend of statistics, psychology, training and various levels of interaction between individual forecasters, consistently produced the best forecast for several years in a row.
Good Judgment Open
A commercial spin-off of the Good Judgment Project started to operate on the web in July 2015 under the name Good Judgment Inc. Starting in September 2015, Good Judgment Inc has been running a public forecasting tournament at the Good Judgment Open site. Like the Good Judgment Project, Good Judgment Open has questions about geopolitical and financial events, although it also has questions about US politics, entertainment, and sports.
Media coverage
GJP has repeatedly been discussed in The Economist. GJP has also been covered in the New York Times, the Washington Post, and Co.Exist. NPR aired a segment on The Good Judgment Project by the title So You Think You're Smarter Than A CIA Agent, on April 2, 2014. The Financial Times published an article on the GJP on September 5, 2014. The Washingtonian published an article that mentioned the GJP on January 8, 2015. The BBC and the Washington Post published articles on the GJP respectively on January 20, 21 and 29, 2015.
The Almanac of Menlo Park published a story on the GJP on January 29, 2015. An article on the GJP appeared on the portal of the Philadelphia Inquirer Philly.com on February 4, 2015. The book Wiser: Getting Beyond Groupthink to Make Groups Smarter has a section detailing the involvement of the GJP in the tournament run by IARPA. Psychology Today published online a short article summarizing the paper by Mellers et al. that wraps up the main findings of the GJP.
The project spawned in 2015 a book by Tetlock, one of the principal investigator, and coauthored by Dan Gardner, that divulges the main findings of the research conducted with the data from the GJP: Superforecasting - The Art and Science of Prediction. Co-author Gardner had already published a book in 2010, that quoted previous research by Tetlock that seeded the GJP effort. A book review in the Economist September 26, 2015, print edition, discusses the main concepts. A Wall Street Journal article depicts it as: "The most important book on decision making since Daniel Kahneman’s Thinking, Fast and Slow. The Harvard Business Review paired it to the book The Power of Mathematical Thinking by Jordan Ellenberg. On September 30, 2015, National Public Radio aired the Colin McEnroe Show centering on the GJP and the book Superforecasting; guests of the show were prof. Philip Tetlock, the director of IARPA, Jason Matheny, and superforecaster Elaine Rich.