One of the most important years in the history of big data was 1997, the year that Deep Blue beat Gary Kasparov at chess. At first blush, this might not seem like a big data challenge; chess after all has only 64 spaces, 32 pieces, 6 different types of pieces, and only two players. However, when chess is analyzed more deeply, its true complexity emerges. Claude Elwood Shannon, the so-called father of information theory, showed that the number of legal configurations a chess board could realize is approximately 1043. This is obviously an enormous number and is sometimes referred to as the Shannon number. A study* by the University of Southern California in early 2011 estimated the world’s total digital storage to be on the order of 1021. In this light, chess is clearly huge when considered against the scale of the digital world. Even beyond that, a Dutch computer scientist, Louis Victor Allis, estimated the game-tree of complexity of chess to be approximately 10123. That number is roughly 40 orders of magnitude greater than the estimated number of atoms in the entire universe. The act of computationally playing chess is clearly a “big data” problem, and 1997 showed us that computers can do this better than humans can.
The next important year in this story is 2005. In that year, Gary Kasparov decided to host his own chess tournament. In light of Deep Blue, Kasparov become extremely interested in the capabilities of computational systems but also in the ways that computers and humans approach problem solving. Kasparov’s 2005 chess tournament was a free-style tournament in which teams could be composed of any combinations of humans and computers available. Grandmasters, prodigies, and chess supercomputers could team up to form super teams. By 2005, it had already been shown that a chess master teamed with a chess supercomputer was far more capable than a supercomputer alone. Computers and humans have different and complementary analytic strengths: computers don’t make mistakes, they are highly precise, while humans can use intuition and lateral thinking. These skills can be combined to build truly formidable chess opponents. However, 2005 was different. The winning team, ZackS, performed so well many thought it was actually Kasparov’s team. However, the truth was much more intriguing. It turns out that ZackS was actually two amateur chess players running open source chess engines on simple off-the-shelf laptops—no grandmasters, no prodigies, no chess supercomputers.
This was a remarkable outcome that surprised everyone, including Kasparov himself. Kasparov drew the only conclusion he could: “Weak human + machine + better process was superior to a strong computer alone and, more remarkably, superior to a strong human + machine + inferior process.” This revelation points to the essential evolution of the conclusion from Deep Blue in 1997—that humans working together with machines can solve big data challenges better than computers alone. Tackling big data means more than just algorithms, high-performance computing, and massive storage—it means leveraging the abilities of the human mind.
* See http://www.computerworld.com/s/article/9209158/Scientists_calculate_total_data_stored_to_date_295_exabytes. See also ZackS - http://chessbase.com/newsdetail.asp?newsid=2461; “Friction in Human-Computer Symbiosis: Kasparov on Chess” at http://blog.palantir.com/2010/03/08/friction-in-human-computer-symbiosis-kasparov-on-chess/; and “A Rigorous Friction Model for Human-Computer Symbiosis” at http://blog.palantir.com/2010/06/02/a-rigorous-friction-model-for-human-computersymbiosis/.