This lab is turning the way social science research is conducted on its head, and that’s not just because the whole thing is online
Those of us outside the field might be horrified to learn that up until now, the typical subjects of social science lab experiments are grad students. Would you believe that, as such, they are not entirely representative as a cross section of humanity? The real surprise is that in order to find a solution to this, you might have to go to a place where you might (wrongly?) imagine everyone was even more geeky.
Here’s an abstract from the paper that the talk was based on:
The online laboratory: conducting experiments in a real labor market
Online labor markets have great potential as platforms for conducting experiments. They provide immediate access to a large and diverse subject pool, and allow researchers to control the experimental context.
Online experiments, we show, can be just as valid—both internally and externally—as laboratory and field experiments, while often requiring far less money and time to design and conduct. To demonstrate their value, we use an online labor market to replicate three classic experiments.
The first finds quantitative agreement between levels of cooperation in a prisoner’s dilemma played online and in the physical laboratory. The second shows consistent with behavior in the traditional laboratory—that online subjects respond to priming by altering their choices.
The third demonstrates that when an identical decision is framed differently, individuals reverse their choice, thus replicating a famed Tversky-Kahneman result. Then we conduct a field experiment showing that workers have upward-sloping labor supply curves.
Finally, we analyze the challenges to online experiments, proposing methods to cope with the unique threats to validity in an online setting, and examining the conceptual issues surrounding the external validity of online results. We conclude by presenting our views on the potential role that online experiments can play within the social sciences, and then recommend software development priorities and best practices.
Dave Rand is a Cooperation Fellow at the Berkman Center for Internet & Society, as well as a Harvard’s Program for Evolutionary Dynamics and a FQEB Prize Fellow in the Psychology Department. Dave’s work focuses on the evolution of human behavior, with a particular emphasis on cooperation, generosity and altruism. His approach combines (i) empirical observations from behavioral experiments with (ii) predictions generated by evolutionary game theoretic math models and computer simulations. David’s research has been published in Science, Nature and PNAS, and featured on NPR’s All Things Considered and Earth & Sky as well as in a range of print media.
The talk was given at a Berkman Luncheon on November the 2nd, 2010. Philosopher and blogging legend David Weinberger was at this talk and liveblogged it here.