Blog
Bits and pieces from the Crowd Scientists
Continuing Debate Over the Problems with Optin Surveys
Some really interesting research out of Stanford just landed in my RSS reader. This blog examines - and summarily refutes - challenges to a study finding that non-probability Internet sampling delivers poorer quality data than probability sampling conducted on the Internet or telephone (RDD).
The study, whose roots began in 2004/5, with further investigation in 2009, shows that non-probability Internet samples, which would refer to opt-in samples such as those from Internet panels, yield results of significantly lower accuracy than RDD or probability Internet samples. This holds true even when weighting of the data from opt-in samples was conducted in an attempt to make the sample more representative.
The current blog addresses various questions regarding the currency, relevance, and validity of these findings – with the answers to these questions unwavering in their support of the original findings. For example:
- The initial findings from 2004/5 still hold true in 2009, in spite of any potential changes in either Internet or telephone sampling methodologies.
- The differences between the probability and non-probability samples were significant.
- Findings from this investigation do not rely on cost-prohibitive or specialized methodologies.
- Differences in results between probability and opt-in samples were not due to a lack of proper balancing or weighting. In fact, while no balancing was applied to the two probability samples, balancing and weighting were applied to the non-probability samples but did not eliminate the error typically found in opt-in sample surveys.
We're (obviously) big cheerleaders for probability Internet sampling here at Crowd Science (it's at the core of our Research Platform), so it's good to see some further evidence from the academic community.
Check out the blog here: The Numbers
Check out the original research here: Comparing the Accuracy of RDD Telephone Surveys and Internet Surveys Conducted with Probability and Non-Probability Samples