13 Oct The problems—and benefits—of social science research
I hate it when I get it wrong. I hate it even worse when I get caught getting it wrong. Of course I have enough experience in both areas that it should be old hat, but I still have to address it.
Of all the columns I’ve written for the Standard Journal the one that gleaned the most personal responses in the community was “The problems of having a hot wife”. We’ll see if readers are anywhere near as passionate on the topic of social science research.
I was taken to task on these pages on several points, and rightfully so. In my efforts to make the article brief and at least mildly interesting I failed to adequately address several challenges common to designing, conducting, analyzing and reporting of social science research. Of course I didn’t claim to be conducting research but it gives us a chance to discuss the importance of understanding these as we make public and personal decisions.
“Sampling”: the number and source of the individuals questioned or studied before coming to a conclusion. Large samples are usually more likely to give us an idea about the group we hope to understand than a very small sample. Anomalies or odd-balls are less likely to “skew” or distort the findings if just three of 3500 instead of three of 35.
The source of the sample also makes a big difference. “Convenience sampling” is a term used to describe studies in which the individuals studied are drawn from a convenient source, as opposed to “random samples” in which the individuals are randomly chosen from the general population.
Both sample errors beset the data presented to the U.S. Supreme Court when they made their two law- and life-changing decisions to change the definition of marriage. The majority of the studies presented by the American Psychological Association claiming no benefits to children being reared in the homes of opposite-sex parents had small numbers (as small as nine) and convenience samples (like volunteers recruited via gay and lesbian publications).
The article that set the immunizations-autism debate into swing had similar problems. The article was written by a doctor who “studied” 8 children whose parents reported the child demonstrated autistic symptoms after receiving routine immunizations in Great Britain. It was later reported that Dr. Wakefield was being paid by the parents of those children to provide evidence in a large law suit. The small and convenient sample problems were unknown by millions of parents who made life-changing decisions regarding the health of their children based on that article.
That leads to our next research challenge: “Correlation is not causation.” Just because two variables coexist does not mean that one caused the other.
It has been observed, for example, that incidence of ice cream consumption coincides with incidence of violence in some U.S. communities. One might conclude that Ben and Jerry are responsible for much of America’s violence, but it would probably be safer to note that both factors—violence and eating cold treats—are more common when weather is hot and days are long.
Some research similarly suggests that autism symptoms commonly occur at about the same age at which the MMR (mumps, measles, rubella) vaccine is administered—irrespective of whether those children actually received the vaccine.
Now please don’t write and complain that I am not an expert in autism; I readily admit that I am not, and simply use this as an example of how even well-intended research and observation sometimes lead to mistaken conclusions and significant decisions.
We social scientists sometimes complain that those who study the physical world get off easy. They can melt and measure; twist and test; explode, implode and probe the matter they study. But human subjects don’t take kindly to such treatment. Can we really experiment on what types of families are best by randomly assigning children from the neonatal unit to different families in the communities? Not so much.
Occasionally we get a rare peak at something sort of like that, as when one identical twin is mistakenly sent home with the wrong parents. Years later we might see differences between how those children think or act and assume it relates to the different homes.
But even then there are many different factors that influence the outcomes for human beings. So it becomes very difficult to reach reliable conclusions unless we are able to isolate—narrow down and account for—the variables we are trying to study.
But even if the research is designed and conducted carefully and effectively, reporting of findings is still a problem. It is very difficult for one to publish articles in scientific journals that run contrary to popular opinion. (PhD’s still like to be among the cool kids.) Popular media is more likely to highlight provocative and titillating (read: odd, unusual or unorthodox) trends and behavior. “If it bleeds it leads”, and if it smells it sells.
So if social science and family research is so darn tough, why even conduct, print or read any of it? Because solid, well-designed and –conducted research can help us make wise decisions in our own lives and the lives around us.
Perhaps more importantly, the truth is that we are all social scientists in our own right, coming to conclusions in response to what we see. We would be wise to carefully consider how we observe, organize, assume, talk and choose, because we don’t just live in this world—we create it.