While I'm On The Subject...

Just got back from a little jaunt to the Teleologic Blog, and saw something that I wanted to respond to here, since it hints at a more general issue.

(Fair Warning: statistics is going to be mentioned again....ok, now, for those of you haven't rightly moved on to Kevin's more gripping posts...)

Rakhiir makes mention of this article, describing the results of a large study that evaluated a comparison between the effects of "talk therapy" (Freudian Cognitive Behavior Therapy) and Prozac, one of the current supposed wonder-drugs many parents use to counter-act the effects of leaving the kids in front TV or not disturbing their death-grips on video game controllers...oops, did I let my bias out? (Please note, I really do believe there are good reasons to proscribe Prozac and a number of other emotional-problem oriented drugs. But certainly the problem either correlates with something about modern parenting to demand such a spike in medicated children, or the drive to get the stuff by parents is causing any number of misdiagnoses or forced prescriptions.)

The study apparently indicates that there is no difference between the effect of talk therapy and a placebo drug. (can anyone find a link? I got tired of looking as a way to put off studying for my econometrics final tomorrow...) That is to say, the effects of both the therapy and the placebo were both not statistically different from zero. Interesting in its own right, and might require a review of the structure of the program. But that's not my main concern here. Rakhiir's reaction is:

One professor of psychiatry was quoted today as saying "It was very close to a significant effect". The more honest way of saying this is that the effect was statistically non-existant! The professor, Dr. Thase went on to say that good psychotherapies sometimes did not work in big studies. Talk about a capacity for self-delusion. The best you can say for talk therapy is that it doesn't actively hurt its subjects - its not worse than the placebo. By contrast, Prozac really does work and helped 75% of the patients.

(Emphasis in the original.)

Not so fast there, sparky. There's a big difference between not being statistically different from zero and being "non-existant." This mistaken view is a problem that arises in a lot of evaluations of programs and is worth noting. Without bogging down in the numbers, the idea of being statistically similar to zero is this: the value estimated for the effect of the treatment isn't -- because of measurment problems, calculation issues, and more -- a single point value. It's actually the middle of a range of values, among which difference can't really be determined. It's called the "confidence interval" for the estimated value. (Apologies to those who sat through that years ago in undergrad stats -- my hope is to appeal to a broad audience, and, frankly, I'm not that smart, so I like simple definitions and those fun "scare quotes.") Which means, of course, that being statistically equivalent to zero, the estimated value for the effect of the treatment on the treated includes zero. But it also includes a number larger than the point-estimate value possibly reported by the study. In other words, the effect could well be greater than even the report says. A lack of precision, however, keeps the researchers from saying "Hey, the effect is actually really huge!!! We think..."

The program may well be effective, and may well hold some benefit for those engaged in it. A lack of precision in the estimation might not be a good reason to toss something out the window since precision is often out of the hands of the researcher (it's not just that they decided to be lazy about rounding or something).

And the claim that the value was close to being significant? Is that just charlatanism running rampant? Not really. Every estimate is going to have some factor that determines the size of the confidence interval; a significance level. These are chosen, for good reasons, by the researcher. Being "close to significant" could indicate that if the researcher chose a more generous significance level, the effect might have been read as "significantly different from zero", in which case talk therapy would suddely be proven as effective in the study! (The magic of numbers!)

The overall point here is, when we all read reports about how this, that, or the other program is clearly useless because a study said the effect was "statistically insignificant", we should dig a bit deeper to see what they're talking about.


Powered by Movable Type 5.02

About this Entry

This page contains a single entry by published on June 8, 2004 8:44 PM.

Thamir Ghadbhan & Iraqi Oil was the previous entry in this blog.

American Fructose vs. Mexican Sugar is the next entry in this blog.

Find recent content on the main index or look in the archives to find all content.