Who guards the RCT guardians?

by David Week on 27 May 2011

Share

I just today read two good posts by @edwardrcarr, outlining a critique of RCTs based on the qual vs quant distinction: The Qualitative Research Challenge to RCT4D: Part 1 and Part 2.

I agree with Ed’s points.

My beefs are different. The main ones are:

  • RCTs may well measure “if something worked”. But the tacit assumption behind funding them is that they are generating knowledge that can be transferred reliably from one context and time to another. This assumption is not being tested.
  • RCTs transfer a methodology developed for extremely unintelligent populations (such as cells responding to drugs), to very intelligent populations, with culture, forethought, learning, and language. This transfer is a massive leap of faith, and the leap has been glossed over in the making.
  • RCT proponents (and the formal evaluation community as a whole) fail to eat their own dog food.

To expand on that last point:

RCTs hang their legitimacy on a larger marketing spiel that investment in development should be “evidence-based”. This is a motherhood statement. Who can be against evidence? It sounds like the alternative would be to pluck investment rationales out of the air, or base them on personal idiosyncracies. The real question is whether RCTs are a good form of evidence upon which to base investment decisions.

Let’s ask this: what evidence is there that RCTs drive good development? This is not a hard question to study. The whole concept of “development” is based on the fact that there dozens of countries around the world that have indeed “developed.”

Let’s look at the United States, one of the wealthier countries in the world. Is it’s wealth being driven by RCT-guided investments? Look at the fabulous technological and social innovations that underpin that wealth. Are these being driven by RCTs? Are there any RCTs being done in Silicon Valley? Do venture capitalists commission RCTs before placing investments?

I don’t have hard data on this, but I suspect that the answer is “no.”

Let’s look at the flip side of this line of questioning: where RCT research has been conducted over a long period of time, do we see resounding success? Since RCTers’ main clients are government, this is the same as asking the question: does RCT-driven government programming work? Does the United States have effective poverty alleviation within its own borders? Does it have the healthiest poor? The best educated? Does it have the cleanest water in the world? (Ironically, the water seems to be polluted by those same RCT-tested drugs, an impact that the RCTs didn’t pick up on.)

The answer, to me, is not immediately obvious. Maybe RCT-guided government programs are the most effective in the world, and the United States (our example) does indeed have the healthiest, best educated, and upwardly bound poor people in the world. But before we export the RCT methodology to poor people around the world, I’d want to see some hard evidence that RCTs have worked inside our own countries.

But the clarion call for the RCT craze is “what about what RCTs did for medicine?” Well, that’s the wrong question.

The developed countries have created demonstrably healthier populations. Life expectancy has doubled over the last 150 years or so, from about 40, to about 80. (If I get some comments on this post, I’ll dig up more precise data.) It is precisely this kind of development impact that we want to share with people in other, “lesser developed” countries.

However, in this doubling, medicine doesn’t get much credit. It’s public health that did the work. And improvements in public health were not guided by RCTs, but rather by scientific developments such as the germ theory of disease, technological developments like the microscope, and engineering developments which made centralised water supplies possible.

Furthermore, even within medicine, the major milestones were not the result of RCT guidance. No RCTs in the development (or testing) of sulfa drugs. Or anaesthesia. Or the early leaps forward in antisepsis.

Here’s my hypothesis (testable: though I haven’t tested it yet) on when and why RCTs were introduced in health. It’s when all the heavy lifting had been done, all the major improvements made, and further development was “at the edges”: improvement became marginal, we saw declining return on investment, and it was no longer obvious if there were any improvements. That’s when complex statistical methods become required to see “if something works.”

I am reminded of an old bit of folk knowledge. The Romans expressed it as a question:

Qui custodiet ipsos custodes

Who guards the guardians?

The RCT fans would like to set themselves up as the guardians of “what works” in development. Before we let that happen, we should ask to see rigorous, hard nosed, empirical evidence that RCTs themselves work to guide effective development investments.

My thesis is that RCTs are late-comers to that development process, in countries where development has succeeded; and that their role has been marginal at best. I’d say that on the face of it, the thesis looks pretty sound.

So if the RCTers want to sell themselves on the basis of the contrary… I want to see the evidence. We have hundreds of years of successful development history in the developed world. Let’s look through it. Where is the evidence that, historically, where development has taken place, it has been guided by RCTs?

And if it is not RCTs, but something else that has guided us effectively so far: then let’s use that.

 

  • Nice post. The thing that bothers me most about RCT’s is its implicit paternalism. “We are gonna study you, recommend interventions for you, and you will respond to our incentives and thus be happier and healthier.” To find out what needs doing, I’m a bigger fan of actually talking to the people, see what they think, and work with them to find solutions. You know, dialog. Most people know what they need, they are not stupid, they are just stuck. 

    • Thanks Jason. I restricted myself about my concerns about the validity of RCTs. But I have another set of ethical, political and practical concerns about academics in Washington attempting to determine the future of peasants in the Sahel.

      Re paternalism, I read recently Banerjee and Duflo’s Foreign Policy piece More Than 1 Billion People Are Hungry in the World, appropriately subtitled “But what if the expert are wrong?”

      It struck me that the interaction at the end, between the authors and one of their informants, might by summed up like this:

      Banerjee and Duflo: “Now you eat your dinner!”
      Oucha Mbarbk: “Aw. I wanna watch TV!”

      • I’m going through some of your posts today.

        I read that Banerjee and Duflo article, and that wasn’t what I got from that anecdote.  My take away was that people, no matter how poor, are not always motivated by what development people have indicated is the best thing for them.  They are also motivated by pleasure and look for moments to escape, as do we all.  

        In my research in Senegal, I found that the poorest of the poor would spend precious CFA’s on tea and cigarettes, and it lulled me, for a second, into thinking…”ah..well things can’t be that bad if people have this extra money to spend” or “What are you thinking?  You have children to feed tomorrow.”  But that is unfair…because who knows if there will be a tomorrow? People are living for today, and finding those moments of luxury are, perhaps, not luxury, but necessity.  We in the west are so ‘pull yourself up by your bootstraps protestant work ethic’ in our judgments.

        I’m going to try to problematize that a bit more in my dissertation…or maybe in future work.

      • Hi Stephanie. Hi think you bring up two important points: the degree to which we are making judgements (including scientific judgements) which are just projections of our own culture, and which may not be relevant in the very different contexts in which aid operates. This is worth a post in itself, and I’ll let you know when I’ve written it.

  • Pingback: How Matters /  RCTs and aid effectiveness: Much to be said()

  • Lee

    The history of advancements in public health and medicine is full of experiments. They may not have been modern RCTs, but they were still experiments. RCTs are just the best kind of experiment. 

    • Hi Lee. I would question that RCTs are “the best kind of experiment”, a number of reasons:
      • the most important advances in science, public health, and medicine have not been the result of RCTs;• RCTs do not have the high status in the hard sciences that they do in among some social scientists;

      • during the period covered by the notion that “RCTs are the best kind of experiment,” there have been many steps backwards in public health. Among these is the use of drugs later to be found ineffective by meta-studies; and the pollution of the water supply with drugs. Both of these backward steps can be understood as a possible result of the narrow, short-term focus of RCTs.

      • the arguments by which RCTs are put forward as “the best” are old-fashioned, Rationalist arguments: that by reason, they should be the best. But in science, Rationalism was supplanted by Empiricism, and with good reason: Nature does not play by Rationalist rules.

      Science is driven curiosity, skepticism, thinking, and experimentation. But you will not find any of the hard scientists obsessed with a particular experimental method, let along be calling some method “the gold standard”. Furthermore, you won’t find among them the idea that you make scientific progress by testing thousands of little independent if->then statements. 

      RCTism has to be subjected to more rigorous testing, including the empirical test: does a research industry which is RCT-centric actually produce verifiably reliable social progress in public health, or education… or does it not.

      • Anonymous

        Let me quote from the obituary of Dr. Paul Meier:

        (http://www.nytimes.com/2011/08/13/health/13meier.html?_r=3)
         
        In an e-mail, Mr. Peto said that Dr. Meier, “perhaps more
        than any other U.S. statistician, was the one who influenced U.S. drug regulatory agencies, and hence clinical researchers throughout the U.S. and other countries, to insist on the central importance of randomized evidence.”

        “That strategic decision half a century ago has already
        saved millions of lives,” Mr. Peto continued, “and those millions should be attributed to Paul.”

        Or try the Cochrane Collaboration, which is “internationally
        recognised as the benchmark for high quality information about the effectiveness of health care.” (http://www.cochrane.org/about-us), named after Archie Cochrane, who “stressed the importance of using evidence from randomised controlled trials (RCT’s) because these were likely to provide much more reliable information than other
        sources of evidence.” (http://www.cochrane.org/about-us/history/archie-cochrane).

Previous post:

Next post: