QA4D

by David Week on 20 November 2011

Share

Okay: we don’t really need yet another of those cutesie “4D = for development” labels, but for a blog post heading, it’s acceptable.

In my last post, Fad Surfing in the Development Boardroom, I took issue with those (and there are many) who think that “development” is completely dissociated from the rest of the world. There’s no good reason to:

  • assume the whole of the development industry is a reflection of the lack of nous demonstrated in some (unnamed) projects
  • inventing new “solutions” from scratch
  • failing to draw upon and improving the body of knowledge which millions of person-hours of development and testing have already been invested—both within, and outside the industry.

Learning and adapting

Don—one of the commenters on that previous post—queried me as follows:

I’m curious if you can take your points to their natural conclusion and provide a more convincing argument. If QA, Six Sigma, etc. are the solution you make them out to be, can you provide examples of them working in the context of development work? By stating that an idea could work without providing evidence for it, aren’t you just providing ‘another potential solution’ instead of ‘the solution’…I’m stuck at this point in your argument because although I understand the benefits of quality assurance practices in the private sector, context is everything in development work, and great ideas have failed and failed again specifically when the context of a situation wasn’t taken into account.

I agree with both points, and this purpose of this post is to talk about some of the lessons learned one might learn from various QA movements, and how these have already or might be  applied to development, ensure that learning improves and the field gets better.

Don is right: we can’t blindly import from other industries. At the same time, we can’t just ignore what has been learned in those industries.

Outputs and variances

The late great W. Edwards Deming emphasised the measurement of variance as being key to quality. What that means, in practice, is that in order to have quality, you have to know:

  • what you are trying to produce, and
  • what constitutes acceptable quality.

Then, whenever you get variance, you track it back to its source.

In development, it’s much harder to specify what we’re trying to produce, and what constitutes acceptable change, than in the production of physical widgets. But it is not impossible, and it is being done.

There’s a standard model of the development process called “the results chain”. The results chain is:

  • inputs -> activities -> outputs -> outcomes -> impacts

Let’s look at a education project, with a school construction subcomponents. Inputs might be:

  • cash grants from the Government to school communities, financed by a donor
  • technical assistance from the Government, financed by a donor
  • a program design

Activities that these inputs are applied to then might be:

  • school community selection based on poverty indicators
  • school community negotiation and mobilisation
  • school planning
  • community training
  • community procurement
  • community construction

The outputs would be:

  • new, appropriately sized schools, built at a reasonable cost, so that no child needs to walk more than 20 minutes to attend primary school.

Outcomes are the intended change within the lives of the beneficiaries, as a result of the outputs. In this case, since school construction is under the “access” components of the project, outcomes can be measured in terms of attendance.

Impact is more difficult to measure, because impacts include both intended and unintended changes in the lives of the beneficiaries. Impacts are so complex and contentious that my next post will be entitled “No Impact”. But for this project, the impact of improved education access through grant-funded community school construction might be:

  • POSITIVES: better future livelihoods for children, increased community engagement in school management, increased ownership, increased capacity and competence, increased community cohesion and confidence through coming together to undertake a major project (and on and on…)
  • NEGATIVES: internal conflicts over resources, school construction conflicts with agricultural time needs, inequity in financial benefits, corrupt or non-transparent procurement, negative social effects of the community process fails (and on and on…)

Impacts are more complex because they are boundless. Let’s say we provide really good school education… and as a result, 20 years all the good students have decamped for the city, depriving the village of its leadership pool. This does happen.

How far along are we in quality assuring outputs? I haven’t seen a project in the last 20 years which isn’t clear on what it’s outputs are supposed to be. Mina, another commenter claimed: “For 99% of implementers and projects it’s usually associated with spending the allotted budget on the planned activities.” But I don’t know any agency these days (though I only deal with large ones) who would get away with this. Today, on the whole, aid is funded on outputs.

What about the quality of those outputs? There are two levels of quality:

  • The quality of the output itself, which is usually not that difficult to define
  • The quality of the outcome.

Measuring outcomes is a large topic in M&E. Two approaches I think worth knowing about:

The basic point here is that quality depends on understanding your production line, and what is coming off the end of the line, and what constitutes a quality product. Towards the end of the line, development activities get very complex. But that’s only because ultimately, they are all social change projects.

Ownership

The lessons I learned from Six Sigma were about the importance of process ownership. Whereas traditional QA seems aimed at Quality Management, Six Sigma highlight the critical role of the process owner. Who has the real stake in ensuring this process works?

And the biggest mistake I made fresh out of university was holding the notion that—because I was full of so many neat theories and ideas that I could, indeed needed to be the process owner myself. Correction: that wasn’t a mistake. It was just a stage in my professional development, and perhaps one we all go through.

Nowadays, the best rule of process ownership in development is:

  • ownership should never vest with the donor
  • ownership should never vest with the implementer
  • ownership should vest, from the beginning, with the beneficiary.

Implicit in this is that the development process has to be on the beneficiary’s agenda, and come from that agenda. It can’t be “sold” in by the donor. The strongest argument for this comes from the development economist A. O. Hirschman, who said simply that you should only invest in what people are already doing, and to invest in what they are not doing is just a form of bribery.

This whole line of thinking is currently encapsulated in the acronyms CDD (Community Driven Development) and Demand Driven Development (DDD). However, these have been criticized as not being sufficiently pro-poor. In practice, development seems to be always pulled back to the centre of three different perspectives:

  • the beneficiary’s demand perspective
  • the implementer’s feasibility perspective
  • the donor’s value for money perspective.

I think this is completely fine. Each perspective adds value, and you wouldn’t want a program designed from any single perspective. However, the implementer and donor may enter the negotiation, ultimately the beneficiary must be the first and final owner. At no point should ownership by any other party be assumed, or taken.

The kinds of project that most fully take the demand perspective (with feasability and value-for-money filters) are:

  • social funds: a pot of money, communities and NGOs can propose projects for funding
  • incentive funds: like a social fund, but geared to helping institutions who are already performing well
  • facilities: another pot of money, from which the government can propose projects.

Perhaps the most extreme form of demand driven program I have ever seen was that of CEDMA: The Centre for Development Madras, an NGO based in Tamil Nadu in India. Nelson Muthunayagam told me that CEDMA community organisers were forbidden to propose projects to communities. An organiser would live in a poor community sometimes for years, just talking, waiting for the community to decide it wanted to do something. Then, they would help. One CEDMA story went like this:

A poor community had settled on a disused railway track, near the centre of Chennai where they worked. The railway company decided to re-open the line, and offered them land far from the centre as compensation. They didn’t know what to do. CEDMA suggested that that they hire a lawyer. The community reps responded that lawyers only helped the rich. CEDMA explained that lawyers help whoever pays them. So the community raised a certain number of rupees from each family, hired their own lawyer, and took the railways to court. The community won, and the railways were forced to given them land nearby, and the costs of moving their dwellings.

The value stream

The value chain is a concept from Lean Manufacturing. The basic notion of lean is that every aspect of production has to be “pulled” by the customer. One has to understand what the customer wants. Then one has to “learn to see” how each step adds—or doesn’t add—to the value stream. All operations that don’t add value to the customer are “waste”, of which there are seven forms:

  • Overproduction
  • Waiting
  • Transporting
  • Inappropriate Processing
  • Unnecessary Inventory
  • Unnecessary / Excess Motion
  • Defects

Quality is achieved by removing waste.

In aid projects, waste often appears as follows:

  • Overproduction: producing a level quality greater than what the beneficiaries want; adding development values where they aren’t going to be used.
  • Waiting: for funds to pass through various level of bureaucracy, for projects to be approved, for projects to be mobilised
  • Transporting: unnecessary flying in and flying out of consultants
  • Inappropriate Processing: excessive reviewing before implementation
  • Unnecessary Inventory: too many projects designed but never implemented; delays during projects; delays between projects
  • Unnecessary / Excess Motion: excessive consultation and negotiation without movement
  • Defects: usually due to the wrong party doing the job
None of these are what we normally consider “waste”. But consider lean definition in the case of a gap of year between Village Water Supply Project No. 1 and Village Water Supply Project No. 2. This gap appears to be necessary while No. 1 is evaluated, lessons absorbed, negotiations held, and No. 2 designed and mobilised. During this period there is no expansion of water to the beneficiaries, and project staff with key knowledge wander off to other projects. From the point of view the “customer”, the people that need the water, that gap is a huge waste. It would be far better to keep No. 1 rolling, with gradual improvements based on ongoing evaluation.

Key points

This post could go on indefinitely, because quality improvement, and thinking about quality goes on indefinitely. On reflection, what’s important:

  • Managing quality and managing a development process are inseparable, in any modern sense of the word “manage”. Organisations that are not managing for quality, are not managing at all.
  • Aid is not some silo that operates beyond the realm of the rest of society, but is (or should be) continuously exchange ideas with other disciplines and endeavours in both donor and beneficiary societies. We should not treat it as a sandbox for novelty.
  • Any solutions to variances in aid have been, and will be, built on tools borrowed from those other realms, adapted to the development context.
  • There are dozens, if not hundreds, of major innovations in process which have been developed and implemented over the last 20 years specifically aimed at improving the quality of aid, in light of the variances of the past.
  • No-one goes to work every day trying to do a bad job. Do not blame the individual worker. Quality problems are rarely the result of an individual. They are systemic. So too, therefore, are the solutions. To understand quality problems, you need to understand the aid system, and work with many others to change the system.
  • Trying to respond to problem, without understanding the system, makes matter worse:
That’s all window dressing. That’s not fundamental. That’s not getting at change and the transformation that must take place. Sure we have to solve problems. Certainly stamp out the fire. Stamp out the fire and get nowhere. Stamp out the fires puts us back to where we were in the first place. Taking action on the basis of results without theory of knowledge, without theory of variation, without knowledge about a system. Anything goes wrong, do something about it, overreacting; acting without knowledge, the effect is to make things worse.
— W. Edwards Deming (1991)
  • MJ

    Hi David, Another very well reasoned post. I would suggest one counterpoint, however. And that is development that is 100% owned by the beneficiaries will be substantially constrained by what the beneficiaries can conceive of. Indeed your CEDMA example actually highlights this fact: the community needed the suggestion from the outsider in order to find the best solution to their problem. I am reminded of the famous quote popularly attributed to Henry Ford that if he’d “asked people what they wanted, they would have asked for a better horse”. A bit of experimental jet-pack development every now and then can also be a good thing, but, to get back to an earlier topic, we should be prepared to fail fast if it doesn’t gain traction.

Previous post:

Next post: