Are key indicators a necessary evil?: Quantification in tke kingdom of God

Note: this piece discusses experiences related to my service as a proselytizing missionary of the Church of Jesus Christ of Latter-Day Saints. I served a mission in Frankfurt, Germany in 2010-2011.

Image may contain: 1 person
Obligatory photo of me from my mission. Germans have a sense of humor.

One of the most uncomfortable aspects of serving a mission for me was counting up our numbers at the end of the day and reporting them to my district leader. Every week, missionaries set numerical goals they would like to achieve. In my day, missionaries recorded:

Number of investigators with a baptismal date.
Number of investigators baptized.
Number of investigators at church.
Number of progressing investigators.
Lessons taught with a member present.
Other lessons taught.
Lessons taught to less active and new members.
Number of referrals contacted

These have since been changed to:

People baptized and confirmed.
People with a baptismal date.
People who attended sacrament meeting.
New people being taught.

The missionary manual Preach my Gospel states why we use key indicators to measure results:

Your purpose is to invite others to come unto Christ by helping them receive the restored gospel through faith in Jesus Christ and His Atonement, repentance, baptism, receiving the gift of the Holy Ghost, and enduring to the end. Key indicators have been identified by Church leaders to help you focus on this purpose. As you focus on these indicators, you will help people progress toward baptism, confirmation, continued activity in the Church, and lasting conversion.

It also contains the familiar adage attributed to Thomas S. Monson:

When performance is measured, performance improves. When performance is measured and reported, the rate of improvement accelerates.

I struggled with this because it seemed to change priorities in ways that seemed counter to the good of people I was serving, and also made me feel bad on what I personally felt were my most productive days. For instance, there is no incentive in these key indicators for service. The white handbook encourages missionaries to serve once a week for ~ four hours. Since service hours don’t count for anything in terms of key indicators, these were more often than not scrapped in my mission. Other times, I would have powerful spiritual experiences with members, or good conversations with strangers that didn’t necessarily lead to a follow-up appointments and end the day with no numbers to show.

Additionally, I felt that it could often lead to a form of “gaming the system.” What counts as a lesson? In my mission, a lesson would count as a lesson if it included a prayer and a gospel principle. I had some companions who got more creative than others in what filled this criteria, and I’ve seen even more creative missionaries here in my mission who will ask to pray with people on their doorstop even if they weren’t invited in. I don’t see any harm in that: in fact, I was impressed with the chance to have a connecting positive experience with someone who had just rejected us. But still: the numbers motivate ways to find “what counts.” It reminds me of Elder Holland’s counsel on home teaching reports:

Now, as for what “counts” as home teaching, every good thing you do “counts,” so report it all! Indeed, the report that matters most is how you have blessed and cared for those within your stewardship, which has virtually nothing to do with a specific calendar or a particular location. What matters is that you love your people and are fulfilling the commandment “to watch over the church always.”

Fortunately, I didn’t see any horrible results occurring from such number games: only a lot of moral agonizing over whether we had 0 lessons or 3 depending on which definition you used. But historically, there have been examples where something does go wrong. I remember this example from “baseball baptisms” that Gregory Prince talks about in David O. McKay and the Rise of Modern Mormonism. Missionaries would take advantage of Church youth baseball teams to teach young kids:

Church policy from the earliest days forbids the baptisms of children under the age of eight, when they are considered to be able to make informed moral judgment. No evidence exists in McKay’s papers that children under age eight were baptized; but the baptisms of minor children age eight and over with parental permission soon deteriorated into the baptisms of children without parental permission. Ultimately children were baptized without even knowing what was happening to them. A British member who was baptized prior to the baseball baptism era described an all-too-common scene:

One Wednesday we went to MIA, we walked in, and there were about fifty or sixty boys and girls there. It was just a shock to us at that particular time. There were boys and girls everywhere…. Half of them didn’t know that they were baptized. I know that is true, because we were asked to go home teaching to these boys and girls a year or two later, and I remember going to see one young lady…. I said, “Don’t you ever remember being baptized? Don’t you ever remember dressing in white and being put in the water?” “No.” “Can’t you remember anything?” She said, “Just a moment. Tea and coffee, Americans!” I said, “That’s right. Do you remember now two young American men who didn’t drink tea and coffee?” She said, “I do.” And that was all she could remember of her baptism.

Because missionaries had such pressure to up their baptism numbers, they had to get creative. I didn’t see anything of this level, but I did see investigators who were interested in the gospel get dropped because they didn’t feel ready to be baptized right away, and missionaries who rushed investigators to get baptized who weren’t ready. When numbers take precedence over the lives of individuals, real harm can occur.

An example from ambulances

I recently read a book called Messy: The Power of Disorder to Transform Our Lives. It had an interesting example– not related to missionary work at all– about how targets change behavior. I immediately thought back to the dilemma of key indicators as a missionary while reading this passage. This is the meat of what I wanted to present for your consideration, so pardon the long quote and my lack of original thought– I just wanted to tie these two things together:

Sometimes a target reflects yesterday’s problems, not today’s. The world tends to change faster than bureaucracies can keep up, which causes problems for any organization that has lashed itself to an unbending framework of performance measures. Imagine a company with targets for resolving queries from customers who phone the call center, but no target for dealing with a problem using the Web. The target demands that resources should be funneled to the call center, when they might be better used on the website. Anyone who wants to improve will have to ignore the target.

There are also two ways to sidestep a target completely: lying and cheating. It’s sometimes possible to lie— miss your target, but say you hit it— and that sounds bad enough. But cheating, or “gaming”— where you cynically distort your behavior to hit the target— may be worse…

Connoisseurs of unintended consequences may appreciate a particular episode in the history of the British National Health Service, where a clumsy target managed to trigger almost every single unintended consequence on Peter Smith’s list.

The target was first set in the 1990s, then given much more political emphasis in the early twenty-first century. It was designed to improve how ambulance services responded to emergency calls. When an emergency call was made from an urban area, and the case was judged to be “immediately life-threatening,” the target became active: the ambulance service had eight minutes to get to the scene.

The unintended consequences soon began to emerge.

The most obvious was the outright lie. This lie becomes obvious if one plots the data for reported response times on a graph. The graph rumbles gently along, showing an even distribution of response times. And then something curious. The graph starts to rise sharply at 7 minutes and 50 seconds, reaching implausibly high levels at 7 minutes and 59 seconds, and soaring further at 8 minutes exactly. Yet by an astonishing coincidence, hardly any calls were recorded as having taken 8 minutes and 1 second. After the fact, who can gainsay the call handler who says the ambulance hit the target rather than missing it by a second or two?

Such lies probably don’t matter too much: they will not be the difference between life and death, because they don’t affect what ambulance crews actually do. But other responses to the target were not so benign— these were the ones that involved “gaming,” changing the behavior of the ambulance service in ways that made the target easier to hit but may well have harmed patients.

One problem was that “within eight minutes” turns out to be a poor substitute for “as soon as possible.” Imagine an ambulance on route to a patient as the clock ticks away. Six minutes pass; then seven. Then eight minutes pass and the case has missed the target—“ breached,” as hard-bitten professionals would put it. Now what is to be done? The logic of the target dictates that the ambulance should now be rerouted to a patient who has not yet breached; the original patient no longer counts toward the target. He may find himself waiting indefinitely. The best defense against such grotesque abuse is the hope that professional ambulance crews would ignore the target— but that does not say much for the target’s merits.

Another serious piece of gaming involved the definition of “life-threatening” cases. The definition varied widely across regional ambulance services and could easily be manipulated. The target gave ambulance services an incentive to say that a call wasn’t urgent— because at that point they were spared from hitting the goal.

Ambulance services even changed their vehicles to flatter the target. Two paramedics in a single ambulance could be split and put onto motorbikes or even bicycles. That makes it more likely that a paramedic will arrive within eight minutes, but if the patient needs to be taken to the hospital, then a bicycle will not do the job.

There were also accusations of short-termism. Ambulance crews complained that rather than being based at the hospital, they were being sent out in their vehicles to face an uncomfortable wait through the night at strategic positions from where they might be able to reach an emergency more quickly. Patient care in the short term may have improved, but the crews’ morale suffered.

Finally, an allegation that was widely suspected but never proved was that managers relocated ambulances to urban areas, where the targets were tough to hit, from rural areas, where the targets operated differently. The target wasn’t designed to favor the cities over the countryside, but it may have had that effect.

The target, then, encouraged ambulance services to lie about the stopwatch, to reclassify urgent cases, to put ambulance crews in vehicles that weren’t ambulances, to pull staff out of rural areas, to risk the health and morale of their paramedics, and to celebrate all of this as having hit their target.

For one simple, tidy target, that is quite a result.

Perhaps it comes as no surprise: of course targets change behavior: that’s what we want them to do! They provide motivation to improve, and so what if along the way some unethical behavior occurs? We can’t control for that: we have to rely on the ethics of individuals. The system can’t be held accountable for that. When I mentioned my frustrations to my mission president in an interview, he sympathized, with only the answer that “numbers are a necessary evil.”

I don’t doubt that; I’m an engineer: I’m all about numbers, constraints, measurements, cost functions. The more things you can quantify the better. I just attended a seminar where they presented a new method whereby scientists have reduced chemical structures to a system of vectors thereby making them differentiable and optimizable: another illustration of how powerful quantification can be. But especially in terms of people, you often lose more than you gain when you reduce it to numbers.

A solution?

I don’t want this to end on a sentimental note: people can’t be reduced to numbers. It’s almost cliche. But Tim Harford offers an intriguing solution to stop attempts to “game the system”: and, ironically enough, if just involves more numbers. Here’s the deal:

The answer is neither the weighty rule book of Basel nor one simple rule of thumb. Instead, we should be defining many rules of thumb, and deliberately leaving it ambiguous as to which will be used in any given situation. With ambulances, for example, you might keep “percentage of calls answered in 8 minutes” as one possible metric to check, and add some more: percentage of calls answered in 12 minutes, or 20 minutes, or 6 minutes, 37 seconds; proportion of patients who die having not been classified as “immediately life-threatening”; consistency of service across rural and urban areas; and so on. You could define hundreds of rules of thumb like this. It would be impossible to game them all, because gaming any one would reduce performance on others. But an ambulance service that was generally well run should be doing well on most of them.

It would be foolish to try to check every one of these rules of thumb, because a regulator’s inspection would end up being both impossibly bureaucratic and absurdly superficial. There’s another risk: if ambulance services knew how all the many targets were weighted to produce an overall score, they would find a way to game that, too. Instead, for each assessment, the regulator would choose a few measures at random and examine them in depth.

Could such an approach be applied to missionary work or other church programs? Am I just overly optimistic? This is too much work now? Perhaps. But even then, I think the church already has provided missionaries with one tool that, while not used for general assessment, does help missionaries perform a self=evaluation. Preach My Gospel includes a section on what makes a successful missionary. And it lists the following:

  • Feel the Spirit testify to people through you.
  • Love people and desire their salvation.
  • Obey with exactness
  • Live so that people can receive and know how to follow the Spirit, who will show you where to go, what to do, and what to say.
  • Develop Christlike attributes.
  • Work effectively every day, do your very best to bring souls to Christ, and seek earnestly to learn and improve.
  • Help establish and strengthen the Church wherever you are assigned to work.
  • Warn people of the consequences of sin. Invite them to make and keep commitments.
  • Teach and serve other missionaries.
  • Go about doing good and serving people at every opportunity, whether or not they accept your message.

Whenever I read this list, I would feel much more at peace with myself than if I focused on the key indicators. I could work on these, whereas the key indicators made me feel stressed and out of control. Perhaps using these as key indicators wouldn’t be beneficial from an institutional standpoint. But maybe we can implement some variation of the ambulance example? I would love it if there were some ways that quantified relationships and work with members in the ward, service in the community (which I think is good and vital!), work with members after baptism, work with youth in the ward preparing for missions, etc. Thoughts?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Blog at WordPress.com.

Up ↑

%d bloggers like this: