Success in the business world is usually easy to recognize: measure your expenses, compare them to your earnings and you know how your enterprise is doing. But how does one measure this in the non-profit and government sectors? How can we know whether a school, a university, a police department, a doctor, a hospital, an army or a charity is doing its job properly?
In recent decades, measurement tools for non-profit organizations have proliferated. Many people believe that they provide crucial information for judging success and failure. But Jerry Z. Muller’s well-researched and easy-to-read book, The Tyranny of Metrics (Princeton University Press), raises grave doubts about the value of these measurements.
Muller, who grew up in Niagara Falls, Ont., is a world-renowned professor of History at the Catholic University of America. (Full disclosure: Jerry and I have been friends for almost 50 years, since our days together in Canadian Young Judea.) He is the author of many erudite books, including The Mind and the Market, Adam Smith in His Time and Ours, and Capitalism and the Jews.
READ: TRUTH AND JUSTICE WERE LIFELONG PURSUITS FOR AUTHORS
Muller does not deny that measurement tells us something. But the problem is that “measurement becomes much less reliable the more its object is human activity, since the objects – people – are self-conscious and capable of reacting to the process of being measured.” In extreme situations, humans cheat when they know that they are being measured. But even when they don’t, Muller points out the many ways that people can, and do, game the system.
If the compensation given to surgeons depends on the survival rates of their patients, they have a strong incentive to mostly operate on patients with good prognoses. When such a system was put into effect in some U.S. hospitals, Muller relates that “as a result of this ‘case selection bias’ … sicker patients were simply not operated on.”
Hospitals sometimes receive government or private funding based on their “success” rate, which is determined by measuring the number of patients who have to be readmitted within 30 days of being discharged. To improve their statistics, hospitals may decide to treat as many returning patients as possible as out-patients (presumably regardless of whether this is in their best interest or not). Likewise, when the effectiveness of police departments’ ability to combat illegal drugs is judged based on their arrest rates, they often concentrate on low-level dealers who are easier to find and more numerous, rather than on finding the lead supplier (if they found that person, it might yield only one arrest).
Similarly, universities are often ranked based on their “seven-year graduation rates” – i.e., the percentage of students who were admitted in a given year and actually completed their degrees. But there is no universally agreed upon standard on how to rate a bachelor’s degree, so there’s an incentive for universities to lower their standards and graduate more students.
What about measuring how many graduates of a given university are gainfully employed three or five years after graduation? That measure would tell us something about the success of the university, if success is defined as providing jobs to graduates. But universities have traditionally not seen this as their mandate. No one could measure “the sort of life-long satisfaction that comes from an art history course that allows you thereafter to understand a work of art in its historical context; or a music course that trains you to listen for the theme and variations of a symphony … or an economics course that leaves you with an understanding of key economic institutions.” As a result of the current obsession with “return-on-investment” metrics, some universities are becoming more like trade schools, to the detriment of society at large.
Muller dedicates chapters to every type of non-profit, providing damning indictments of unwise ratings that are used for each of them. His concerns extend to the heart of such organizations. In most non-profits and government departments – charities, schools, hospitals, police forces, etc. – the strongest motivators for employees have traditionally been intrinsic, that is, the worker’s belief in the mandate of the organization and the desire to make the world a better place. Once measurement is introduced, workers substitute the extrinsic motivation of trying to make the numbers look good and non-profits inevitably suffer as a result.
Another argument in the book should be required reading for those who run, and fund, Jewish communal institutions. Aside from showing the problems with some measurements, Muller argues that the non-profit sector has been adversely affected by the dubious assumption that measuring outcomes is both crucial and possible. Many non-profits are now led by someone who has no subject expertise, but has managerial experience acquired in business, or at a business school. Some even assume that a Jewish school, for example, would be better off if it were run by an expert in measuring inputs and outputs, even one who knows little about Judaism. One wonders whether such a person would be able to sense if students in the school love Torah, feel Jewish pride, want to learn Hebrew, behave ethically towards each other, feel an attachment to Israel or gain a sense of Jewish history. Muller builds a cogent case for what most people assumed until recent times: that the best person to sense whether a school, a charity or any non-profit is succeeding is an expert in the field who understands and believes in the goals of the institution, not a number cruncher who seeks out outcomes that may not be important, but are easy to measure.
