‘Metrics: What Counts in Global Health’ reviewed by Dr Megan Wainwright.

‘Metrics: What Counts in Global Health’ edited by Vincanne Adams (Duke University Press, 2016).

Metrics: What Counts in Global Health is a collection of ethnographic case studies of the process, effects and limitations of quantifying global health interventions and outcomes. Metrics here refer to the mathematical approaches and tools that quantify the implementation and outcomes of a health intervention. It is a broad term that can refer to simple counting, calculations such as disability-adjusted life years (DALYs), cost-effectiveness measures and the analyses stemming from randomized controlled trials (RCTs). Yet, far from being simple calculations, the book demonstrates how metrics act as a ‘new kind of “global sovereign”’ (Vincanne Adams, p.45). The case studies, which span Alaska, Haiti, Malawi, USA, Senegal, and Nigeria, offer rich accounts of ‘what they [metrics] do far beyond the rhetorical claims to improving health’ (Adams, p. 225). In fact, a key claim of the book is that the emphasis on performing quantitative metrics may not work in the interest of mitigating health inequities at all.

The authors (mostly anthropologists based at universities in the USA and Canada), while presenting a critical perspective on metrics, are careful to repeat — like many of the interviewees they quote — that they are not arguing that metrics are of no use and nor should they be abandoned. Their point is to instead expose the limitations and consequences of metrics and question their supremacy in the practice of ‘proving’ what works, ‘what counts,’ and what global health interventions should be funded.

Adams in the introduction of the book places the rise of metrics in a broader historical context, and outlines how the supremacy of metrics has arisen with the transition from ‘international health’ to ‘global health.’ Whereas in international health the emphasis was on counting the reach of an intervention (how many children were vaccinated, how many women consulted for pre-natal care), in global health the emphasis has shifted to measuring the impact of the intervention on health (i.e. did vaccinating all children in a village lead to fewer deaths before the age of five?). Also, whereas funding for international health was a project of ‘giving money away,’ in the age of global health, funders invest in global health and expect financial return (Chapter 6, Susan L. Erikson). Erikson’s case study is a fascinating and frightening depiction of the shift from accountability metrics to market metrics, and is unique within this edited collection. The fact that investors in global health can make money whether an intervention improves human health or not, and the fact that their expectations are driving the field of global health, is itself a potential risk to public health ‘…of yet unknown magnitude and scope’ (Erikson, p162).

Most of the contributions however concern the metrics of accountability and a key contribution of the book is making clear through ethnographic evidence that metrics is a form of storytelling that is always political and tells only partial stories (Chapter 1, Adams). These stories may be partial because parts of the story that cannot be quantified are left out (Chapter 3, Adeola Oni-Orisan), or that the pressure to get the good and the right data ‘can also exact a type of violence of erasure’ (Adams, p.226). In Chapter 5, Molly Hales gives an example of how the demand for metrics to show a Native Alaskan-run intervention’s impact on sobriety and healing is in direct conflict with Yup’ik conceptualization of healing as a process. The nature of ‘knowledge’ is also a problem, as for Yup’ik empirical observations and intuitive experience are equal forms of knowledge, but metrics can only measure the former. Lily Walkover, in Chapter 7 on the public health publisher of Hesperian Health Guides, discusses how the publishers measure impact in terms of the demand for the books, their translation and adaptation, while funders want to know impact on the health of individuals — something impossible to count. In Chapter 8, Carolyn Smith-Morris argues that while the quantitative metrics showed a positive outcome of an intervention on employment of veterans, the guidelines for RCT reporting made no space for the qualitative evidence that explained why it had worked. Because the “intensity of work” of the individual health care workers could not be quantified, this part of the story and its implications for understanding success, and for planning replication in other settings, was silenced.

Pierre Minn brings in the important role ‘temporality’ plays in the partial stories captured by metrics in Chapter 9. In his example of a US NGO’s struggle to show the impact of their efforts to strengthen a health system (rather than deliver services themselves), the issue becomes a temporal one — that while service delivery can be counted now, strengthening health systems is an outcome that is farther into the future than most funding cycles. This brings up the issue of using metrics to compete for funds, and in turn satisfy funders to ensure continued support. Marlee Tichenor in Chapter 4 tells the story of health workers in Senegal withholding data as a form of political resistance. The aim of the health workers was to threaten the government’s relationship with outside funders to whom they owed numbers in exchange for funds. For Tichenor ‘participation in these data regimes and accepting global health governance are requirements for access to global health markets’ (p.122). This highlights how the production of numbers becomes the currency with which one trades to receive funding. The stakes are high, and so the risk of leaving out numbers that do not tell the ‘right’ story becomes likely — something we see in Claire Wendland’s chapter where she recounts her direct observation of a woman’s death not being counted at all.

The themes of politics and data-ownership cut across case studies. Who takes ownership of data? Who makes the data work for them? In Chapter 2 Wendland describes how the Malawian president took the credit for favourable maternal mortality statistics that in fact corresponded to a period that preceded her period in office. Tichenor argues that in the data withholding strike, some health centres, particularly in rural areas simply stopped collecting data at all, which only served to widen the gap between urban and rural health. With an emphasis on producing data for the outsider, health workers could not see their ownership of the data and what the data could do for them and the health of the population they served. Similarly, Minn described the large, dust-covered ledgers that represent the performance of good data in a Haitian clinic and how local staff felt little ownership over this data or the reports or publications eventually produced. The Yup’ik on the other hand were clearly interested in how they could mobilize metrics for their own interests, even though the obligation to count neglected part of their story (Chapter 5, Hales). Through these varied case studies, we see clearly how ownership is imagined and performed differently across contexts.

For those of us who do qualitative research, Metrics explains a phenomenon we know well. One that our research in the current regime does not ‘count’ when evaluating the success or failure of public health interventions, and two that numbers on the other hand are, to put it lightly, not all they are cracked up to be. Thus, Metrics offers us a firmer leg to stand on when discussing or arguing that an uncritical, decontextualized faith in numbers is not only short-sighted, but potentially damaging to health. Note that it is not the numbers themselves that are a problem, it is what they are considered to stand for, what decisions they engender and how they are produced. In the words of Wendland, metrics ‘…tell a story like any other kind of data. It is when we fail to recognize them as partial, created, fallable stories that the metrics of global health are especially troubling’ (p. 79).

So, if quantitative metrics are not enough on their own, then what? In the epilogue Adams takes up this question briefly, reminding readers that the stories left untold by current metrics in the book’s case studies highlight the role ethnographic research can and does play. On the one hand, she makes the point that ethnography is needed to study the sociology of global health metrics. On the other hand, ethnography and other social research methods could offer new models for evaluation. But how should ethnographic stories — stories like any others — avoid the traps of power, ownership, distortion and erasure? The book left me yearning for a more in-depth discussion of how, in the wake of the increased stakes and pressures inherent in the shift from ‘funding’ to ‘investment’ in global health (Erikson), can ethnography learn from the critiques of quantitative metrics in practice. This part of the story remains to be written.

Reviewed by Dr Megan Wainwright, a medical anthropologist and Postdoctoral Research Fellow in the Division of Social and Behavioural Sciences, School of Public Health and Family Medicine, at the University of Cape Town. She is also a collaborator on the Life of Breath project at Durham University and a member of the GRADE-CERQual project group.

Correspondence to Dr Megan Wainwright.