Debunking Myths and Modernizing Impact Measurement Practices, an interview with Tom Adams

This month, we interviewed Tom Adams, co-founder and chief strategy officer at 60 Decibels. He helps us to understand the complex world of impact measurement by debunking old persistent myths and he is providing us with solutions to truly modernize impact measurement practices.

Why is it important to measure the impact of investments accurately?  Is it possible to scale the practice?

It’s important because impact isn’t some abstract academic concept (even if it can sometimes feel like that), it’s real change in real people’s lives. Moreover, if an investor or organization has made a claim about impact (which they themselves are likely to benefit from) and it turns out not to be true, it’s not the investor who experiences that jeopardy, it’s someone else. And that person is very likely to be relatively less advantaged in society. Accurate impact measurement is therefore not only a tool for greater knowledge or improving our impact performance, it’s also morally essential too.

There is no fundamental limit to scaling good impact measurement. Like anything else, it’s predominantly a question of reducing its cost and increasing its value. What we’re focused on at 60 Decibels, is creating repeatable, simple, scalable social performance products. After a lot of experimentation, this has ended up being mainly 10-15 minute, phone based social research surveys, where we use standardized mixed-method surveys delivered through local researchers. Of course there is lots of devil in the detail in terms of designing good questions, quality control of data, etc. but it works because at its core that isn’t a very complicated approach to understand.

Measuring environmental impacts seems more scientific and straightforward: is this a misconception? Is it possible to measure social impacts with a similar level of rigor?

It’s a total misconception. It’s true that today the world is much further along in measuring environmental impacts. Part of this is undoubtedly about the effectiveness of the social and political movements around climate change (more power to their elbow) as well as the sense that we are up against the clock. But I also think it has got a lot to do with the perception of simplicity and focus on an ‘objective’ focal measure: carbon dioxide.

For some reason many professionals such as investors or donors seem unduly concerned about measures and KPIs always needing to be objective rather than subjective. Whereas social value is more often than not a combination of both the objective and the subjective, and using one without the other seriously limits our ability to fully understand it.

Maybe the reason for our collective caution regarding subjectivity is because we’ve been told that political polling is inaccurate, or because behavioral economists have been so successful in showing how irrational our brains are. Whatever it is, it seems very strange to me that we so often lose sight of the fact that social value is fundamentally about our perceptions, as humans, of our lived experience. The best way to understand that is simply to listen to people. 

As for robustness, there is a long established history of measuring objective and subjective measures through social research. Indeed quality social research has been around for centuries. Clearly you have to be deliberate about things like bias (which can come in a myriad of forms), but like anything else, with care this is something you can control and mitigate for.

What are the best measurement tools to get a clear picture of an investment’s impact?

This is where I have a lot of sympathy with people approaching impact for the first time. There are a dizzying number of tools and platforms claiming to measure and modernize impact, and a wide array of frameworks and standards to which they apply. It often makes my head hurt and I’m working on it every day. 

For the most part I’d divide all these tools in two: frameworks that help you decide ‘what’ to measure, and tools that provide a solution for ‘how’ to measure. 

Deciding on a measurement framework may depend on your strategy. Is it about managing financial-risk associated with social factors (ESG), avoiding harm, or maximizing positive social value. The Impact Management Project is rightly a well-recognized resource to better understand these approaches and what they may mean for measurement (though it is not a standard, nor does it actually guide you on how to measure).

How to actually measure – which has received comparatively less attention compared to what –  is in many ways a harder task. I’ve long thought this is the elephant in the room for impact measurement, and why good quality measurement remains pretty rare. 

In response to the challenge of a better ‘how’ the tools available to our sector have tended to focus on proxy-based measurement as a kind of ‘good enough’ short-cut. This is primarily done in two ways. First, platforms that use existing operational and financial data that are repurposed as impact (lives touched, jobs created, etc.). Second tools that consolidate existing research and apply it to other contexts. The issue is that proxies are not well suited for performance optimization. If we’re going to get as serious about impact performance as we are financial performance we need to move beyond them. Clearly, I’m biased, but I believe the best tools are those that listen directly to representative and statistically significant groups of stakeholders and ask them directly about their lived experience. These tools should then be repeatedly used at scale to create performance benchmarks. This is exactly how our products are built and used.  

On your website, you highlight the importance of data gathering being done by real humans. Why is that?

Yes we talk about data from (and by) real human beings. Despite the fact that it still remains the exception rather than the norm to collect data about social impact from real people, we find very few people would contest this. After all, if you’re trying to create a benefit in someone else’s wellbeing, who should be the ultimate judge of whether that happens, you or them?

The ‘by’ bit is more of a tactical choice. We knew that in-person surveying (what is often the de facto choice of traditional evaluation) was too expensive to scale and we experimented with a lot of tech-based tools to administer social surveys such as SMS, IVR and online surveys that are much cheaper. However, those tech tools had their own shortcomings (low response rates, short surveys, poor UX) and so we settled on person-to-person phone calls as a happy medium. We haven’t looked back. No tool is “perfect” but phone calls can be used to gather great data, offer a better experience for the interviewee, and critically can be scaled effectively.

Once you have the data, how do you use it? How do you identify the correct benchmarks to enable meaningful comparison?

Fabulous question. Ultimately the data is only as good as the decisions it is used for. This is usually a question for the company management/investors themselves. Of course we provide guidance based on our experience, but we are a data collection company, and at the moment our focus is on scaling that. We’re therefore cautious of providing definitive recommendations to people who know their business better than we do.

The point of our data is to either provide validation to what you may already suspect (in which case the decision is to carry on your current path) or to provide a new perspective that makes you ask new questions of your product or operational model. The data shows you’re not reaching the target population you aim to serve. Then what is your route to market, are you outsourcing or integrating your distribution model, is your pricing or financing model correct? If the data suggests you’re underperforming against our benchmarks. Consider what about your product or service is not creating the value you hoped for, what might a peer be doing that is better than you?

In terms of the benchmarks themselves these come from the repetition of questions over time. We can offer individual question benchmarks, such as the proportion of users of product X that live under income level Y, or our new approach is to combine multiple indicators of impact into impact indices. So take our microfinance impact index, our benchmarks here are made of the aggregated performance of lenders in achieving financial inclusion, better financial management, greater financial resilience, as well as impacts to a microbusiness and/or household.

What challenges do you think still need to be overcome in this field?

In truth the field of impact measurement remains somewhat nascent. There is much still to be built, norms over who pays for measurement, consolidation of standards, better clarity over reporting practices, and providers of assurance of those reports. 

This will take time. Essentially we are building the equivalent tools for social accounting that have been developed over hundreds of years for financial accounting.

However for those of us who are impact measurement geeks the pace of change and improvement is pretty remarkable, and speeding up each year. I have a lot of optimism that in the near future impact measurement and reporting will be completely overhauled. 

In doing so, business as we know it will change, and it will become entirely normal for businesses to measure and manage their social performance with as much focus and rigor as well as financial performance.

This article was originally published in Sustainable Finance Geneva in February 2023 here