What do you do when your most important poverty measurement tool on a poverty outreach programme looks as if it is not working? Guest blogger Julie Lawson-McDowall writes.
Not so long ago, we at CRS’s Expanding Financial Inclusion (EFI) project (https://efiafrica.crs.org ) had a bit of a moment: it looked as if our main poverty measurement tool wasn’t working very well. We were using the Progress out of Poverty Index (PPI) to measure poverty (for those who haven’t come across it, the PPI is a poverty measurement tool that uses the answers to 10 questions about a household’s characteristics and asset ownership to compute the likelihood that the household is living below different national and international poverty lines). The PPI draws on national living conditions survey data to estimate a household’s poverty level and is thus a type of proxy means test. Since the EFI project’s objective was to reach the poorest people to include them in our savings groups (Savings and Internal Lending Communities), we had a problem if we couldn’t measure whether we were reaching poor people or not.
So, why did we think we had a poverty measurement problem? The EFI project reached over half a million people across 4 countries (Burkina Faso, Senegal, Uganda and Zambia). But, uniquely, for one group of 255 households in Zambia, we had two sources of data – and these two data sources were not lining up. In addition to the PPI survey data, for these 255 households, we had data from a 2-year financial diaries study. So, while the PPI data was telling us that in Zambia around three-quarters of our group members had at least an 89% chance of living on less than $1.25 a day, our financial diaries income and expenditure suggested differently. What to do?
Thanks again to the diaries, we had a third – and unusually rich – data source to hand: the 14 field officers who had spent two years living in the communities, each visiting the same 25 households weekly to collect financial data. These officers knew ‘their’ households’ inside out. We pulled them in for an in-depth wealth and wellbeing ranking exercise and reviewed the situation of the households from the point of view of income, resilience, assets and livelihoods.
When we compared the results from the 3 data sources (see Table 1, below), they all followed the same broad trend (which was a relief): for example, households that the wealth ranking placed as ‘well-off’ households were also less likely to be poor according to PPI likelihoods and had the highest expenditures, as measured by the Financial Diaries.
But, when we looked more closely, discrepancies began to emerge: for example, those households classified as ‘managing’ spent over one and a half times more than households classified as ‘poor’ but this difference was not reflected in the PPI scores where the two categories scores were only 7 points apart. In short, the PPI was not sensitive to differences across households that were highly likely to be poor but where some were much poorer than others.
It was clear that the PPI was struggling to measure differences in the depth of poverty. In other words, we had an absolute/relative poverty issue – nearly everyone in our Financial Diaries study was below the poverty line – but some scored 80% of the poverty line and some scored 40%. Nonetheless, in relative terms, there were important enough wealth, asset, resilience and livelihoods differences between these households for them to be considered to be managing well or poor by local ‘experts’; this suggested that the PPI was not doing a great job in distinguishing between households with different income levels where most were below the poverty line.
But, as we dug deeper into the data, another problem became clear: PPI likelihood of poverty and income were not matching up. Households falling into the higher likelihood of poverty categories should have lower per capita expenditures and the relationship should be linear. This means that the higher the poverty likelihood according to the PPI – in other words, the greater the chance of being poor – the lower should be the per capita expenditure. Yet we found several situations where those in a lower PPI poverty likelihood category – i.e. better off households – had a lower per capita expenditure than supposedly poorer households. This is illustrated in the graph in Figure 1, below, which, for each mark, shows the number of households in each poverty likelihood category and the poverty likelihood percentage. Generally, the graph slopes from left to right, which is what one would expect—as the poverty likelihood decreases so per capita expenditure should increase. But the devil is in the details: at poverty likelihoods above 80 percent, as measured by the PPI, the line sometimes turns back on itself, sloping from right to left, suggesting that some households with a greater likelihood of poverty according to the PPI actually had higher per capita household expenditures than households with lower poverty likelihoods.
What can we conclude? Overall, the PPI did a reasonable job of tracking broad trends in poverty for the households in the Financial Diaries study. But it struggled to tell the difference among households that were living below the poverty line: and in Zambia, around 40% of the population is below the extreme poverty line and around 64% are below the basic poverty line.
What are the consequences for programming if the PPI is insensitive to depth of poverty differences? First, it suggests that the PPI – a PMT – should not be used for targeting when the goal of targeting is to extend the depth of outreach among the poor, not simply to reach the poor. This means that, in Zambia at least, a PMT alone is not going to identify the poorest where so many are poor.
For those still wanting to use the PPI for targeting, we would recommend that the PPI should be complemented with other tools in future (although those tools, as ever, would depend on the objectives and the budget available). Second, the data presented in this report suggest that the PPI should not be used to measure progress from one level of poverty to another—in other words it should not be used to measure progress within poverty, but only more general progress out of poverty—because the PPI is not precise enough.
Finally, what was great about this piece of work was that we were able to share and discuss our findings with the PPI team and listen to their ideas about how they are planning to continue to refine and improve this tool, for example, by making it more sensitive to sub-national poverty variations.
Anyone wanting to read the brief or full report for this study should click here.
The Expanding Financial Inclusion in Africa Project (2013-2017) was a Catholic Relief Services project, funded by the MasterCard Foundation. This research was carried out with Samuel Beecher and Benjamin Allen from CRS and Guy Stuart of Microfinance Opportunities.