Poxy Means Testing

PMT is a curse! Sisters, you all know that: inescapable, debilitating, emotionally draining, a regular cause of extreme irritability!

But I refer here not to Pre-Menstrual Tension, but rather to a new form of PMT that is sweeping the globe: Proxy Means Testing.

This variant of PMT is a method of selecting poor people to become beneficiaries of social transfer programmes, currently being advocated strongly by, among others, my good friends at the World Bank. Proxy means tests generate a score for each applicant household, based on “fairly easy to observe characteristics of the household such as the location and quality of its dwelling, its ownership of durable goods, demographic structure of the household, and the education and, possibly, the occupations of adult members” (http://go.worldbank.org/SSMKS9WUT0).  The specific indicators used in calculating this score and their relative weights are derived from statistical analysis (usually regression or principal components analysis) of data from detailed household surveys.

PMT is touted as generating “impressive” results; it claims to be based on “statistically rigorous methods”; in Chile (where it all began), it exhibits an “excellent record” of targeting; in Fiji, it has been pushed as being “highly reliable”; in Jamaica, “leakage errors are less than 3 percent”; etc. As a result, the World Bank claims that PMT is “objective”, that it has fewer disincentive effects than a true means test, and that it has been “proven to work particularly well in countries with high levels of informality and where personal and household income is difficult to verify with any degree of precision”. Overall, the advocates of PMT paint a happy picture of a scientifically sound, technocratically robust and dispassionately objective solution to poverty targeting.

But a recent paper, Targeting the Poorest[i], suggests that the reality is very different, and cautions policymakers strongly against the dangers of being steamrollered into the adoption of PMT. It suggests that the PMT approach is demonstrably deficient in five main areas.

First, the datasets on which PMT is based are not fit for purpose.  The household surveys on which proxy means tests are modelled are designed to build an aggregate picture of poverty at national, regional and – less often – district levels; they are not appropriate for a detailed understanding of poverty in individual households. In addition to the intrinsic sampling errors, household surveys also suffer from substantial non-sampling errors, such as (i) lack of clarity on what constitutes a household, (ii) incomplete coverage, and (iii) the reticence of households to provide accurate information. These difficulties are further compounded by the fact that household surveys typically measure only consumption and expenditure, not income; that household surveys reflect only a single moment in time (often once every five or ten years), while poverty at household level is highly dynamic; and that data on asset-ownership (much used within PMT) is a reflection of past income, not of present income, which would tend to penalise, for example, older households, who have accumulated assets over a long period but whose current income is diminishing.

Second, PMT analysis can be unduly influenced by arbitrary statistical choices. The paper looks at three specific examples:

  1. How equivalence is calculated. Some analysts do not apply equivalence scales (ie they treat a child as having the same level of consumption as an adult), whereas others treat children up to 12 (or 14 or 16) as being equivalent to 0.5 (or 0.8) of an adult.
  2. How missing variables are interpreted. Inevitably there are missing data in household surveys; and analysts must decide whether these non-responses should be imputed or treated as absent. This decision impacts on the estimated value of the coefficients in the regression, and hence on the weights used in the PMT score.
  3. How sampling errors are treated. Household surveys only provide estimates based on a sample; their precision therefore falls within a range, at a given level of confidence. The paper tested two scenarios, one using the lower bound at the 95 percent confidence interval, and one using the upper bound.

The paper looks at the actual datasets for specific countries where PMT is used, and demonstrates that each one of these three sets of assumptions can arbitrarily change the eligibility status of some 10% of households. Cumulatively, this could mean that the eligibility of over 30% of households is determined not by its inherent poverty status, but by the statistical whim of an analyst.

Third, the regressions used in PMT do not provide sufficient clarity to distinguish between poor households. Clearly the choice of variables to be included in the PMT influences the outcome: by selecting only a subset of “fairly easy to observe” variables, the PMT model is inherently less able to reflect the same degree of variation as the more comprehensive list in the full consumption measure. Looking at examples in four countries where PMT is being used, the paper shows that PMT regressions typically only explain about 50% of the variation in consumption between households. What is worse is that they are particularly weak at the poorer end of the scale, thus making it especially difficult to distinguish between the poorest households. In other words, PMT “performs the weakest at the point where it would be expected to find the best correlation between assets and consumption”. The paper’s analysis of data from four countries (Bangladesh, Rwanda, Sri Lanka and Indonesia) shows that, depending on programme coverage, targeting error in PMT programmes is typically between 35% and 43% at a 30% coverage level, between 44% and 55% when 20% of the population is covered, and a staggering 57% to 71% at a (more common in reality) 10% coverage level.  Interestingly enough, a separate study in Pakistan, this one by the World Bank itself (Report No: 47288-PK, May 8, 2009), reported even higher exclusion errors: of 61% at 20% coverage and of 88% at 10% coverage. And remember that this is just the theoretical error of the PMT regressions: it will inevitably be further compounded by errors connected with the household data, with statistical analysis and with implementation.

Fourth, PMT faces significant challenges at implementation, and the paper cites a number of examples of this. There is the problem of finding the beneficiaries, using either a census or on-demand method, with examples of enumerators not wanting to enter urban slums because of security concerns; male enumerators being barred from  entering households where only females were present; evangelical families refusing to take part in the enumeration process; and nomadic groups, temporary migrants and remote communities being deliberately excluded. Another issue is the objectivity of enumerators, often with excessive demands placed on them, inadequate training and insufficient supervision: examples have been documented of corruption, inadequate time per interview, and deliberate changing of results where PMT was perceived to be wrong. Then there is the question of the verifiability of the indicators: assets can be hidden; ownership is hard to prove or disprove; education, occupation and even age can be falsified; indeed the fear that proxies may be easily manipulated has led advocates of PMT to suggest that proxies and weights should be kept secret – not exactly an advertisement for “transparency”! Other documented weaknesses include the fact that community verification is rarely effective; there is evidence in many countries of political interference; recertification is seldom sufficiently regular to capture the dynamics of poverty; and there is often no effective appeals mechanism – indeed there is an inherent irrationality in even introducing an appeals system against what is claimed to be a fair, objective and transparent system of selection. In this regard, it is notable that the World Bank’s social protection handbook (For Protection and Promotion: The Design and Implementation of Effective Safety Nets) clearly states: “Proxy means tests are most appropriately used where a country has reasonably high administrative capacity” … which raises the question of why so many countries with relatively weak administrative systems (e.g. Pakistan, Kenya, Nepal, Fiji, Niger) are being encouraged to adopt the PMT methodology.

Finally, PMT does not avoid the social, moral, incentive or political costs of targeting. In terms of social costs, the paper cites qualitative research in Mexico, Nicaragua and Peru indicating that some community members ascribe the omission of poor households to luck or God’s will, describing the PMT methodology as similar to a lottery; the apparent unfairness of selection leads to feelings of despair, frustration, resentment, anger and envy, and there is evidence that this has resulted in a breakdown of community cohesion and even conflict. Morally, there is clearly the issue, as noted by Sen, that people may be rewarded for being deceitful and punished for being honest, which may in time corrode the fabric of society. Nor is it clear why incentive costs should be any less when using PMT: if potential beneficiaries are aware of the proxies, such as possession of animals or farm implements, they will be less likely to invest in them. And among the political costs of PMT is that poverty-targeted programmes, especially when perceived as arbitrary, tend to alienate the middle classes: evidence suggests that programmes using PMT never command as significant a share of GDP as, for example, universal programmes such as child grants and social pensions.

What do these deficiencies mean in practice? All in all, the paper finds that – despite all the grandiose claims of its proponents – PMT performs lamentably in targeting the poor.  It concludes that a striking finding of the analysis was the consistency of the magnitude of errors across countries, suggesting that such levels of error are to be expected using PMT methodologies as currently employed. It counsels policymakers to bear in mind “this combination of theoretical errors means a majority of eligible poor households may be permanently excluded from social grant benefits as a result of PMT scoring”.

As I said, sisters, in another context, PMT appears to be “debilitating, emotionally draining, a regular cause of extreme irritability” … but at least we can put a stop to this variant!

Authored by Afternoon (PM) Teese

[i] AusAID, Targeting the Poorest: An assessment of the proxy means test methodology, September 2011

Also published on Medium.

Leave a Reply

Your email address will not be published. Required fields are marked *