Missing top income recipients and what to do about them

Low response rates among rich households are thought to be a serious problem in many applications using household surveys, including the measurement of poverty and inequality and in distributional policy evaluation. A new paper, “Missing Top Income Recipients” (link below), discusses the various ways the problem can be dealt with, and makes some recommendations for practice. The context includes developing countries with weak statistical systems, as well as rich countries with “state-of-the-art” systems.

Finding that richer households are less likely to participate in surveys when sampled does not necessarily imply that we under-estimate inequality, by any standard measure. That is an empirical question. (We can actually be more confident on theoretical grounds that standard poverty measures will be over-estimated for this reason.) The empirical studies so far (mainly for the U.S.) do suggest sizeable under-statement of the extent of inequality based on survey data, including with the commonly used methods of trying to correct the problem, either ex-ante or ex-post.  There are also serious concerns about under-estimation of mean income from surveys. On balance, poverty measures tend to be reasonably robust to this problem.

What can be done about the problem? It will be obvious to any user of micro survey data for empirical analysis that it is desirable to correct for the bias in top-income shares due to selective compliance internally to the survey data. Doing so can retain both the statistical integrity of the survey design and the great many applications for micro-data files in distributional analysis. How can that be done?

Non-compliance with survey sampling can take the form of outright refusal or not being home for interviews. Either way, it is an outcome of choices made by those sampled. A theme of the paper is that thinking about behavioral responses helps to clarify the concerns about the non-behavioral methods underlying current practices, and it helps in thinking about better methods and measures.

As has long been recognized in the statistics literature, under certain conditions, income-selective non-compliance with an initially randomized assignment can be corrected by reweighting the survey data. Economists have given too little attention to how survey weights are estimated; indeed, it seems that the weights are almost always taken for granted.

Past methods of calculating these weights do not appear to take proper account of the behavior of those selected for a sample, and thus do not deal fully with the survey compliance problem. Newer methods are reviewed in the paper—methods that can do a better job, and their data requirements should make them feasible in many settings. Modelling compliance requires identifying assumptions, and these are contestable, though certainly no more so than the assumptions made by ad hoc non-behavioral methods.

The (old and new) reweighting methods require that the surveys pick up at least some top incomes. That may be a serious concern—called a “failure of common support”—although there is remarkably little evidence one can point to. Indeed, the literature on top incomes using non-survey data has boomed with rather little evidence on whether common support holds, such that survey data could be adequately re-weighted.

If common support does not hold, then income tax records can help, including in estimating distributional national accounts. There has been much progress in obtaining and carefully using these data, also in combination with survey data. There are also some reasons for caution. Tax data come with their own concerns including tax avoidance/evasion, weak coverage of informal sectors and illicit incomes, including capital flight, and concerns about construct validity, given the limitations of taxable income as a basis for inter-personal comparisons of economic welfare. The paper reviews the methods and issues going forward, but it is clear that more work is needed.

While such measurement problems need not prevent us making the best estimates possible from the data available, the validity of the data and methods can, and should, be challenged continually, and under public scrutiny. That is how we make progress in measurement.

How much all this matters to policy remains to be seen. Just as top incomes are hard to reveal for measurement purposes—and we may never know just how unequal income and wealth are distributed—they can be hard to tax for financing and redistributive purposes. The hope remains that progress in advancing measurement will also help foster better policies.