Integrating Survey and Geospatial Data for Geographical Targeting of the Poor and Vulnerable: Evidence from Malawi

Abstract

To address the challenge of identifying the poorest villages in developing countries, this study introduces a cost-effective strategy that leverages a combination of household consumption surveys, geospatial data, and a partial registry. The study simulates a partial registry, containing data from 450 villages across 10 impoverished districts of Malawi, and contains proxy poverty indicators. These indicators are used to impute estimates of household per capita consumption, which in turn are used to train a prediction model using publicly available geospatial data. This method is evaluated against an imputed reference of village welfare, derived from the 2016 household survey. The partial registry approach is benchmarked against three alternatives: proxy means test scores, the Meta Relative Wealth Index, and predictions from household surveys with geospatial indicators. Results show the partial registry model's rank correlation with actual welfare measures at 0.75, outperforming the other methods significantly, which ranged from −0.02 to 0.2. These findings hold under various robustness checks, including the addition of Gaussian noise, indicating that collecting household-level proxy poverty data in low-income areas can significantly improve the performance of machine learning models that integrate survey and satellite imagery data for village-level geographic targeting.

1. Introduction

Identifying the poor in developing countries is crucial to inform development policies and programs, particularly those related to social assistance. However, governments and the development community are severely constrained by the high cost of collecting household surveys, censuses, or social registries that are typically used to inform targeting decisions. For example, between 2002 and 2011, 57 countries had conducted either zero or only one nationally representative household budget survey, preventing them from producing timely poverty estimates; and typically four years pass between nationally representative surveys on consumption or asset wealth in most African countries (Serajuddin et al. 2015; Yeh et al. 2020). Even when household data are collected, household surveys are typically too small to provide reliable estimates of welfare for small geographic areas such as villages. Estimating poverty at the village level requires alternative sources of data, traditionally census data, and utilizing this type of auxiliary data for small area estimation can provide resources to the poor more efficiently (Van Der Weide et al. 2022). However, census data are always expensive and sometimes politically fraught, and therefore collected rarely. Due to this lack of timely and adequate information on measures of well-being indicators in small areas, satellite imagery and other types of nontraditional data have great potential to fill these data gaps and complement traditional household surveys to provide more timely and accurate estimates for local areas (Yeh et al. 2020; Van Der Weide et al. 2022).

This paper investigates the benefits of combining traditional data with publicly available remote sensing indicators to predict welfare across approximately 4,500 villages in 10 districts in Malawi. The 10 districts correspond to the ones selected for phase one of the Unified Beneficiary Registry (UBR), which was conducted in 2017, collecting information on living standards to determine the eligibility of households for social programs.

Four alternative village geographic targeting methods are evaluated and compared to a benchmark welfare measure derived from an extract of the 2018 household census: (1) Proxy-Means Test (PMT) scores calculated in the 2017 UBR administrative data, (2) the Meta Relative Wealth Index,¹ (3) predictions derived from a village-level model estimated using a 2016 household survey and publicly available geospatial indicators, and (4) a two-step procedure that utilizes a hypothetical partial registry of 450 randomly selected villages—10 percent of the villages in the population—in addition to the 2016 household survey and publicly available geospatial indicators. This hypothetical partial registry would collect selected proxy welfare indicators such as asset and demographic information from all households in the selected villages. The first step entails predicting per capita consumption from the household survey into the partial registry data, simulated by sampling from the census extract. The second step uses the partial registry predictions to train a model using publicly available geospatial data to generate estimates for the remaining 90 percent of nonregistry villages.

For the purposes of evaluation, a measure of benchmark welfare is constructed by predicting household per capita consumption as a function of household and demographic characteristics in the 2018 census extract, using extreme gradient boosting, a popular machine learning algorithm. This model was then used to generate a prediction of per capita consumption for each household in the census extract, which was aggregated across households in each village to construct an “imputed reference measure of poverty” as an indicator of village welfare. This model is referred as the “census model” because it is used to impute predicted per capita consumption into the census extract. This census model captures over half of the variation in log per capita consumption in the household survey, with an out-of-sample R2 of 0.537. It is used both to generate the imputed reference measure of poverty using the full census extract and to impute welfare into the simulated partial registry, derived from a subsample of the census extract. To construct this imputed measure of village welfare, the values of predicted household per capita consumption in the 2018 census extract are aggregated to the village level.

For the main set of results, the reference village welfare measure used is the average predicted per capita consumption of the poorest 50 percent of households in the census, when ranked according to their predicted per capita consumption. This measure is selected as the main one to facilitate a clean comparison with the UBR, which only collected information on the poorest 50 percent of households. Since the PMT scores in the UBR are based on the average predicted welfare of the households in the bottom half of the distribution, evaluating it against a different benchmark could introduce an additional source of noise in the UBR comparison. However, the results are robust when using a more typical measure, namely the average predicted per capita expenditure across all households in each village.

The 20 percent census extract contains village, traditional authority (TA) and district names but not household or village geocoordinates. Obtaining information on the physical location of census villages is crucial for this exercise. Therefore, census village names were matched with UBR administrative data, containing the names of the administrative areas as well as the geocoordinates of interviewed households. This enabled the calculation of centroids based on the minimum and maximum latitude and longitude of households living in that village in the UBR administrative data, as an approximation of the village centroid in the census for about 4,500 villages. These centroids were then matched to a set of grids constructed to cover the country, in order to link remote sensing information to the census.

The remote sensing indicators were obtained from Google Earth Engine, WorldPop, and Meta. For each village, the average of approximately 40 indicators was calculated, including landcover indicators (e.g., percentage of vegetation, water, or build-up coverage), global precipitation measurement, soil moisture, nighttime data, and year of the transition from pervious to impervious areas (Gong et al. 2020). This was supplemented with gridded maps of building patterns (e.g., number, area, and length of buildings, among others) in 2017, population density indicators, build settlement growth, and distance to major roads taken from Worldpop. Grid-level averages of these satellite-derived features were then linked to the census data using the village centroids obtained from the matched UBR data.

This paper considers four main research questions: (1) How much does a hypothetical partial registry improve predictions of village-level welfare in this context, as compared with the existing UBR and two other feasible alternatives? (2) How much does increasing the size of the partial registry improve the accuracy of prediction models using geospatial indicators? (3) How robust is the partial registry to introducing noise in the training data? (4) How much less accurate is simple spatial interpolation based on a partial registry, relative to using a prediction model based on geospatial indicators?

The main result is that introducing the simulated partial registry yields predictions of village welfare that are vastly more accurate than the other methods considered. When using a partial registry, the rank correlation, Area Under the Curve (AUC) coefficients, and R2 values are 0.75, 0.89, and 0.57 respectively. In contrast, the rank correlation for the other three methods ranges from −0.02 to 0.2, the AUC scores range from 0.5 to 0.6, and the R2 of the predictions range from 0.001 to 0.04. These are huge differences in predictive accuracy.

These results are based on a simulated partial registry of approximately 450 villages, about 10 percent of the total number of villages with available data. However, the accuracy of the predictions does not substantially improve when the size of the simulated partial registry increases to 675 or 900 villages, which is 15 percent or 20 percent of the census extract. In other words, a partial registry of 450 villages is sufficient to train a high-performing predictive model in this context. Introducing substantial noise into the training data, as detailed in section 5, only modestly degrades the accuracy of predictions. Finally, simple spatial interpolation based on the partial registry generates estimates that are nearly as accurate as those from the baseline model, suggesting that geospatial indicators and a machine-learning model are not critically important for generating accurate predictions when using a partial registry. However, simple spatial interpolation is less robust to the introduction of noise into the training data than estimates based on geospatial features, which highlights an important benefit of incorporating geospatial data into the model.

This paper contributes to a growing literature on using satellite imagery to predict welfare. Initially, the most common remote sensing indicator used for this kind of analysis was nighttime lights, which measure the intensity of light in specific areas. Previous studies show strong correlations between night-time lights and GDP (Henderson, Storeygard, and Weil 2012; Pinkovskiy and Sala-I-Martin 2016). However, the association between night-time lights and other measures of household welfare is weak in most contexts, suggesting the limitation of this indicator for predictions of welfare in small areas (Mellander et al. 2015). More recent literature has demonstrated that indicators derived from daytime imagery is better suited for predicting welfare (Engstrom et al. 2015; Jean et al. 2016; Babenko et al. 2017; Engstrom et al. 2017; Head et al. 2017; Yeh et al. 2020; Chi et al. 2022; Engstrom et al. 2022; Masaki et al. 2022). Recent literature has also demonstrated that advances in machine learning and available nontraditional data can improve the targeting of social programs (Aiken et al. 2021; Smythe and Blumenstock 2021; Van Der Weide et al. 2022). Less attention, however, has been paid to how predictive performance relates to the nature of the training data.²

There are a few issues worth highlighting. One important issue is the choice of imputed welfare, based on a 20 percent census extract, as a benchmark reference measure. This differs from the gold standard of measured consumption taken directly from the household survey, which, all else equal, would be preferred. However, in this case, using imputed welfare as the reference measure instead of using measured consumption has several important advantages, including a broader village coverage, reduction of random noise, and more accurate geocoordinates, all of which enhance the reliability of welfare assessments. Notably, the decision to use census data is partly motivated by the significant impact of seasonality on consumption patterns in Malawi, as the census was collected at a single period in time and therefore eliminates noise due to the random timing of survey interviews across the year. However, imputation based on slow-changing characteristics poses a potential limitation in capturing transient welfare shocks, meaning that the benchmark welfare measure should be interpreted as a longer-term measure of well-being.

It is important to acknowledge additional limitations of the analysis. First, it only pertains to one context: 10 districts in Malawi. While there may be shared characteristics with other developing countries, additional research is needed to validate the applicability of these findings in different settings. Furthermore, the sample of villages is constructed by matching village names by hand, which raises the possibility that the sample of villages used in the study is not fully representative. However, as demonstrated in table 3, most of key observed characteristics in the census are similar on average between matched and unmatched villages.

A final methodological issue arises in the villages included in the simulated partial registry, where the same census model and data are used to construct both the benchmark measure and the partial registry predictions. As a result, the partial registry by construction perfectly predicts benchmark welfare in the 450 villages randomly selected for the registry. To address this issue, it is shown that replacing the perfectly accurate predictions from the partial registry for these villages with imperfect predictions generated by the geospatial model results in only a modest fall decline in predictive performance. Additionally, predictions using the simulated partial registry continue to significantly outperform the considered alternatives after introducing a large amount of noise into the partial registry predictions, equal to double the variance of predicted welfare across grids.

In this context, the present findings provide convincing evidence that partial registries that collect a limited set of indicators for all households in a sample of villages, if collected reasonably well, can greatly enhance the accuracy of geospatial predictions of village welfare. Many surveys routinely undertake full listing exercises in sampled enumeration areas, which could be extended to collect information on welfare proxies. To the best of the authors’ knowledge, the integration of household surveys, partial registries, and geospatial data remains unexplored. Yet the cost would be relatively modest; a preliminary estimate suggests that the marginal cost of collecting proxy welfare indicators from 18 households per village in 450 villages would be between $70,300 and $72,900.³ Moreover, this strategy appears to offer a large improvement over existing feasible methods when geographically targeting social assistance programs to poor villages in contexts where conventional data sources are incomplete or outdated. If the results from this paper are corroborated in other contexts and with actual partial registries, integrating partial registries into current systems of data collection could help existing surveys benefit more from the wealth of publicly available geospatial data.

The remainder of the paper is organized as follows: section 2 describes the data sets used to analyze and construct the benchmark welfare measure. Section 3 presents the statistical methodology used to generate alternative estimators of village-level comparison to evaluate against the benchmark. Section 4 describes the main results. Section 5 includes the robustness checks, and finally section 6 consists of a discussion and the main conclusions.

2. Data

The analysis utilizes data from approximately 4,500 villages in 10 districts in Malawi. The 10 districts correspond to those selected for the first phase of the UBR data collection. These districts are poorer than the rest of the country, according to data from the 2016 integrated household survey. For instance, households in the UBR districts tend to have lower educational attainment, as measured by the share of households where the highest educated male or female has completed secondary or tertiary education. Also, households in UBR districts are in rural areas and have lower-quality houses in terms of roof, wall, and floor materials. The UBR households are also less likely to have access to piped water or flush toilets and own fewer assets (e.g., cellphone, fridge, computer, cars, radio, television) (see table A1.1 in the supplementary online appendix to see the full set of statistics). Finally, as expected, UBR districts have significantly lower per capita consumption (fig. 1).

Description of the Datasets

The primary sources of information are (1) the Unified Beneficiary Registry (UBR), collected in 2017; (2) a 20 percent extract of the 2018 census provided by the National Statistical Office of Malawi; (3) the Integrated Household Survey (IHS) collected in 2016; and (4) publicly available remote sensing indicators.

Unified Beneficiary Registry (UBR)

Malawi's Unified Beneficiary Registry contains information on the households’ socioeconomic characteristics to determine their eligibility for social programs.⁴ The analysis uses the dataset from the first phase of the UBR, collected in 2017 across 10 districts: Lilongwe, Ntchisi, Kasungu, Rumphi, Chiradzulu, Nkhotakota, Blantyre, Karonga, Ntcheu, and Dowa. During this phase, half of the households in these districts were registered based on Malawi's average poverty rate. The UBR data set contains approximately 595,000 households, spread across 14,986 villages in the 10 districts.

The UBR is a crucial data set for this analysis for two reasons. First, it contains the PMT scores for the poorest 50 percent of households, which makes it possible to evaluate the PMT scores as a targeting mechanism. Secondly, these data include the geocoordinates of sample households, which makes it possible to merge the satellite data with the census data.

Census Data

The analysis uses a 20 percent extract of the 2018 census data for 10 districts provided by the National Statistics Office of Malawi. The census extract includes 235,600 households in 26,150 villages in the 10 UBR districts. The study utilizes data from the 4,500 villages that were matched, by name, with the UBR data.

The census extract serves two main purposes. First, it is used, along with the parameters of the census model estimated in the household survey, to generate the benchmark welfare used for evaluation. Second, it provides a randomly selected sample of villages that is used to simulate a partial social registry, as explained in the methodology section.

Survey Data

The analysis uses the fourth Integrated Household Survey (IHS) of 2016, which is made publicly available through the World Bank's Living Standard Measurement Survey (LSMS) program. The survey includes a cross-sectional sample of 12,447 households surveyed in 779 enumeration areas (EAs). Thus, there are roughly 16 sample households per enumeration area. It is considered to be representative at the district level. Because it is an LSMS survey, jittered enumeration area coordinates are also publicly available.⁵

The IHS is used in the analysis for two primary purposes. The first is to estimate a model that predicts per capita consumption as a function of household variables common to the survey and the census extract, as a basis for constructing the benchmark measure of welfare in the census. Besides serving as a benchmark for evaluation, this predicted welfare measure is also used to simulate a partial registry in a subsample of villages. The second main purpose of the IHS is to train a model that predicts welfare based on publicly available satellite data, which is one of the candidate prediction methods that is evaluated in this study.

Satellite Data

The satellite data come from three different sources: Google Earth Engine, WorldPop, and Meta. Table 1 summarizes the indicators used for the analysis, and table A1.2 in the appendix provides additional information.⁶ For annual data, the collection targeted 2017 and 2018 to align with UBR and census years. These data are collected at the grid level, and then are merged with the village centroids from the UBR.

Matching the Census and the UBR Data By Village

Matching villages between the UBR and the census data is a critical step that enables the linking of census villages with remote sensing indicators. The matching is based on names using an algorithm that matches two text variables and assigns a similarity score. The matching starts at the biggest administrative unit, Traditional Authorities (TA), followed by Group Village Names (GVN), and finally at the village level. This process resulted in 32 percent (4,727⁷) of the UBR villages being matched with the census villages. Six UBR TAs and 24 percent of the GVNs are not in the census. Among the nonmerged villages (10,181), 38 percent (3,894) are in nonmatched GVNs. The second panel of table 2 shows the matching at the district level. The districts with the lowest percentage of matched villages are Rumphi, Nkhotakota, and Lilongwe.

It is important to check whether there is any systematic selection bias in the sample. Table 3 shows the comparison of census means in the matched and unmatched villages. It shows that in most of the features, the difference in means between matched and unmatched villages is very small and not statistically significant. However, matched villages have a higher share of households with more educated adults. Also, households in matched villages have a lower child dependency ratio and slightly higher elderly dependency ratio. Finally, a higher share of households in matched villages have better roof quality in their houses. Households in matched villages appear to be slightly better off, but the differences are minor.

3. Methodology

Four geographic targeting methods are proposed, using census and survey data along with geospatial data, to determine the most effective way to target the poor population: (1) the PMT scores calculated in the UBR data of 2017; (2) the Relative Wealth Index from Meta (3) combining survey data and geospatial indicators to predict average welfare; and (4) a census sample to simulate a partial registry data set used to train models using satellite data.

Rank correlations, Area Under the Curve coefficients, and R-squared coefficients serve as the basis for comparing the accuracy of each targeting method. In each case, the predictions are compared with a benchmark welfare measure, an imputed reference measure of welfare constructed from the census model.

Construction of Imputed Reference Benchmark Welfare Using the Census Model

A benchmark welfare measure is defined to evaluate different targeting methods. The primary welfare measure is the average predicted log per capita consumption of the poorest 50 percent of households in each village, mirroring the UBR's focus on the lower half of households based on the average poverty rate in Malawi. This approach leverages the detailed data in the census sample, and is easier to predict than measured consumption due to reduced measurement error and its inability to capture temporary shocks.⁸ To construct the benchmark welfare in the census, the IHS 2016 serves to estimate a model for later welfare imputation in the census.

The variables included in the model are selected so that both data sets contain the same information. All the variables are measured at the household level. The model includes education variables such as the literacy of the household head, the maximum level of education achieved by men and women in the household, dependency ratios, household size and overcrowding, house characteristics, and household assets. The dependent variable is the log household per capita consumption.

A machine learning model is trained to predict household per capita consumption using an extreme gradient boosting model with optimal hyperparameters chosen via 5-fold cross-validation. The parameters used in the models correspond to the average of the selected parameters in each fold. Details on the range of parameters considered are shown in table 4. Additional technical information regarding these parameters and other aspects of the extreme gradient boosting procedure can be found in  appendix 2 in the supplementary online appendix.

Four models are estimated using different samples in the IHS: (1) the full sample (All districts-all households), (2) all households in the UBR districts (UBR districts-all households), (3) the poorest 50 percent of the households in all districts (All districts-poorest 50 percent), and (4) the poorest 50 percent of households in UBR districts (UBR districts-poorest 50 percent). In each case, the calculation of out-of-sample R-squared uses the corresponding dependent variable and sample. Estimating four models makes it possible to assess the trade-offs along two dimensions: (1) using all districts in the sample rather than restricting to UBR districts and (2) using all households rather than just the lower half. Table A1.3 shows the out-of-sample R-squared for each model, estimated using a held out 50 percent test sample, and the main explanatory variables in terms of the gain measure.⁹ Including all households (model 1) leads to higher R-squared, while limiting to the bottom half of UBR districts (model 4) leads to by far the lowest R-squared. All models assign high importance to similar variables, mostly household assets, household size, and the urban or rural location (for the specific gain measures of each variable, see fig. A1.1).

The model that best matches the UBR sample is model (4) since it uses only UBR districts and the poorest 50 percent of households; however, the sample size is small, and the R-squared is the lowest. Although model (1) has the highest R-squared, it also has a wider range of variability in the dependent variable, making R-squared a potentially misleading metric. Models (2) and (3) have similar values of R-squared.

Model (3) is selected as the main census model because it resembles the UBR's sample structure, using only the poorest 50 percent of households in each village. However, it also takes advantage of additional data on log per capita consumption by using the full set of sample survey enumeration areas nationwide. Table 5 shows the values for the out-of-sample R-squared and the importance of the top 15 features in the selected model. These features explain around 87 percent of the model, with the “urban/rural” indicator being the main contributor. A robustness analysis using the other three samples is presented later. This selected model is used to predict the benchmark welfare variable in the census extract, which is generated at the household level first, and then aggregated to a village welfare measure as the average of the bottom half of households in each village.

Criteria Used for Evaluating Prediction Accuracy

Three main criteria evaluate the accuracy of the predictions against the benchmark: the Spearman Rank Correlation, the Area Under the Curve (AUC), and the R-squared, which is equal to the share of the variation explained by the prediction. While the latter is a standard measure of prediction accuracy, the first two criteria require a brief explanation. The Spearman rank-order correlation coefficient (⁠|${r}_s$|⁠) is a statistical measure of the strength and direction of a monotonic relationship between two variables measured on a continuous scale (Zar 2005). The rank correlation between two variables, X and Y, is calculated as follows:

Where |${\rho }_{R( X )R( Y )}$| denotes the Pearson correlation coefficient applied to variable ranks; |$cov(R( X )R( Y )$| is the covariance between two ranked variables, and |${\sigma }_{R( X )}{\sigma }_{R( Y )}$| are the standard deviations of the ranked variables.

Finally, the Area under the Curve (AUC) is a measure of the efficacy of a targeting method in identifying poor villages at different targeting thresholds (Wodon 1997; Hanna and Olken 2018). The curve in question is a receiver operator characteristic (ROC) curve, illustrating the trade-off between true and false positives at different poverty lines. Plotting the ROC curve occurs by using each percentile of the imputed reference welfare measure in the census. Specifically, for village poverty lines defined at each percentile of the benchmark distribution, the curve plots the true positive rates (TPR) on the Y axis and the false positive rate (FPR) on the X axis. The TPR measures the proportion of poor villages that are correctly identified as poor using predicted village welfare from each candidate prediction method, whereas the FPR measures the proportion of nonpoor that are incorrectly predicted to be nonpoor. Because the true positive rate is on the Y axis, a higher AUC score represents an improvement in true positives for a given level of false positives, or a better geographic targeting method. The 45-degree line, which is the expected result if villages were ranked randomly, corresponds to an AUC score of 0.5. Meanwhile, a perfectly accurate ranking that correctly identifies poor villages under all hypothetical village poverty lines would receive an AUC score of 1.

Candidate Geographic Targeting Methods for Identifying Poor Villages

Proxy Mean Test scores in the UBR

The administrative data from Malawi's UBR provides information on households’ characteristics to assess their prospective eligibility for social programs. The data set contains an extensive range of variables such as geographic location, households’ assets, food security questions, and economic characteristics. This information was used by the Malawian government and the World Bank to calculate Proxy-Means Test (PMT) scores to identify poor and vulnerable households.

In the first stage of rolling out the UBR, PMT scores were calculated using a two-step method. Initially, the PMT was applied to all households after data collection. Subsequently, the community participated in a validation process, leading to another round of PMT application to create the final UBR version (Lindert et al. 2018). The PMT acts as a tool to generate a proxy score, weighing variables strongly linked to household consumption, essentially indicating well-being. Importantly, PMT is structured as a measure of chronic poverty, using variables resistant to sudden changes.

The variables and their weights for the PMT in Phase 1 districts were derived from a statistical analysis of Integrated Household Survey 3 (IHS3) data from 2010–2011. This PMT version uses a nonincome and expenditure approach, forming an asset-based wealth index through Principal Components Analysis (PCA) (Kachaka 2012). About 70 variables from household composition, education, housing materials, lighting sources, house conditions, occupancy, land ownership, assets, food security, and livelihoods contribute to the PMT. Each proxy is assigned a weight, resulting in a calculated score for each household, establishing a household ranking.

Kachaka, Kalimba, and Lunhanga (2020) suggest an updated PMT aligned with chronic poverty characteristics observed in the most recent household survey at the time, the IHS4 (2015–2016), for future UBR phases. This updated version uses PCA based on 26 variables, covering demographic characteristics (like household size or composition), education of the household head, housing features (floor, roof, walls), durable goods, and productive assets.

Several challenges emerge when implementing a PMT-based targeting methodology, especially in verifying proxies such as age, education, and occupation. Examples include the absence of birth certificates in Malawi, under-reporting of education, and the provision of false occupational information. Additionally, verifying productive assets like livestock, land size, and ownership is challenging, and smaller durable goods can be easily removed during inspections (Narayan and Yoshida 2005). Lastly, enumerators may not administer surveys in a fully objective way and often face time constraints in validating proxies within households.

Because the PMT variable is available in the data and was used for the UBR, it is useful to evaluate it against welfare predicted into the census extract.

Relative Wealth Index

The Relative Wealth Index predicts the relative standard of living within countries using nontraditional data sources such as satellite imagery, cellular network data, topographic maps, and proprietary connectivity data from Meta. Using supervised machine learning models, the team predicts the relative wealth for grid cells of 2.4 km². The estimates of wealth are relatively accurate. Depending on the method used to assess the model's performance, the model explains 56 to 70 percent of the actual variation in household-level wealth in 56 low- and middle-income countries (Chi et al. 2022). However, the model is trained on a wealth index, which may perform less well when predicting income or consumption-based poverty measures. For example, the Relative Wealth Index (RWI) only explains 32 percent of the variation in average per adult equivalent consumption across cantons in Togo (Aiken et al. 2021). Furthermore, for a microcensus conducted in rural Kenya, the RWI explains 70 percent of the variation in wealth but only 17 percent of the variation in the predicted probability of being poor, defined using household consumption (Chi et al. 2022). Thus, the performance of the RWI varies greatly depending on the context, and particularly depends on whether it is evaluated against a wealth or consumption-based measure of welfare. This study therefore contributes additional information on the performance of the RWI in distinguishing among very poor villages by comparing it to the benchmark measure of village welfare, based on predicted per capita consumption, in Malawi.

IHS Plus Geospatial Indicators

The third alternative method for geographic targeting consists of using the survey data to train a welfare model against satellite indicators. This model can then be used to generate out-of-sample predictions into villages for which matched census data is available, to compare against the benchmark. This method has the advantage of not imposing any additional cost, but may suffer from limited training data. The model estimation targets only the poorest 50 percent of households in each village to resemble the structure of the UBR dataset, employing extreme gradient boosting techniques for training.

Partial Registry

The final alternative method to target the poor involves considering a hypothetical collection of a partial registry data from a sample of villages. This exercise would consist of collecting a subset of household welfare proxies, used in the census model, from all households in a random sample of villages, similar to an expanded sample listing procedure of the type typically carried out for household sample surveys. For this analysis, the approach includes simulating a partial registry by drawing a random sample of villages from the census extract. This sample, consisting of all households in the sampled villages, is used as the hypothetical partial registry in a two-step procedure. The first step involves utilizing the parameters from the census model that were used to predict per capita consumption, which creates the imputed reference welfare measure in the simulated partial registry. The predictions are based on 39 independent variables common to both the census and survey. The resulting predictions for households are then aggregated into a measure of village welfare.

The second step of the prediction process entails estimating a second model, the geospatial model. The dependent variable in this model is the imputed reference welfare measure from the simulated partial registry, and the independent variables are village geospatial indicators, also estimated using extreme gradient boosting. This “geospatial model” is used to predict out-of-sample welfare predictions for villages not included in the partial registry. The predictions from the two models are combined, by using the census model predictions from the simulated partial registry in the villages for which they are available, and the geospatial model predictions for the remaining villages not included in the registry. Therefore in villages covered by the simulated partial registry, predictions from the census model are used instead of those from the geospatial model. This approach is chosen because the simulated partial registry provides more accurate predictions than geospatial data, which are in fact exactly equivalent to the benchmark welfare measure by construction.

Additional analysis reported below demonstrates that predictions from this procedure become only modestly less accurate when using only out-of-sample predictions or full-sample predictions for all villages. Consequently, although predictions from the census model exactly match the benchmark in partial registry villages, this is not a major factor in explaining the improved performance of the partial registry predictions. Instead, the partial registry provides richer data with which to train a more accurate geospatial model. Furthermore, for a robustness exercise, a simulation introduces additional noise to the imputed welfare reference measure in the partial registry, replicating potential errors that could occur in real-world data-collection processes. The findings indicate that, even with the inclusion of Gaussian noise in the reference measure, the partial registry method remains the most effective option when compared to alternative methods.

4. Main Results

This section shows the results of the four alternative geographic targeting methods. Comparing welfare predictions generated by each method to the imputed reference welfare measure, involves using rank correlations, Area Under the Curve (AUC) coefficients, and R-squared coefficients.

Figure 2 presents the metrics for each method. The partial registry is clearly the most accurate method for targeting the poor villages in the 10 UBR districts in Malawi.¹⁰ The second-best is using the RWI to rank villages. However, this method performs only moderately well, with an AUC coefficient close to 0.58 and a 0.2 rank correlation. In contrast, using the PMT scores as a proxy for welfare does not show promising results; the PMT scores show zero correlation with the benchmark welfare and have an AUC coefficient equivalent to guessing poor villages at random. The IHS plus the geospatial variable model shows the second-lowest coefficients, only slightly higher than the PMT scores in terms of AUCs. This is due to the limited sample size in the household survey. Because the sample size is restricted to the bottom half of households within UBR districts, there are only an average of about eight households per village available to train the model. One indication of this is that performance improves noticeably when predicting average welfare across all households in the village.

The ROC curves of the four methods are presented in fig. 3. This is the graphical representation of the AUC results. The partial registry method leads to far superior targeting outcomes, especially at low poverty rates. The RWI is the second-best method but is still far from the partial registry results. The curves for PMT scores and IHS are very close to each other and to the 45° line that corresponds to randomly selected villages.

Two factors may contribute to the superior predictive performance of the partial registry method relative to the direct predictions from the household survey. The first factor is that the geospatial model is trained to a measure of village welfare that is far more precisely estimated than the one used in the sample. The welfare measure is more precisely estimated both because it is based on data from a much larger number of households from the census extract, and because it is a predicted welfare measure that largely eliminates classical measurement error, which partly results from the influence of seasonality present in the survey data. More precise training data improves the ability of machine learning to construct a predictive model, by reducing the risk that particular predictive variables will be fit to random noise in the training data, and by improving the accuracy of the cross-validation procedure used to select models. Hence, the superior performance of the partial registry might be explained by both a larger quantity of training labels and also a large reduction in random noise in the training labels.

A second potential factor for the better performance of the partial registry method is that it uses predictions derived from the census model, which is also used as the imputed reference welfare measure. Using the same values for prediction and evaluation can be defended for enhancing accuracy, yet it also inflates measured performance. To address this issue, two assessments are conducted. The first assesses the accuracy of the geospatial model's predictions for all villages (full sample predictions), instead of relying on the imputed reference welfare measure available for 10 percent of the villages and the out-of-sample predictions for the remaining 90 percent. The second evaluation concentrates on the out-of-sample predictions exclusively. Figure 4 shows that when considering the full sample predictions, the rank correlation falls from 0.75 to 0.73 and the AUC coefficient falls from 0.89 to 0.78. Nevertheless, these values still significantly outperform the RWI method, which is the second-best approach in the comparison. The RWI method exhibits a lower rank correlation of 0.20 and an AUC of 0.60. Finally, when considering the out-of-sample predictions for the evaluation, the metrics are still close to the study's preferred results. The analysis reveals a rank correlation coefficient of 0.72, and an AUC coefficient of 0.84. The R-square shows very little variation among the three options considered. Thus, the superior performance of the partial registry method is largely due to the presence of more accurate training data on village welfare.

The RWI, although the second-most accurate method for predicting the benchmark welfare, is far less accurate than the partial registry results, and only explains 4 percent of the variation in the study's imputed measure of village welfare. This is probably because the RWI is based on a measure of household wealth instead of consumption or predictive consumption. Wealth may not be as accurate at distinguishing the welfare levels of villages within 10 poor districts. Moreover, the wealth index reflects the full distribution of households, whereas the benchmark welfare measure only pertains to the bottom half.

The RWI, however, performs better than efforts to integrate the household survey with publicly available geospatial data in the absence of the partial registry. This is because of the noise in the household survey data used to train the model. Given that only the bottom half of the household survey data is used to train the model, there are only roughly eight households per EA with which to generate a measure of consumption. The resulting machine learning model is therefore not particularly accurate.

Finally, the PMT scores from the UBR 2017 are the least accurate proxies for village-level welfare, as defined by the imputed measure of village welfare. These inaccurate proxies are not entirely due to outliers. Figure 5 shows the presence of some outliers in the raw scores; however, even after trimming the values, the correlation is low. Some of the relatively poor performance of geographic targeting based on the UBR PMT might be attributable to problems with data collection, given that it was the initial effort to collect data for the UBR (see “Candidate Geographic Targeting Methods for Identifying Poor Villages” in section 3).

Figure 6 displays another way to present the main results. It plots the predicted welfare measures (on the Y axis) against the actual imputed reference village welfare (on the X axis). Villages are divided into four groups depending on whether their benchmark and predicted welfare falls into the bottom quartile. The bottom left and top right quadrants represent villages that are correctly predicted to be in or out of the bottom quartile. The upper left quadrant shows targeting errors of exclusion, and the bottom right shows targeting errors of inclusion. Of the four methods, it is clear that the partial registry approach has by far the lowest prevalence of points in the top left and lower right quadrants, and that these errors are closer to the center. Meanwhile, the PMT has the highest prevalence of errors, especially errors of exclusion. The Meta RWI and the geospatial household survey model have similar error rates, with the latter slightly less likely to suffer from large exclusion errors.

Finally, fig. A1.2 presents heat maps of the predicted per capita consumption using each method and the benchmark welfare. In the maps, the PMT scores fail to predict welfare mainly in the central region of Malawi which includes Lilongwe, Dowa, Ntchisi, Nkhotakota, and Kasungu districts. Most methods make accurate predictions in two districts: Rumphi and Chiradzulu.

5. Robustness Checks

This section presents the results of robustness checks along six dimensions: The size of the partial registry (training sample), the nature of the village welfare measure, the benchmark welfare used for evaluation, geographic composition of the census model, adding Gaussian noise to the study's imputed reference welfare measure in the partial registry, and assessing the accuracy of spatial interpolation models in this context.

The Size of the Partial Registry

Given the impressive predictive performance of the partial registry predictions, one might wonder whether increasing its size would further improve performance. Figure 7 compares the performance of the partial registry predictions for partial registries of 5 percent villages (224 villages), 10 percent (444 villages), 15 percent (668 villages), 20 percent (888 villages), 25 percent (1109 villages), and 30 percent (1332 villages). Overall, expanding the size of the hypothetical partial registry offers only limited improvements in predictive accuracy, especially when going from a 5 percent to 15 percent sample sizes. However, there are no improvements when expanding the size to 20 percent or larger, which is therefore not worth the added expense it would entail.

Welfare Measure: Mean of the Bottom Half Versus Mean of All Households

A second issue is the impact of the choice of welfare measure on the results. This is important because the results until now have used a nonstandard welfare measure, namely the average predicted per capita consumption of the bottom half of households in the village. This was based on a conscious decision to match the UBR administrative data, which only contains PMT scores for the bottom 50 percent of households in each village. This section assess how the results vary when considering mean village consumption across all households, as the main welfare measure.

Changing the welfare measure has three main implications for the methodology. First, the census model must now be retrained using all households, not just the bottom half, in the survey. This, of course, also changes the predicted values of household per capita consumption in the simulated partial registry, which is equal to the imputed reference welfare measure for villages included in the registry, which entails re-estimating the geospatial model. Finally, the IHS plus geospatial model is re-estimated to train it against average per capita consumption across all households in the survey, rather than just the bottom half, to provide a cleaner comparison.

Table 6 displays the results when using mean village consumption including all households instead of the mean of the poorest 50 percent of households as the village welfare measure. Three main findings are clearly apparent. First, the partial registry approach continues to perform vastly better than the other alternatives when attempting to predict mean village welfare. Second, both the partial registry approach and the RWI suffer moderately when predicting the mean over all households rather than the mean of the poorest half of households, particularly when it comes to rank correlations. This may be because of idiosyncratic positive outliers in the upper half of the household predicted welfare distribution, which are more difficult to predict using both geospatial data and predictions trained on asset indices.

Third, the method that combined survey and geospatial predictors without a partial registry (IHS plus geospatial predictors) performs much better when using the full sample of households than when only using the bottom half for each village. The rank correlation increases from 0.09 to 0.19, and the AUC increases from 0.54 to 0.59. This is because on average there are only approximately sixteen households interviewed in each village in the IHS, and average per capita consumption is much more accurately measured when all sample households in each EA are used to train the model rather than only the bottom half. The resulting predictions, when using the full IHS sample, also perform better than the Meta relative wealth index. In this context, when trying to predict the average predicted per capita consumption from a census extract, the fact that the RWI uses additional training data from many countries and proprietary indicators on connectivity does not fully compensate for the fact that it is trained to predict an asset index rather than a consumption-based welfare measure. Therefore, a model that predicts average village per capita consumption directly on the basis of publicly available geospatial characteristics is slightly superior for geographic targeting in this context, though both are far worse than collecting additional partial registry data to train a better geospatial model.

Using Survey-Measured Consumption Instead of Imputed Per Capita Consumption as the Reference Measure

This section turns to the choice of reference measure used to evaluate the different geographic targeting methods. The main reference measure used in the paper is an imputed value of consumption based on household assets and demographics. Ideally, one would use a reference measure that reflects information on measured consumption for all households in the selected enumeration area. Unfortunately, no known cases exist where such data are available, either in Malawi or elsewhere, due to the high cost of collecting consumption data. Therefore, consumption measures are available only in two-stage sample surveys that collect data from a sample of households in each selected enumeration area. As a result, the choice narrows down to two options: using a measure of imputed per capita consumption in the census or using measured consumption taken from the household survey. This study elects to use a census-based imputed welfare measure, instead of the commonly used survey-based per capita consumption measure, because the census covers many more villages and imputing into it reduces random noise in the welfare measure. The reference benchmark when using the census data is available for 18 times as many villages as the survey. Specifically, the household survey contains 4023 households in 252 enumeration areas in the UBR districts, which amounts to approximately 16 households per enumeration area. while the census extract contains 83,890 households spread across 4,716 villages (on average, 18 households per village). The prediction model used for imputing per capita consumption into the census explains about 53 percent of the variation in measured per capita consumption. Even in the extreme scenario where household consumption was measured entirely without random noise in the survey, the effective variation in the imputed census measure across all UBR areas would still greatly exceed that in the survey measure.

Another important advantage of the census-based approach is its ability to provide an imputed welfare measure that corresponds to the same time period for all villages. In contrast, survey-based measures are collected from villages at randomly selected months, leading to varying levels of noise in the data when comparing welfare across regions. For instance, the consumption survey data in the IHS, collected between April 2016 to April 2017, may not accurately capture the dynamic seasonal variations in consumption patterns across villages in the 2018 census, as villages are interviewed at different times during the year. The use of census data therefore addresses an important source of measurement error in village-level survey-based estimates.

This temporal misalignment gains significance when considering Malawi's consumption patterns, which exhibit high seasonality primarily due to the dependence on rain-fed agriculture. As highlighted by Chirwa, Dorward, and Vigneri (2013), over 80 percent of the population dependent on agriculture experiences highly seasonal income and consumption patterns, which can significantly impact poverty levels. Similarly, findings from de De Janvry, Duquennois, and Sadoulet (2022) suggest that gaps in rural-urban annual consumption are strongly associated with differences in time worked due to seasonality differentials. Additionally, the literature supports the importance of seasonality in the region, highlighting its significant impact on consumption patterns, food prices, and household strategies to cope with seasonal hunger (Kaminski, Christiaensen, and Gilbert 2014; Anderson et al. 2018).

Additionally, using an imputed census measure reduces the extent of random noise more generally, because that noise is not correlated with the predictors. In addition, in this particular case using the imputed census data makes it possible to get the exact village geocoordinates from the UBR census data, which are presumed to be accurate; in contrast, the geocoordinates in the survey contain random noise designed to maintain confidentiality.

In contrast, the main disadvantage of using the imputed reference measure is that the imputation is carried out using slow-changing characteristics such as household size, head's education, child dependency ratio, and household assets. Therefore, the benchmark welfare measure is a longer-term measure of welfare that will only partially include transitory welfare shocks. It is not clear how big of a disadvantage this is for the purposes of targeting poor villages.

While consumption survey data are valuable when estimating aggregate parameters, including metrics of targeting performance, its limitations become pronounced in geographically granular analyses, particularly given the substantial influence of seasonality in Malawi. Hence, a census-based imputed welfare measure is selected to provide more reliable and consistent welfare assessments, especially at the village level. This in no way diminishes the continued value of consumption survey data for aggregate analyses that largely average out classical measurement error, as well as analyses that are robust to high levels of measurement error at the village level.

The Geographic Composition of the Sample

The census data serve as the basis to evaluate the estimates and as the source for the simulated partial registry. However, because the census extract is only available for 10 districts, it is not immediately clear whether it would be best to use only the survey data from those 10 districts, or the full set of survey data to train the census model. The latter takes advantage of a wider set of training data, but the former may better capture the specific relationships between welfare and household characteristics in those poor districts.

Table 7 shows the results when varying the household survey sample used to estimate benchmark welfare. Specifically, an experiment is conducted wherein only the UBR districts in the household survey data are used to train the census model, instead of the full sample. While the partial registry method remains the most accurate method by far, it doesn't do nearly as well when the benchmark welfare measure is derived from a model trained on data from only the UBR districts. This is because the sample size used to train the models declines significantly from 6,000 to 2,000 of the poorest households when limiting the training sample to households in UBR districts, leading to a less informative benchmark welfare model and measure. The partial registry method is particularly sensitive to the weakening of explanatory power in the census model, due to limiting the training data to UBR districts. This is because the predictions from the census model are also used as the dependent variable to train the second-stage geospatial model that generates estimates for nonregistry villages. Interestingly, however, the predictive performance of the RWI also declines substantially, due to the increase in noise in the benchmark measure of welfare. Nonetheless, the predictive performance of the UBR improves, suggesting that the PMT may have picked up some of the heterogeneity in welfare patterns within the 10 districts.

Adding Noise to the Partial Registry Welfare Measure

Up until now, the proposed partial registry method has been evaluated by training the geospatial model using the same imputed reference measure that is used to evaluate the performance of the model. This procedure represents an optimistic scenario where the welfare variable collected in a hypothetical partial registry is collected without error, which is unlikely to hold in practice. To address this concern, Gaussian noise is introduced to the village-level measure of welfare derived from the census model, specifically targeting the average predicted per capita consumption of the bottom half of households. This adds noise to the training data and makes it possible to examine how much this degrades performance of the partial registry approach. The variance of the imputed reference measure of welfare, predicted per capita consumption in the census, across villages is approximately 0.04. Tests are conducted with five possible scenarios, each involving the addition of Gaussian noise at varying levels of variance. These levels correspond to 25, 50, 75, 100, and 200 percent of the actual variance of the mean predicted per capita consumption of the bottom half. Table 8 shows that when evaluating the methods using three criteria (R-squared, rank correlations, and AUC) for out-of-sample predictions, their magnitudes decrease. However, even in the most extreme scenario, when adding noise equal to twice the true variance of the measure, the partial registry still outperforms the other three methods considered. The partial registry method is quite robust to noise in the measure of predicted per capita consumption because of the relatively large number of grids available to train the model, which collectively contain sufficient information to distinguish signal from noise.

Using Spatial Interpolation for Prediction

As a final robustness check, the results from a simple ordinary Kriging (spatial interpolation) model with no covariates are considered. This model is estimated using the gstat package in R (Pebesma and Gräler 2022). This method has the advantage of only requiring information on predicted welfare from the partial registry, and not any information from geospatial auxiliary data. Similar to the gradient-boosting models, a training set equal to 10, 15, and 20 percent of the matched villages is randomly selected. Spatial interpolation is then applied to predict into the remaining test set. The out-of-sample predictions from the testing set are combined with the dependent variable in the training areas, following the same approach used in the main results.

The results in table 9 show that the simple Kriging model performs nearly as well as the geospatial XGboost models. In the 10 percent sample, the rank correlation of the Kriging predictions is 0.70, only modestly lower than the 0.72 for the predictions obtained using XGboost. Similarly, the AUC of the Kriging predictions is 0.83, only slightly below the 0.84 obtained for the no noise case.

Because the Kriging method applies spatial interpolation between points, however, this method may be more vulnerable to measurement error in the partial registry. The second and fourth columns of table 9 report results when training the model with a noisy version of predicted per capita consumption. This was created by adding a Gaussian noise term with variance equal to 0.04 to the imputed reference benchmark measure. Indeed, the spatial interpolation model is less robust to the addition of this noise. The R-squared measure in particular falls significantly, from 0.45 to 0.19 in the 10 percent sample case, when training using the noisy measure. In contrast, for the XGboost model the R2 only falls from 0.52 to 0.4. While the measures for rank correlation and AUC are less stark, these also indicate that the Kriging model is less robust than the use of auxiliary geospatial data to noise in the imputed reference benchmark model.

6. Conclusions

This paper, evaluates different alternative methods to identify poor villages in 10 Malawian districts. This is a challenging prediction exercise because villages are highly geographically disaggregated. The results show that a two-step approach utilizing a hypothetical partial registry from 450 villages performs much better than the PMT, geospatial prediction based solely on the household survey, or the Meta relative wealth index. The main measure used to identify poor villages is the mean predicted per capita consumption of the bottom half of households in each village, but key results hold when using the mean predicted per capita consumption of all village households as the village welfare measure. In addition, the results hold when adding significant amounts of noise to the benchmark reference measure of imputed per capita consumption before training the geospatial model. The strong performance of the partial registry approach is maintained when using a simple spatial interpolation model rather than incorporating geospatial features, although this is less robust to noise in the measure of imputed per capita consumption used to train the model. The findings suggest that the training data are as, or more, important than the choice of predictors for generating accurate predictions, and that there are therefore large returns to collecting better training data in the form of partial registries.

Implementing the partial registry method requires nationally representative survey data and the collection of a partial registry containing a subset of household characteristics found in the survey data. Several similar household surveys that collect information on welfare proxies have been fielded with the support of the World Bank through the Survey of Well-Being with Instant and Frequent Tracking (SWIFT) program, including in Malawi (Yoshida et al. 2022). Although none have surveyed the full population of households in selected villages, it is quite standard for household surveys to list all surveys in sampled enumeration areas. Based on discussions with the World Bank LSMS team, the cost of collecting approximately 40 variables from an average of 18 households in approximately 500 villages could be in the ballpark of $24,000 to $73,000.¹¹ The evidence presented in this paper suggests that this is a worthwhile investment to greatly boost the accuracy of village welfare measures constructed using geospatial data. Some countries field periodic community surveys, which could potentially be tweaked to collect partial registries.

The relatively poor performance of the PMT scores derived from the UBR data is a puzzle. The UBR PMT scores performed a bit better when the benchmark measure of welfare was constructed using data only from the 10 Malawian districts. Even so, the UBR PMT scores do not appear to be consistent with welfare predicted using the census data collected from these districts. Partly this may be due to the UBR data being taken from the initial phase of data collection, although it is also possible that the purpose of the partial registry may have led to measurement error. It would be useful to make these types of evaluations with further rounds of the UBR, even if compared against old census data, to see if later rounds of the UBR produce predictions that are more consistent with the census.

An important issue when making this evaluation was the choice of benchmark welfare measure. A census-based imputed welfare measure offered several advantages over commonly used survey-based per capita consumption measure, including broader coverage and larger sample size, with the reference benchmark available for 18 times as many villages compared to the household survey. Additionally, the imputed census measure provided a snapshot of welfare measures across villages during a single month, minimizing noise caused by seasonality. Moreover, the use of imputed census data allowed for the extraction of accurate village geocoordinates, enhancing spatial accuracy in analyses. However, it is important to note the limitation of using slow-changing characteristics for imputation, which may only partially include transitory welfare shocks. Although consumption survey data retain their usefulness for specific analyses, especially when examining aggregated data, the decision to employ a census-based imputed welfare measure provided the most accurate benchmark possible in this context for evaluating different predictive models.

Two additional limitations of this study are that it only applies to 10 districts in Malawi and is based on a 20 percent extract of the census. Additional work could demonstrate that similar results hold in different contexts and when using a full census. A third limitation is that the hypothetical partial registry is taken from the census, and therefore assumed to match the census exactly. In reality, measurement error in data collection for the partial registry will reduce its performance relative to census-based predictions. It is reassuring, however, that key results are robust to the introduction of substantial noise in the prediction based on the partial registry. Nonetheless, a project that pilots the collection of a partial registry for the purpose of training a geospatial model would provide a more realistic test of the partial registry approach and could shed new light on whether such a partial registry would be prone to systematic bias.

Finally, future research could leverage household-level information on geocoordinates if they can be obtained in census data. This would enable estimating models relating predicted welfare to geospatial indicators at the household level, which may perform better than the village-level models considered in this analysis. Overall, the results convincingly demonstrate both the limitations of existing methods, and the potential for partial registries to add massive value when combining survey and geospatial data to identify the poorest villages in a very low-income setting.

Data Availability Statement

Some of the data underlying this article were provided by the National Statistics Office of Malawi under licence / by permission. Data will be shared on request to the corresponding author with permission of the National Statistics Office of Malawi.

Author Biography

Melany Gualavisi is a consultant in the Development Economics Data Group at the World Bank; her email address is [email protected]. David Newhouse (corresponding author) is a senior economist in the Development Economics Data Group at the World Bank; his email address is [email protected]. The authors thank Lina Cardona, German Caruso, Chipo Msowoya, and Boban Paul for their support in providing data, financing, and support for this project. The authors also thank Ifeanyi Edochie for assistance with obtaining geospatial indicators, and Professor Sarah Janzen, Nobuo Yoshida, and seminar participants at the University of Illinois and the World Bank for constructive comments. Funding was provided entirely by the World Bank Group.

Footnotes

References

Appendix 1. Additional Tables and Figures

Appendix 2. Xtreme Gradient Boosted Models (XGBoost)

XGBoost is a gradient-boosting algorithm that provides a parallel tree boosting that solves data science problems in a fast and accurate way. It is designed to work with large and complex data sets.

This appendix describes the use of Xgboost for regression. The algorithm fits a regression tree to the residuals as gradient boost but uses a unique regression tree. Each tree starts with a single leaf that is called a root, and all the residuals go to the leaf. The algorithm calculates similarity scores and gain to determine how to split the data.

The similarity score for the residuals on each leaf equals

Where λ is a regularization parameter intended to reduce the prediction's sensitivity to individual observations and prevent overfitting the training data. If the leaf has several different residuals, the similarity score will be relatively small, since they will cancel each other out. In contrast, if the residuals are similar or the leaf has very few residuals, the similarity score will be relatively large.

To quantify how much better the leaves cluster similar residuals than the root, it is necessary to calculate the gain of splitting the residuals into groups. The gain is equal to

Then the algorithm compares the gain calculated for each split and selects the one with the highest value since that would mean that a particular feature is better at splitting the residuals into clusters of similar values. Then it continues with another split. It is possible to limit the tree depth or the splits to different levels; up to six levels is the default.

To determine output values for the leaves, the following is calculated:

The output value is like the similarity score, except that it does not square the sum of the residuals. After this, the tree can be used for making predictions. Like gradient boost, xgboost makes new predictions starting at the initial prediction and adding the output of the tree scaled by a learning rate ε. The new predictions will have smaller residual values. Then the algorithm builds new trees based on the new residuals until the residuals get very small or it reaches the maximum number of trees.