Measuring Religious Non-Affiliation in China: A Comparison of Major National Surveys in China

Abstract

Measuring the rate of religious non-affiliation is a difficult task in China, one of the least religious countries in the world, and elsewhere. This study evaluates the trend of religious non-affiliation in China using 22 surveys spanning three decades. We found that two surveys asking beliefs in a mix of gods and religions yielded dramatically lower estimates of non-affiliation compared to the others that measured affiliation explicitly. Among the latter, using a filter question (a yes/no question for religious affiliation) increased the reports of non-affiliation but allowing multiple choices and putting “no religion” toward the end of the lists of response options moderately decreased the reports of non-affiliation. After accounting for question design and demographic differences across the surveys, the decline in religious non-affiliation is reduced and becomes not significant during more recent years.

Is China becoming more religious? Answering this question appears straightforward: ask a well-written set of questions about religiosity, or lack thereof, to a representative¹ sample of adults from the Chinese population. Wait a few years, ask the same questions in the same way to an independent sample from the same population, and compute the difference between the two estimates.

Like most survey research on religion, however, this project is not as straightforward as it may at first seem. Religion is a notoriously difficult concept to measure well. This is true in China, the West, and elsewhere (Brenner, 2011, 2016; Yang, 2011; Yao, 2006). The equivalence of religious concepts (e.g., services, prayer, and scripture) across countries and religious traditions may be poor (Brenner, 2016). Measurement equivalence is arguably more achievable for religious affiliation as it is autobiographical, in contrast with more subjective or transitory states (e.g., strength or importance of belief). Thus, affiliation, or the lack thereof, may be the most reasonable starting point for this investigation given the relative comparability of its measurement between surveys (Brenner, 2016).

Our particular interest in assessing religious affiliation is in its absence: non-affiliation. Given the focus of the extant literature on the supposed sacralization of China in recent years, we estimate the rate of religious non-affiliation in China from 22 surveys over the past three decades after accounting for differences in survey design and respondent demographics.

Prior research

Relatively little work on the rate of Chinese religious non-affiliation has been undertaken. Although a few research articles note a single estimate from a cross-sectional survey (e.g., Lu, 2014), few others tackle the primary question of interest here by comparing rates of affiliation (or non-affiliation) over time to estimate trends in Chinese religiosity. Nearly all of this extant work, albeit scant, agrees that religious affiliation is increasing in China.

Two studies compare surveys across years to estimate change over time. Yao (2007) compares estimates of religiosity from two surveys: a 1995 survey of six urban areas conducted by HorizonKey, a Chinese survey firm; and a 2005 survey conducted by the Religious Experience Research Center at the University of Wales (see also Yao, 2006). This analysis finds that, along with increases in religious belief and practice, religious affiliation also increased modestly from two to five percent. Similarly, Stark and Liu (2011) compare two surveys conducted a few years apart: a 2001 survey conducted by the Research Center for Contemporary China at Peking University; and a 2007 survey conducted by HorizonKey. Finding that the percentage reporting “no religion” decreased from 93% to 77% between these two studies, they argue for a religious “awakening” in China.

One additional study uses data from the 2010 Chinese General Social Survey to compare respondents’ reports of their own current religious affiliation with that of their parents (operationalized as respondents’ retrospective reports of their parents’ religious affiliation during the respondent’s childhood). Although straightforward cross-tabulation of parents’ and children’s affiliations shows strong stability in reports of non-affiliation and a tendency for those with parents who practice traditional Chinese religions to become non-affiliated, Hu and Leamaster (2015) argue that more sophisticated statistical techniques demonstrate religious mobility; more specifically, a movement away from non-affiliation and toward a religious affiliation.

Taken together, these findings are cited as evidence against the secularization thesis (Stark, 2015; Stark & Wang, 2014, 2015; Yang, 2011). Defined briefly as “the idea that modernization tends to undermine religious belief and activity” (Voas & Chaves, 2016), evidence for secularization is arguably found in the United States and Canada, Europe, other highly developed countries, as well as some of the developing countries of Central and South America (Brenner, 2011, 2016; Voas & Chaves, 2016). However, if the evidence against secularization (and for sacralization), such as increasing religious affiliation, can be found in already supposedly secularized countries, such as China, secularization theory would be contradicted and potentially falsified.

Thus, we critically review the existing evidence and address two major limitations of this research. First, as demonstrated previously, the extant work relies on relatively few studies, each reporting findings from, at most, two surveys. Although any two data points can be connected to form a perfectly fitted trend line, erroneous inferences may result if either data point includes bias. Thus, more data are needed. We pool and analyze data collected in 22 surveys from five probability-based social survey programs that include measures of religious affiliation in China. We use these data to predict the rate of non-affiliation for each survey and estimate the trend of religious non-affiliation.

Second, differences between surveys related to the particularities of their design (e.g., question wordings or contexts) have the potential to generate artificial variation between estimates (Brenner, 2019). Thus, we control for study-level factors, such as question wordings, to estimate the contribution of each to variation between estimates. We also include year of survey administration as a key explanatory variable to estimate the change in religious (non-)affiliation over time.

Methods

Data

This study analyzed data from multiple waves of five surveys: Chinese General Social Survey (CGSS) (National Survey Research Center at Renmin University of China, 2014a, 2014b, 2014c, 2016a, 2016b, 2016c, 2018), Chinese Social Survey (CSS) (Li, 2017a, 2017b, 2017c, 2019), China Family Panel Studies (CFPS) (Institute for Social Science Survey at Peking University, 2015), China Labor-force Dynamics Survey (CLDS)(Center for Social Science Survey at Sun Yat-sen University, 2015, 2016), and World Values Survey (WVS) (Inglehart et al., 2020). Each of these surveys was conducted via face-to-face interviews.

The studies varied in important survey design components. Each survey has been conducted at least twice in the past. CGSS, CSS, and WVS used cross-sectional designs with independent samples from year to year. The other two survey programs used longitudinal designs to track the same sample (CFPS) or rotated samples (CLDS) over time. Analytic sample sizes ranged from around 1000 in the WVS (1990 and 2001) to over 30,000 in the CFPS (see Table 1).

Although each of these surveys reported using a national multi-stage probability sample, CFPS and CGSS provided more detailed sampling design information than did the other surveys.² The CFPS sampling design excluded five autonomous regions in western China (i.e., Xinjiang, Tibet, Qinghai, Inner Mongolia, Ningxia) and Hainan island. The CLDS sampling design excluded only Tibet and Hainan. CFPS, CGSS, and WVS (wave 6 and 7³) used a stratified sampling design that based the selection of primary sampling units on factors related to the socio-economic status of each area (e.g., GDP per capita, population density, geographic locations, etc.).

Measures

Our dependent variable is respondents’ reported religious non-affiliation. However, the religious affiliation questions varied in their designs both within (from wave to wave) and between survey programs (see Supplementary Appendix A for question wordings). As we discuss, design differences could potentially add variation to the survey estimates, making it hard to interpret the overall trend.

1. Use or lack of a filter question. Survey questions about traits, such as religious affiliation, often employ filter questions (e.g., “Do you have a religious affiliation?”) before asking for more detailed information of only those respondents who report “yes.” Three of the five WVS surveys used a filter question before asking specific affiliations. Research has demonstrated that filter questions increase the percentage of respondents who report no religious affiliation (Dougherty et al., 2007; Lim et al., 2010). Thus, we expect the use of a filter question to increase the estimate of religious non-affiliation in China.

2. Belief in or affiliation with religion. While most of the surveys explicitly asked about religious affiliations, CFPS 2014 and 2018 asked respondents “what do you believe?” as a measure of affiliation. The response options of CFPS 2014 were Buddha/Bodhisattva (菩萨[pusa] in Chinese), Taoist Immortals, Allah, Catholic God, Christian God, and Ancestors. (If respondents only select “ancestor,” we code it as non-affiliation.) CFPS 2018 had a similar list of response options except that some did not explicitly refer to the religion. As an example, “Taoist immortals,” a response option offered in CFPS 2014, was changed to “Immortals” in CFPS 2018. What is captured in those two surveys tends to be a mix of belief in gods and belief in religions, not the same as the rest of the surveys that explicitly measure affiliation.⁴ Arguably, using these belief questions as measures of affiliation will likely produce lower estimates of non-affiliation. To evaluate such effect, we keep the belief questions in our analysis to show their effect on the estimates of non-affiliation, despite the two types of questions not measuring the same concept.

3. Placement of “no religion” in the list of options. Without a filter question, response options included the specific religions and a “no religion” option. However, surveys differed as to whether the “no religion” option was at the top or toward the bottom of the list. When respondents lack the motivation to answer carefully, they may select the first response option that appears to be reasonable (Krosnick, 1991). Respondents may interrupt the interviewer and report “no religion” when the option is presented first without waiting to hear the rest of the response options. If so, the estimates of religious non-affiliation could be higher when “no religion” is on the top of the list compared to when it is at the bottom.

4. Number of response options. Surveys differed on the number of religions provided in their response options. Most of the surveys include Buddhism, Christianity, Catholicism, Islam, and Taoism. CGSS in recent years also included Judaism, Hinduism, and various Christian denominations. Because longer lists of affiliations will include less common religions, we speculate that respondents may interpret religions included in these expanded lists as being more common in China than they are and, therefore, may be more likely to be reported.

5. Multiple choices vs. Single choice. Some of these surveys allowed respondents to select multiple religions as their affiliations while others just permitted one. This design choice varied both within (over the multiple waves of a study) and between studies. Allowing respondents to choose multiple options may appear to lower the bar for reporting a religious affiliation, encouraging respondents to report nominal or residual affiliations. Therefore, we expect fewer reports of non-affiliation as a result of allowing respondents to select more than one religion.

Analysis

We first calculate the rate of non-affiliation for each survey and the standard error of each estimate. Next, we use logistic regression to estimate the probability of being religiously non-affiliated. Explanatory variables include previously outlined survey design factors, survey year, and respondent demographics (sex, age categories, and education levels) available across all the surveys. The aim of the regression analysis is to (1) examine the effects of survey designs on reported non-affiliation and (2) estimate the potential temporal change in religious non-affiliation beyond what can be attributable to the variation in survey designs and respondent demographics. As most of the surveys do not make their weights and relevant documentation publicly available, we do not use weights in the analyses.

Results

Survey estimates of non-affiliation

Figure 1 presents the percentage of non-affiliated individuals accompanied by a confidence interval for each survey over the nearly three decades of available data (see Table 1 for detailed statistics). The WVS has the longest timespan (1990, 2001, 2007, 2012, and 2018). WVS data alone show that religious non-affiliation has decreased steadily for two decades from 1990 to 2010, and the trend after is less clear given close estimates in 2012 (non-affiliation: 85.2%) and 2018 (non-affiliation: 86.8%). CFPS estimates have remarkable changes over time. The 2014 estimate of non-affiliation dropped to 80.7% and the 2018 estimate plunged to 60.1% in 2018, while the estimates in the previous waves were 89.0% (CFPS 2012) and 85.6% (CFPS 2016). This is likely due to a modification of the question. The CFPS 2014 and 2018 asked a mix of belief in gods and religions rather than the religious affiliations. Specifically, CFPS 2014 found 16.9% of respondents believing in Buddha or Pusa, higher than the percentages of respondents reporting being affiliated with Buddhism in the adjacent waves (CFPS 2012:7.1%; 2016: 10.0%). CFPS 2018 offered the option of believing in the “immortals” and 19.7% of respondents reported doing so, while in 2014 the corresponding option was worded as “Taoist immortals” and only 1.1% of respondents selected it. The estimates from the other surveys, despite demonstrating much less turbulence than the CFPS, still show some fluctuations across waves. Table 1 also shows that the item missing rates of the religious affiliation questions were generally low for most of the surveys (around or less than 1%).

Demographic distribution

The sex distributions are similar, around 50% for each sex, across different surveys, except for the 1990 WVS (female = 40%, SE = 0.015). There are some variations in respondents’ distributions on age and education across different survey programs. This is largely due to different age inclusion criteria (CFPS: 16 and older; CGSS and CSS: 16–69; CLDS: 15–64; WVS: 18 and older). See Supplementary Appendix B for respondent demographic distributions in each survey along with the data from the last two Decennial Censuses.

Question design variations

Table 1 also provides a summary of how the religion questions differed given the five design features. A filter question was only used in the early waves of WVS (1990, 2001, and 2007). Most of the surveys did not use a filter but instead included “no religion” as one of the response options together with a list of religions or beliefs. Among those surveys, six surveys placed “no religion” as the first in the list while the other 12 surveys put it toward the end. The number of response options varied from 7 to 12, with variations occurring mostly between survey programs. CFPS 2014 and 2018 asked about beliefs in a mix of gods and religions with the rest explicitly asking about religions (separate questions for each belief in CFPS 2018). About one-third of the surveys allowed respondents to select more than one belief or religion.

Estimating question design effects

We use a series of logistic regression models to evaluate the effects of different question design features on the likelihood of reporting of non-affiliation (see the regression results in Table 2). We omit the two surveys measuring beliefs (CFPS 2014 and 2018) from these models because they do not measure the same concept (i.e., affiliation) as the other surveys. Notably, CFPS’s 2018 estimate is far lower than the others and including it in the models may substantially influence the trend of non-affiliation.

Model 1 has only survey year as the independent variable and it shows an overall significant, yet small negative effect of year on non-affiliation since 1990 (logit coeff. = −0.020, OR = 0.98, SE = 0.002, p < .001). In Model 2, we add respondent demographics (age, sex, and education levels) and dummy variables representing each survey program, without any survey design variables. Note that Model 2 does not contain WVS 1990 because it measures education differently compared to the rest of the surveys. Therefore, Model 2 essentially measures the changes in non-affiliation since 2001 (the next oldest survey in our analysis). The estimated effect of year (logit coeff. = −0.018, OR = 0.98, SE = 0.002, p < .001) on non-affiliation in Model 2 is similar to that in Model 1.

In Model 3, we add the first survey design variable: whether a filter question precedes the question on specific religious affiliations. The model shows that using a filter question has a significant positive effect on the likelihood of reporting non-affiliation (logit coeff. = 0.427, OR = 1.53, SE = 0.079, p< .001). Although still significant, the effect of year is reduced (logit coeff. = −0.013, OR = 0.99, SE = 0.002, p< .001).

In Model 4, we evaluate the effects of the other two design features: presenting “no religion” first (vs. toward the end), and allowing respondents to choose multiple responses (vs. single choice). In this model, we restrict the analysis to the surveys without a filter question given that respondents who choose “no” for the filter question (and therefore counted as non-affiliated) would not even be presented the subsequent question with these design features. Unfortunately, we are unable to model the effect of the number of options because it is collinear with the dummy variables for the survey programs (as the number of options is mostly the same within, but different between survey programs.) The surveys included in Model 4 range from 2006 to 2018.

Model 4 shows a significant positive effect of offering “no religion” first (logit coeff. = 0.101, OR = 1.11, SE = 0.031, p= .001) and a significant negative effect of allowing multiple choices (logit coeff. = −0.169, OR = 0.84, SE = 0.022, p< .001) on the report of non-affiliation. In addition, Model 4 shows the effect of year on non-affiliation is no longer significant since 2006 (logit coeff. = −0.002, OR = 1.00, SE = 0.003, p = .459).

The regression coefficients of the demographic variables in Model 1–4 generally show the same pattern. More education is associated with more non-affiliation. The interaction effects between sex and age suggest men are more likely to be unaffiliated and such differences tend to be larger for older age groups.

Overall, the effect of year becomes smaller as survey design features are added to the model. In particular, the analysis of 17 surveys (excluding three WVS surveys that used a filter and the two CFPS surveys not measuring affiliations explicitly) shows that the effect of year is not significant after 2006 with survey design features and demographic variables controlled.

Discussion

This study attempts to address two questions on religious non-affiliation in China with the responses from 22 domestic and international probability-based surveys since 1990. First, how do different question designs affect reports of non-affiliation? Second, what is the trend of religious non-affiliation in China after taking those question design differences into account?

Question design choices do clearly influence reports of non-affiliation and may add significant error to estimates. Compared to directly measuring affiliations, asking respondents about beliefs in a mix of gods and religions was associated with lower estimates of non-affiliation. Yet, operationalizing religious affiliation with questions about belief is a mismatch between concept and measurement, significantly contributing to survey error. Moreover, a measure of belief is not equivalent for comparison with estimates of religious affiliation from other surveys and using it as such may lead to invalid conclusions about religious change.

For the questions that explicitly asked about religious affiliation, we found that presenting a filter question before asking affiliation with specific religions and listing “no religion” as the first response option were both associated with more reports of non-affiliation. Conversely, allowing respondents to choose multiple religions in a mark-all-that-apply question was associated with fewer reports of non-affiliation. These question design effects are consistent with our hypotheses, reflecting the general survey response process and not unique to Chinese respondents.

It is worth noting that these design effects are estimated from comparisons across the existing surveys. Therefore, we cannot fully exclude the possibility that other unidentified variations across surveys might potentially confound the findings. For example, the observed effect of a filter question might be partly due to that the filter question was only used in the earlier waves of WVS (1990, 2000, and 2007) and the changes in non-affiliation might be larger in these years, thus confounding the effect of the question design. Given such limitations, we encourage more experimental research to formally assess the impacts of various design factors.

What do these data say about temporal changes of religious affiliation in China? Taken at face value, these survey estimates show a steady decline in religious non-affiliation during the 1990s and 2000s, albeit with considerable fluctuation in more recent years as more surveys included measures of religious affiliation. However, when we account for differences between the design of survey questions, the effect of year is reduced and represents a nearly flat trend since 2006. Notably, even with the statistical power that comes with pooling together approximately 200,000 responses, the changes in non-affiliation during past 15 years are not significant. Thus, once the idiosyncratic design features of each survey are taken into consideration, no evidence for an increase in religious affiliation, or a larger trend of sacralization can be found in these data from the recent years.

The absence of evidence for a clear trend on religious non-affiliation in China does not exclude the possibility of other changes in the religious lives of the Chinese population. This study only analyzes religious non-affiliation and it is possible that other measures of religion (e.g., behavioral or psychological measures) may reveal a different story. Notably, our findings are largely based on the reports of Han Chinese as we do not have adequate sample sizes to report estimates for individual ethnic groups.

Overall, our study illustrates that religion in general, and in the Chinese context in particular, is a complex concept and a difficult one to measure well. Even subtle differences in survey design, such as the wording and ordering of questions and response options, can affect responses. To better understand religion in China and elsewhere, we need to better understand how various questions measuring religion are interpreted and ensure that measures within and across survey organizations are equivalent and harmonizable.

Supplementary material

Supplementary Data are available at IJPOR online.

Conflicts of interest: None declared.

Footnotes

References

Chan Zhang is a researcher at the College of Media and International Culture at Zhejiang University, Hangzhou, China. Her current research interests include interactive designs of web surveys and survey methods in developing countries and societies in transition.

Philip S. Brenner is Associate Professor of Sociology and Senior Research Fellow in the Center for Survey Research at the University of Massachusetts Boston, where he also serves as Director of the Graduate Certificate Program in Survey Research. His research examines social desirability bias and other errors in interviewer- and self-administered surveys.

Lirui He is a survey methodologist at the Survey Data Center, Institute for Economic and Social Research, Jinan University, Guangzhou, China.