Perceived legitimacy of agricultural support schemes: An investigation using factorial survey experiments

Abstract

Agricultural support payments are a significant position in public budgets, and the legitimacy of such payments is subject to continuing debate. The legitimacy rests on the social acceptance of citizens for support payments to farmers, which is the focus of this study. Social acceptance is investigated using evaluations of farm and farmer descriptions in a factorial survey experiment. The results reveal higher acceptance of payments for farms demonstrating enhanced animal welfare, biodiversity, and a lower carbon footprint. The acceptance of support payments is negatively associated with payment amount, but payments to farmers who are financially struggling are more accepted than payments to profitable farmers; indicating respondent preferences that align with the need justice principle. Study findings can be used to inform priorities for legitimate policies of agricultural support schemes, to identify areas of consensus or disagreement regarding social acceptance of support, and to facilitate effective communication on agricultural support policy.

1. Introduction

The profitability of farming can be heavily dependent on subsidies. Globally, |${\$}$|245 billion per year is spent on fiscal subsidies to agricultural producers (Food and Agriculture Organisation of the United Nations (FAO) et al. 2021); just over half of these subsidies (55 per cent) are paid in Organisation for Economic Co-operation and Development (OECD) countries. In the European Union (EU), the primary mechanism administrating subsidy payments―the Common Agricultural Policy (CAP)―accounts for |${\$}$|58 billion, 37 per cent of the total EU budget (Pe'er and Lakner 2020). In the UK, total direct CAP payments in 2018 accounted for 12 per cent of gross revenue from farming, or 70 per cent of total income from farming (gross revenue less taxes and production costs); this rises to 80 per cent in Scotland alone (Coe 2020). The magnitude of subsidies and associated opportunity costs calls for an ongoing need to scrutinize spending of public money on the agricultural sector. Consequently, the design and implementation of agricultural policy that governs subsidy payments has been subject to much political and academic debate over previous decades in many regions of the world (e.g. Chatellier and Guyomard 2023).

Recognising the interlinked opportunities for sustainable development arising from changes in agricultural support systems, FAO, United Nations Development Program (UNDP) and United Nations Environment Program (UNEP) call for reconfiguring agricultural producer support across the globe to enhance the efficiency, sustainability, and equity of food systems that support healthy lives, nature, and climate (FAO et al. 2021). In the EU, justifications for CAP payments have undergone a paradigm shift from a post-war focus on rewarding production for greater levels of self-sufficiency via price support mechanisms, to an emphasis on direct income support conditional on cross-compliance rules that include positive environmental practices, such as a ban on cutting hedges and trees during the bird breeding and rearing season to protect biodiversity and ensure the retention of landscape features (European Council 2023). This is in line with an economic perspective that public goods are undersupplied in the absence of appropriate price signals, justifying government intervention. The recent greening of the CAP and England's post-Brexit policy to ‘paying public money for public goods’ broadly follow this economic rationale (Defra 2020; European Commission 2020). Such changes in rhetoric are clearly meant to align the normative basis for agricultural subsidies with values held by the public that concern the delivery of rural development objectives and environmental benefits. Indeed, a considerable evidence base exists that demonstrates demand for improved provision of (environmental) public goods and rural development objectives in agricultural landscapes.

A large body of previous research has focused on public demand for outcomes of agri-environmental schemes and programmes, often using stated preference methods. For example, Colombo et al. (2005) investigate preferences of Andalusian citizens to reduce soil erosion, and Grammatikopoulou et al. (2020) derive Czech citizens’ willingness to pay (WTP) for food, soil, water climate, and biodiversity outcomes of agricultural policy. More recently, Tienhaara et al. (2020) estimate WTP of Finnish citizens for ecosystem services provided in agricultural environments, and Li and Ando (2023) estimate demand for Tallgrass Prairie restoration in the USA. Schaak and Musshoff (2020) and Häfner et al. (2018) both study visual landscape preferences for pastures and a mixed use agricultural region in Germany, respectively. Both studies find preferences for point elements such as trees in the landscape, and Schaak and Musshoff (2020) find a preference for a pasture protection programme. Taken together, such studies demonstrate demand of members of the public for enhanced provision of environmental outcomes such as enhanced climate change mitigation or biodiversity from changes in land use and land management. Other studies focus on the consumer perspective to study price premia for attributes related to agricultural production systems in food products. In a meta-analysis of food preference studies, for example, Yang and Renwick (2019) find that consumers prefer livestock products that indicates greater animal welfare and are certified to stem from organic production. Considering such findings, we expect that a greater prospect of delivering environmental outcomes would be associated with an increase in the acceptance of agricultural income support by members of the public.

Often in conjunction with public demand for environmental outcomes, a number of studies investigate impacts of proposed agri-environmental programmes on rural communities. In a choice experiment study, Bennett et al. (2004) include an attribute reflecting the number of farmers in a rural area. Attributes operationalised as employment in rural areas can serve as a proxy for income generation in rural areas and have been included in Glenk and Colombo (2011) and Adamowicz et al. (1998). Rural development is one of the attributes in Moran et al. (2007), operationalised as maintenance of farming communities and promotion of locally grown food. Preferences for such attributes may differ depending on whether respondents live in urban or rural communities. For urban residents, preferences for rural development attributes could be rooted in values associated with current (recreational) use and option values regarding the maintenance of cultural landscapes and maintaining vibrant rural communities as an asset to be enjoyed on future visits to the countryside. Rural residents may, for example, care about direct implications of living in a vibrant community. Overall, the empirical findings suggest that respondents prefer maintaining or enhancing rural employment and maintaining vibrant rural communities. This may point to general acceptance of agricultural income support for farmers and especially payments to farmers who struggle to make a living in the absence of support payments.

In stark contrast to policy ambition, and to public demand for environmental and social outcomes of agricultural policy, is ample scientific evidence of limited effectiveness of subsidy payments to achieve stated environmental and social objectives (Pe'er et al. 2014; Pe'er and Lakner 2020; Scown et al. 2020; Scown and Nicholas 2020; Biffi et al. 2021; FAO et al. 2021; Springmann and Freund 2022; Hasler et al. 2022). In other words, there is a mismatch between the currently dominant justification for agricultural subsidies based on environmental and social outcomes and their ability to deliver. This mismatch creates potential for weakening public support for agricultural subsidies. The aim of this paper is to provide new empirical evidence on the policy legitimacy of agricultural income support. The focus of our study shifts from quantifying socially desirable quantities of public goods and rural development objectives to social acceptance of payments made to farmers conditional on farm and farmer characteristics. Further, our empirical assessment concerns basic income support payments to farmers as opposed to specific agri-environmental programmes that target selected environmental outcomes.

Legitimacy has previously been used as a lens through which to evaluate the delivery of agricultural transitions and schemes and is often understood through two overlapping conceptualizations (de Boon et al. 2022). A normative conceptualization develops and uses criteria to appraise how a system should function to deliver outcomes. In an empirical conceptualization, next to other dimensions legitimacy critically depends on the beliefs and perceptions that people hold regarding governance structures, including policy (see Weber 2019 [1922]; Beetham 2012). Normative and empirical concepts are related: A higher degree of normative legitimacy is often associated with or comprises greater public support, i.e. ‘socially accepted beliefs’ (Beetham 2012, 123). de Boon et al. (2022) use a combination of empirical and normative conceptualizations and describe three dimensions of legitimate transitions in agricultural transitions; input, output, and throughput. The authors stress that social acceptance or perceptions of legitimacy are critical to increase clarity, diversity, and meaningful inclusion in the policymaking process. However, they do not investigate different dimensions of legitimacy per se, but stages of policy formulation at which legitimacy should be considered. This notion is similar to Jagers et al. (2016), who found that failure to legitimise policy at any stage of policymaking and implementation risks lingering perceptions of illegitimacy at subsequent stages. In this paper, we use an empirical conceptualization of perceived legitimacy and investigate different dimensions of social acceptance and support for agricultural subsidies.

Towards this end, we employ a novel survey-based approach that centres around acceptance of payments to individual farmers conditional on farm and farmer characteristics. Methodologically, we use factorial survey experiments which, to our knowledge, have not been widely used in an agricultural policy context. In our application, respondents were asked to make judgements concerning the social acceptance of payments to farmers following a description of the farmer as a person, and the farm as an individual business entity. In factorial survey experiments, information shown to respondents is experimentally controlled. This means that we can directly measure the effect of farm and farmer characteristics on judgments made. We argue that the clear attribution of effects provides strong evidence for the degree of ex ante policy legitimacy, which ideally aligns strategies, objectives, and motivations regarding policy with values and beliefs held by the public (Matti 2009; Christensen et al. 2020).

We apply our approach to the case study of Scotland, UK. Current agricultural support represents a significant contribution of the Scottish taxpayer. About two thirds of the total profit from farming in Scotland has been coming from support payments in recent years. In 2019, approximately £0.5 billion worth of support payments were made (RESAS 2020). On average, this amounts to £200 per year and Scottish household, and £90 per year and capita. It thus seems important and timely to re-establish the legitimacy of the basis for agricultural support funding in Scotland through understanding public acceptance of such payments.

Scotland faces important decisions on the structure and model of future support for agriculture following the exit of the UK from the EU. An Agriculture Reform Implementation Oversight Board (ARIOB) is considering options for future support models, with a Scottish Government commitment to bring forward a new Agricultural Bill to replace the Common Agricultural Policy by the end of 2024. Support is anticipated to be organised around four tiers (Scottish Government 2022a). Tier 1 and 2 are direct payments made for cross compliance with (environmental) production standards (Tier 1), with additional payments for enhanced action related to, for example, reducing greenhouse gas emissions and enhancement of habitat to support local biodiversity (Tier 2). Tier 3 includes elective payments to support further engagement towards environmental targets, support for alternative forms of production (organic), and, importantly, encouraging innovation, including the use of technology to enhance productivity and business resilience while reducing environmental impact. Finally, Tier 4 is aimed to support, for example, continuous professional development, farm advisory services, farming in naturally constrained areas, as well as funding environmental measures of those not receiving direct payments.

The paper proceeds as follows. In Section 2, we introduce our approach to measuring social acceptance of agricultural support payments, using four outcomes of interest in relation to varying farmer and farm characteristics. We continue in Section 3 with an introduction to the factorial survey experiment as a method, and its implementation in a survey on agricultural support with Scottish citizens. Section 4 presents summary results and models for each of the four social acceptance outcomes. We illustrate ways to meaningfully report and communicate model results, before discussing our findings both from a methodological and policy perspective.

2. Dimensions of social acceptance of agricultural subsidy payments

Our conceptual starting point is that the legitimacy of agricultural subsidy payments as a policy instrument includes ‘socially accepted beliefs’ about the rightfulness of rules and ends of governance structures as well as policies (Beetham 2012, 123). On the other hand, if rules are ‘only weakly supported by societal beliefs’ (Beetham 2012, 123), there is a ‘legitimacy deficit’ or ‘illegitimacy’. To further examine ‘socially accepted beliefs’ and keeping in mind the challenges in defining and measuring social acceptance, we rely on a well-established conceptualisation of social acceptance that in its broadest form comprises socio-political acceptance, community acceptance, and market acceptance (see Wüstenhagen et al. 2007). This broad conceptualisation of acceptance implies that we not only look at the legitimacy of subsidy payments per se but also how they are designed. Our study refers to citizens’ acceptance of agricultural income support schemes for farmers and, hence, our outcomes of interest are related to socio-political acceptance by the public as well as market acceptance by consumers (Figure 1).

In our study, socio-political acceptance comprises the acceptance of payments to farmers under an agricultural support scheme, as well as the perceived fairness of payment amounts. It is important to note that acceptance and fairness are distinct concepts (Liebe and Dobers 2020; Bal et al. 2023) and refer, in our study, to different ‘objects’ of evaluation. Generally, any (or most) agricultural support schemes include some element of inequality in the sense that some farmers will receive more than others. Such inequalities in payments are, however, not necessarily equal to (perceived) injustice in payments. In other words: not all inequalities need to be perceived as unfair (e.g. if they are perceived as unavoidable).

Specifically in our study, we ask about the acceptance of changes in payments (first outcome) and to what extent the payment amounts are unfairly low or unfairly high (second outcome)¹. Both acceptance and fairness refer to perceived distributive justice (Jasso and Rossi 1977), given that we ask who is (not) entitled to or deserving of support scheme payments. As another component of socio-political acceptance, we consider an individual's intention to petition in support of payments for famers (third outcome), which is related to civic activism and protest behaviour for a cause that is perceived to be deserving (e.g. Opp 2009 for an overview). In the UK, (e-)petitions are commonly used by citizens to express concerns and the Petitions Committee of UK parliament considers petitions with over 10,000 signatures and reports every month on several petitions, often up to 20 concerned with various topics, with 10,000 and 100,000 signatures, respectively.² As in our empirical study (more below), we refer to the discontinuation of the agricultural support scheme, this outcome most closely refers to the legitimacy of the policy instrument per se.

The intention to be supplied by a farmer (fourth outcome), who is supported by an agriculture support scheme is a dimension of market acceptance. Increasing demand by consumers for produce generated under certain conditions is an important part of the diffusion of production standards, especially given that actual diffusion processes are complex (Rogers 1995; Meade and Islam 2006). We expect that market acceptance is primarily associated with farm and farmer characteristics that are perceived to impact product quality and utility of consumption. This might include aspects such as experience and qualification of the farmer that signal competence, but also so-called credence attributes, for example, animal welfare, organic production, or environmental footprint, that are not directly experienced or identified during product consumption (Moser et al. 2011; Yang and Renwick 2019).

In our study, agricultural support schemes are characterised by several farm and farmer characteristics (Fig. 1 and Table 1). These are our main explanatory factors in the light of our four social-acceptance outcomes of interest. Considering the persistence of gender pay gaps in many countries, including Scotland (e.g. Hamilton and Richmond 2017, 9 per cent for skilled agricultural traders), we explore whether female recipients of support scheme payments are less accepted than male recipients. This would mean, ceteris paribus, payments to female farmers are less accepted and perceived as unfairly high, as well as individuals having less intention to buy products from female farmers and to petition support for them. As we include many farm characteristics, especially productivity, statistical discrimination (Phelps 1972) could not explain potential gender differences. Rather, gender differences would most likely be due to status beliefs, i.e. a belief that men have higher competence and worthiness (Ridgeway 2001; Auspurg et al. 2017 on gender pay gaps).

Having a degree and bespoke agricultural education, and having longer experience in the agricultural sector signals competence (Spence 1973; Mincer 1974). On the one hand, higher competence of farmers is expected to lead to a more positive evaluation, compared with farmers who have less education and less experience. An expected positive effect related to competence is also related to the justice principle of merit (Miller 1999; Shamon and Duelmer 2014). On the other hand, lower experience may invoke a perceived need for additional support as another well-known social justice principle (e.g. Miller 1999).

While we do not expect a strong preference for a certain farm size per se, research indicates that individuals might prefer smaller over larger farms (e.g. Krystallis et al. 2012; Busch et al. 2022), as this is in line with an ‘idealised’ view of farming. The presence of viable small farms may also be perceived as an important element of vibrant rural communities. Regarding efficiency of production, individuals might be more willing to accept payments to farms with higher efficiency of production that can be seen as a signal of effort and following prominent notions of social justice such as Aristotle's (1985) rule of proportionality of rewards. It follows from the rule that payments should increase with effort (see also merit justice principle, e.g. Homans 1961; Shamon and Duelmer 2014). Additionally in our context, additional payments should be also perceived as fairer if productivity is higher.

All else equal, individuals are assumed to prefer organic over conventional farming; this can have several reasons such as a positive evaluation of health and other perceived benefits of organic farming (e.g. Lazaroiu et al. 2019). Attitudinal and stated preference data also suggest a strong social acceptance of farms with higher levels of biodiversity, higher animal welfare standards, and lower carbon footprints (e.g. Moran et al. 2007). In other words: Individuals prefer policies with positive effects on other species as well as the environment. The latter is especially important in the context of climate change mitigation. Regarding profitability, it can be expected that individuals might perceive payments to profitable farms less acceptable compared with non-profitable farms if all other characteristics are equal. If we assume, for example, the same level of productivity, type of production, environmental impact, and farmer characteristics, then payments to those in need might be rather justifiable compared to those who do well financially. This would be grounded in the justice principle of need (d'Anjou et al. 1995; Miller 1999); however, it might depend on the context whether the need-justice principle or another principle such as merit is (on average most) preferred by individuals (Reeskens and van Oorschot 2013).

3. Method

3.1 Factorial survey experiments as a tool to assess social acceptance

Factorial survey experiments were first developed in the 1950s and have been applied more widely since the 1980s, mainly in quantitative sociological research. In factorial surveys, respondents are asked to evaluate descriptions of hypothetical situations, individuals, or objects. The descriptions of hypothetical situations, individuals, or objects are called vignettes and often come in the form of text, but may be presented and summarised in alternative formats, including tables or audio and visual formats (see Treischl and Wolbring 2022 for an overview). While factorial survey experiments share many design principles with other types of multifactorial survey experiments such as stated choice experiments and conjoint experiments, as well as best-worst scaling, they differ regarding theoretical foundation, outcome measures of interest, as well as statistical analysis (for an overview, see e.g. Wallander 2009; Auspurg and Hinz 2015; Liebe and Meyerhoff 2021).

Factorial survey experiments rest on the idea that there are often multiple dimensions that influence a person's judgment. For example, a person's judgment of the fairness of wages paid to individuals has been found to depend on factors such as occupational prestige of the position evaluated, and age, gender and degree of training, and education of the job holder (Auspurg et al. 2017). Judgments in factorial surveys may be made about fairness, but may also relate to subjective beliefs, social norms, and attitudes.

Our study is among the first to successfully apply a factorial survey experiment in an agricultural context, but the field of potential applications is wide. From a landowner or farmer perspective, potential applications closely related to earlier applications in sociology include gender gaps with respect to pay for farm labour and with respect to access to (micro-)credit. Factorial survey experiments are also well suited to measure social norms and cultural norms in farming populations, and their impact on intention to adopt technologies, and their attitude towards interventions (Liebe et al. 2020). Liebe et al. (2017) use factorial surveys to assess the social acceptance of local wind energy expansion scenarios that included characteristics such as number of turbines, type of investor, the opportunity to participate in planning, and how tax revenue would be used. In a recent paper, Parkins et al. (2022) assess landowners’ acceptance of windfarms, with attributes covering governance dimensions such as ownership structure, inclusion criteria, and compensation of neighbours.

The multiple dimensions of hypothetical situations are often referred to as characteristics or attributes. In factorial surveys, the expressions that the attributes take vary across hypothetical situations that respondents are asked to evaluate. Crucially, variation follows an experimental design that ensures the influence an attribute has on the judgment can be robustly measured or identified. Such a design can be trivial if there are few attributes, but the complexity of the design increases considerably as the number of attributes that describe a hypothetical situation increases. To increase efficiency in data collection, respondents participating in factorial surveys are typically asked to evaluate several vignettes within a single survey.

Beliefs, attitudes, and norms may be measured using alternative means, for example, using psychometric scales based on Likert-scale type questions that ask respondents to indicate their position concerning an item on typically four- or five-point scales. Items may be statements (e.g. ‘Farmers are the backbone of rural communities’) and response scales are ordinal (e.g. Strongly Disagree—Somewhat Disagree—Somewhat Agree—Strongly Agree). There are several key advantages that factorial surveys have compared to asking (a series of) Likert-scale type questions. First, variation in the attributes of interest can be experimentally controlled. This allows to directly assess the causal effect of that attributes have on judgments (Manago and Mize 2022). Second, the evaluation of hypothetical situations takes place considering all attributes simultaneously—not single aspects in isolation. This can reduce measurement error, for example, due to social desirability bias (Auspurg et al. 2015; Liebig et al. 2015). Third, the factorial survey avoids a common finding for Likert-scale type questions where ‘everything matters’—indicated by a low degree of discrimination between the attributes of interest. This can render Likert-scale data rather non-informative (Liebe and Meyerhoff 2021). Fourth, the design of factorial survey experiments is very flexible and can accommodate a broad range of topics and response dimensions (see Liebe and Meyerhoff 2021 and Treischl and Wolbring 2022 for topical reviews). An alternative to direct evaluation of each vignette via response scales is, for example, ranking of sets of vignettes (Walter et al. 2023).

3.2 Study design

To investigate dimensions of acceptance of support payments to farmers, we designed an online survey for implementation with members of the Scottish public. The survey was structured in four sections. The first section provided background information on farming in Scotland followed by the six vignettes to be evaluated as part of the factorial survey experiment. We focus on the factorial survey experiment element in this paper. The second section investigated perceived priorities for a future agricultural support scheme in Scotland using best-worst scaling. The third section obtained information on behaviours and attitudes regarding food, the environment and farmers, and farm support. The fourth section asked further details about respondents and their households, including their political position (if any). On average, the survey took between 15 and 20 minutes to complete.

In terms of background information on farming in Scotland, respondents were shown an overview and description of main farming systems in Scotland (beef, dairy, sheep, and cropping). They were made aware of existence and amount of income support payments by government, and the fact that payments are conditional on compliance with basic environmental and production standards (cross-compliance). Respondents were informed that in recent years prior to the survey, about two-thirds of the total profit from farming in Scotland has been coming from support payments, and informed about total payments made in 2019, also presented in the form of payments per household and year. In the subsequent factorial survey experiment, respondents were asked to make judgments about hypothetical farmers and their farms. To ensure findings represent the agricultural sector in Scotland, respondents were randomly allocated to one of the four main production systems. In this paper, we consider systematic differences in evaluation between production systems at the level of the individual respondent.

Each farmer and each farm described in the vignettes was characterised by eleven attributes (Fig. 1 and Table 1). An additional attribute described changes in government support payments. To add to realism of vignettes, the absolute payment amounts (in GBP) shown on vignettes were determined as a function of farm size and type of production system (beef, dairy, sheep, and cropping), and descriptions of hypothetical farms and farmers in vignettes were adjusted depending on their allocated production system. A detailed list of attributes and attribute expressions used in the survey can be found in the Appendix (Table A1). Because the full factorial, i.e. the number of all possible attribute-level combinations is too large to be presented to survey respondents, we employed an orthogonal fold over experimental design to reduce the number of vignettes shown in the survey. This design comprised 72 combinations of attributes (i.e. it included 72 unique vignettes), which allow the unconfounded estimation of all main and two-way interaction effects. Additionally, perfect level balance holds; i.e. all attribute levels appear equally frequently across vignettes. For each respondent, six vignettes were drawn randomly without replacement from the set of seventy-two vignettes and presented to respondents.

Vignettes were textual descriptions of a hypothetical farmer and their farm. Figure 2 shows an example vignette for a sheep farm as shown to respondents in the online survey.

Each of the six vignettes presented to each respondent was shown for a minimum of 10 seconds before respondents could evaluate the hypothetical farm and farmer with respect to the four outcomes, all measured on an 11-point scale (Table 2), as recommended in the literature (Auspurg and Hinz 2015). The outcome variables cover different dimensions of social acceptance in relation to payments to farmers that overlap but are not congruent. As a result, we expect that evaluation scores for the social acceptance outcomes will be correlated but differ across outcomes, which will show in differences in statistically significant influence that attributes have on evaluations. A screenshot of response scales as shown in the online survey for the example vignette in Fig. 2 can be seen in the Appendix, Figure A1.

3.3 Survey implementation and sample

The survey received ethical approval of the [Institution omitted for review] Social Science Ethics Committee. The online survey was administered in January and February 2022 by a professional market research company to a sample of 2,011 Scottish adults based on a quota sampling approach with hard quotas set for age and gender. Consequently, our sample compared well to population statistics with respect to age and gender (Table 3), meaning there was no significant difference in the composition of these two populations with respect to gender (X² = 0.957, df = 1, P = 0.328) or age (X² = 7.563, df = 4, P = 0.101).

Quotas based on an access panel, as was the case with our sample, are a practical and more affordable option but are not as representative as a significantly more expensive probability-based sample. Therefore, we found a significant difference in the composition of educational achievements between respondents in our sample and in the 2011 Scottish Census (X² = 0.957, df = 4, P < 0.001), with Pearson's residuals indicating that our sample slightly overrepresented respondents with higher levels of education and underrepresented respondents with no qualifications. However, it is worth mentioning that the differences in education are considerably less pronounced if we compare our outcome to probability-based samples of the Scottish population such as the Scottish Household Survey (Scottish Government 2020). Randomisation worked well with respect to the allocated production system (beef: N = 504; dairy: N = 503; sheep: N = 501; cropping: N = 503), with no significant difference in the composition of respondent characteristics allocated to each version with respect to gender (X² = 0.027, df = 3, P = 0.999), age (X² = 0.424, df = 12, P > 0.999) or education level (X² = 3.321, df = 12, P = 0.993).

3.4 Econometric approach

Data analysis of factorial survey experiments typically employs regression modelling, where judgements of vignettes serve as dependent variables that are explained by attributes. Additionally, regressions can consider whether there are systematic relationships between respondent characteristics and judgments.

Here, we employ a random intercept linear regression model taking the multilevel structure of the vignette data into account (i.e. that each respondent answered multiple vignettes, and that vignettes are nested within respondents). This model can be expressed as follows:

|${{Y}_{ij}}$| refers to the vignette evaluation for vignette i (with |$i = 1,\ \ldots 6$|⁠) by respondent |$j\ $|(with |$j = 1,\ \ldots N,\ $|the number of respondents). We estimate |$\beta $| coefficients for the attributes V at the vignette level and |$\gamma $| coefficients for respondent characteristics P at the respondent level (⁠|$p\ $|and q refer to the number of vignette attributes and person characteristics, respectively). The random intercept model includes an additional error term |${{u}_{0j}}$| to capture between-respondent variation in vignette evaluations, i.e. |${{\beta }_0} + {{u}_{0j}}$|⁠. We estimate one model for each of our four outcome variables including all vignette attributes and selected respondent characteristics such as gender, age, and education. For one of the outcome variables—acceptance of changes in payments—we estimate the model above but include interaction effects between the vignette attribute ‘payment’ and all other vignette attributes. This model shows to what extent payments are more or less accepted if other vignette attribute levels are present; for example, whether higher payments for more profitable farms are more or less accepted compared with payments to less profitable farms.

Furthermore, for the fairness outcome, we estimate ‘payment offsets’ indicating payments that would be required to obtain a certain vignette characteristic while holding perceived fairness constant. For each non-payment vignette attribute (⁠|$nonpay$|⁠) the offset value can be obtained by the ratio |${{\beta }_{\textit{nonpay}}}/{{\beta }_{pay}}$|⁠, i.e. the coefficient for the non-payment attribute is divided by the coefficient of the payment attribute (⁠|$pay$|⁠). Such offset values can also be calculated to compare vignette profiles, in our case farmers with several different vignette characteristics. Corresponding offset values involve, based on the estimated coefficients of the vignette attributes at hand, the calculation of the difference in predicted evaluations between two vignette profiles, and dividing this difference by the coefficient capturing changes in payments (⁠|${{\beta }_{pay}})$|⁠. This can be understood as the payment differential that is required to perceive income support to two farmers as equally fair.

4. Results

4.1 Distributions of ratings for evaluated outcomes of interest

Distributions of evaluation scores across the four outcomes of interest (acceptance of changes, fairness, intention to consume, and intention to petition) are shown in Fig. 3. Respondents utilised the whole range of the 11-point evaluation scales across all four outcomes. This is a desirable feature of factorial survey experiments, showing that respondents clearly discriminated between different vignettes.

We not only expected overlap between responses to the four outcomes of interest, but also differences given that they represent distinct dimensions of social acceptance in relation to support payments made to farmers. Solely based on visual inspection of Fig. 3, distributions appear similar, with a notably greater right skew for intention to consume produce supplied by a farmer. However, a statistical investigation into bivariate correlations of all combinations of the four social acceptance outcomes supports our expectation. We find significant positive correlations at the 1 per cent level for all combinations. However, the strength of association is weak to moderate, ranging from 0.14 for perceived fairness and intention to consume to 0.4 for intention to consume and intention to petition. We thus expect to see differences in relative influence of attributes on evaluations in the regression models.

4.2 Model results

Results of random intercept regression models are shown in Table 4 for all four outcomes of interest. All variables entering the models and their summary statistics are shown in the Appendix, Table A2. The top panel of the table shows attribute coefficients. They directly reflect the influence of the presence of attribute levels in vignettes on evaluation scores. For the acceptance of changes in payments evaluation only, we report two-way attribute interactions of all attributes with changes in support payments (except for the change in payment itself, reported as a main effect). Positive (negative) and significant coefficients in the acceptance model indicate that an increase (a decrease) in payments is more acceptable if an attribute is present in a vignette. The bottom panel of the table shows respondent level variables. These are shifters of the constant. Therefore, related coefficients indicate whether respondents have on average a systematic tendency for lower or higher evaluation scores. The sum of respondent level variables at sample means and the constant provides the average evaluation score if all vignette attributes take a value of 0. The number of respondents entering the models is reduced by sixty respondents with missing data on perceived financial status or level of environmental activism. It is further reduced for the model of intention to consume by 190 respondents who report to not normally consume the produce of the farm types described in the vignettes.

Across the models, the amount of variance explained at the level of respondents and at the level of vignettes varies. For example, for a constant only model, intra-class correlation is 0.193 for acceptance of changes in payments. This implies that 19.3 per cent of variance is explained at the level of respondents and 80.7 per cent at the level of vignettes. For intention to petition, 65.8 per cent of variance is explained at the level of respondents and 34.2 per cent at the level of vignettes. Jointly, the coefficients are significant at the 1 per cent level for all models based on the χ² test statistic. Starting with acceptance of changes in payments, we find that an increase in income support payments is more acceptable (and a decrease less acceptable) if farms are smaller, have a lower carbon footprint per unit of output, and if they are not profitable at present. An increase in payments is also more acceptable (and a decrease less acceptable) if farms are more efficient in production, excel in animal welfare performance (beef, dairy or sheep farms) or product quality (cropping farms), and if they offer better conditions for biodiversity. Among farmer characteristics, respondents are more accepting of an increase in payments for both more experienced farmers (20 years) and less experienced farmers (5 years) relative to farmers with moderately long experience (10 years). Indicated by a negative and significant coefficient for Change in Payment, respondents are less accepting of increasing payments (more accepting of decreasing payments) if all attributes are at their reference levels. In the model of acceptance of changes in payments, this reflects the least acceptable combination of farm and farmer attributes.

For evaluating the fairness of income support payments, we reversed the scale for analysis, such that higher scores reflect a perception of unfairly low payments. Like acceptance of changes in payments, smaller farm size, not making a profit and lower carbon footprint of the farm and per unit of produce have significant positive influence on perceiving payments as unfairly low. This also applies to better conditions for biodiversity and better animal welfare performance or product quality. As expected for evaluation of fairness of payments, greater increase in payments has a negative influence on fairness perceptions, ceteris paribus. Payments are perceived to be unfairly low for organic farms than for conventional farms, but the effect size is small compared to other attributes.

The third outcome of interest captures willingness to lobby with local politicians on behalf of a farmer. A clear penalty in evaluation applies if a farmer has been described to have a high carbon footprint per unit produced as well as a high carbon footprint for the whole farm. Larger farms affect evaluations negatively, although the influence is less strong than for fairness and acceptance perceptions. Better animal welfare performance or product quality, better biodiversity conditions, above average production efficiency, and organic production all have a significant influence on intention to petition on behalf of farmers described in the vignettes. It is also worth pointing out that, unlike for acceptance of changes in payments and fairness of payments, farmer characteristics have significant influence. Greater experience of a farmer and having an agricultural degree both increase intention to petition. In addition, we see a relatively strong effect of financial situation and changes to income support payments. Relative to farms that make a loss, intention to petition with members of the Scottish parliament decreases if described farms are coping or making a profit. A greater decrease (increase) in payments has a positive (negative) effect on intention to petition but not on intention to be supplied.

The fourth outcome concerns intention to be supplied with produce by a farmer described in the vignettes. Responses to this outcome have been influenced by a similar set of attributes as for intention to petition. Attributes that relate to production and environmental performance have a great influence, and there is a relatively small penalty to smaller farms compared to larger ones. Again, farmer characteristics play a role in evaluations. Greater experience (20 years rather than 10 years) and having an agricultural degree both increase intention to be supplied with produce. Different to intention to petition, financial attributes, including the farm's financial position and the level of change to income, support payments do not affect evaluations.

We now turn to respondent level variables. First, acceptance and fairness perceptions related to payments are not systematically and significantly affected by whether a respondent saw vignettes of either beef, dairy, lamb or cropping farms. We do, however, see a significant negative effect of respondents seeing vignettes of livestock farms relative to cropping farms regarding intention to be supplied with produce. This might reflect a greater scrutiny that consumers apply to being supplied with meat products as opposed to staple food items such as cereals and potatoes. There is also a significant and negative effect on intention to petition with the relevant local member of parliament if the petition is on behalf of a beef farmer rather than a cropping farmer. This might be related to beliefs that the beef industry is a major contributor to environmental problems, especially climate change, making it less likely that a respondent would stand up for the cause of a farmer.

With respect to socio-economic characteristics, we find U-shaped relationships of age on evaluations for fairness perceptions and intention to petition. Being female (rather than male) has a significant negative influence on acceptance of changes in payments evaluations, but is not significant in regressions for other social acceptance outcomes. Further, lower (rather than higher) levels of education are associated with a tendency to judge income support payments as unfairly low. Living in a rural area has no effect on evaluations. Having children in the household increases the acceptance of changes in payments, intention to be supplied and the likelihood to lobby on behalf of a farmer, but does not affect fairness perceptions. The more difficult respondents find it to manage financially, the lower is acceptance of no change in payments, and the more pronounced are perceptions of payments as unfairly low. With respect to political orientations, having voted to leave the EU has only a positive effect on acceptance of changes in payments, intention to be supplied and intention to petition. Relative to having no affinity for any political party, there is an overall small impact of stated proximity to a political party. Notable exceptions are affinity to the Conservatives increasing acceptance of changes (including the possibility of a decrease in income support payments), affinity to Green party decreasing intention to be supplied by a farmer, and affinity to Liberal Democrats increasing intention to petition. Finally, an indicator of environmental activism based on participating in the environmental movement through petition, demonstration or membership to environmental groups has a significant positive effect on acceptance of changes in payments and intention to petition, and decreases perceptions that payments are unfairly low. In summary, there is no consistent pattern across social acceptance outcomes regarding effects of individuals’ characteristics on vignette evaluations, but differences in effects illustrate that, on average, respondents clearly discriminated between the four social acceptance outcomes.

4.3 Illustrative summary of results

The results presented in Table 4 can be further utilised to demonstrate how the average respondent perceived differences in farmers described in the vignettes. To do so, we predict scores and analyse difference in ratings for three stylized farmers with farm and farmer characteristics as in Table 5. The characteristics are purposefully distributed such that Farmer 1 will be seen more critically and Farmer 3 more approvingly by the average respondent given the estimated coefficients, with Farmer 2 somewhere in between. Based on attribute effects described in Section 4.2, this implies amongst other things that Farmer 1 runs a large, conventional farm with low efficiency and comparably poor environmental performance, which nevertheless makes a profit. In contrast, Farmer 3 manages a small organic farm that is efficient and exhibits excellent environmental performance despite making a loss. Equivalents for all three stylized farmers could be realistically found in Scotland. For example, for Farmer 3, given the market position for organic produce in Scotland, it is plausible they are making a loss, which may also be due other reasons, for example, interest payments on loans that were used for investment.

Estimates of predicted scores for the four social acceptance outcomes are reported in Table 5. Acceptance of changes in payments is reported for a decrease of 10, 25, and 50 per cent, no change and an increase of 10, 25, and 50 per cent in payments. For the remaining three outcomes of interest, we predict scores setting the level of changes in payments to 0 per cent (no change). Overall, the results clearly demonstrate the discriminatory power of the models and the cumulative effect of attributes on evaluation scores. There is a clear acceptance of a decrease in income support for Farmer 1 and clear acceptance for an increase of payments for Farmer 3. Compared to a scenario of no changes in payments, a 10, 25, and 50 per cent increase/decrease in payments results in a significantly lower/higher acceptance score for Farmer 1 at the 5 per cent level, and vice versa for Farmer 3. Respondents are on neither particularly accepting nor dismissive of changes in payments for Farmer 2. Payments to Farmer 3 are perceived as unfairly low and neither unfairly low nor high for Farmer 2, but unfairly high for Farmer 1. Respondents are relatively indifferent to being supplied by Farmer 1 with produce—indicating that respondents are more accepting in general concerning farmers’ core role as food producers. This changes again for intention to petition on behalf of farmers, where there is reluctance to act on behalf of Farmer 1 and clear intention to support Farmer 3.

For fairness of payments, an intuitive interpretation of model results can be accessed by estimating the percentage change in payments required to offset the difference in predicted scores between stylized farmers. This can be understood as the payment differential that is required to perceive income support to two farmers as equally fair. Such payment offset estimates can be calculated by taking the difference in predicted evaluations between two stylized farmers, and dividing the difference by the coefficient capturing changes in payments. Comparing Farmer 1 and Farmer 2 as defined in Table 5, an increase in payment of 109 per cent [95 per cent confidence interval: 95–123 per cent] would have to be added to Farmer 2 for payments to both to be seen as equally fair. To illustrate using the average payment amounts for different farm types used in the survey (Appendix, Table A1), a moderately sized beef farm receiving currently £20k in income support would have to see an increase to £42k for payments to be considered equally fair compared to a large beef farm currently receiving £60k. A comparison of Farmer 2 with Farmer 3 (as in Table 5) suggests that to be perceived as equally fair, Farmer 3’s income support would have to increase by 43 per cent [29 –57 per cent].

For the fairness evaluation, we can also express effect size of attributes in terms of payment offsets required to keep the fairness score unchanged if a farm or farmer characteristic is present in the vignette, shown in Table 6 and Fig. 4. These are cross-elasticities between vignette dimensions (Auspurg and Hinz 2015, 99). For example, a large farm's income support would have to be reduced by 56 per cent to maintain the fairness score of an otherwise equivalent small farm. Regarding biodiversity, support payments of a farm that provides good conditions for wildlife would have to be increased by 23 per cent for payments to be perceived as fair as for a farm with poor conditions for wildlife.

5. Discussion and conclusions

Agricultural subsidies, including payments to farmers, represent a significant position in public budgets. As agricultural policy evolves over time, the legitimacy of subsidy payments must be re-established. In this paper, we demonstrate how factorial survey experiments can be a tool to generate empirical evidence on the social acceptance of support payments and its influencing factors. When designing our study, we were confronted with scepticism by policy analysts about the ‘literacy’ of respondents needed to make meaningful evaluations. The high discriminatory power of the four outcomes and low to moderate correlations among the four social acceptance outcomes studied serve as evidence that, across our sample, such concerns are not justified. Respondents’ evaluations were sensitive to relative complex concepts such as levels of efficiency of production, described as greater or smaller amount of meat per animal or higher or lower crop yield per area, carbon footprint on the farm and carbon intensity (i.e. carbon footprint per unit of output).

Only few of the farmer characteristics have a significant effect on social acceptance. On average, respondents do not differentiate between male and female farmers. Our evidence does not point to effects that would be consistent with a gender wage gap or gender-based status beliefs (Auspurg et al. 2017; Hamilton and Richmond 2017), and likely implied assignment of greater competency to male farmers. Greater competency as indicated by greater experience and an agricultural qualification enhance evaluations of market acceptance. Effects on market acceptance might be due to a perceived correlation between respondents’ competency and quality and safety of produce. In line with justice concerns that propose proportionality between effort and reward (e.g. Homans 1961; Shamon and Duelmer 2014), we found that higher payments for farmers are more accepted if farms are more productive. Notably, the productivity effects are smaller and weaker for the intention to petition and to be supplied, and especially for fairness of payments. Since individuals evaluate larger farms much more negatively than smaller ones, our findings support an ‘idealised view’ of farming in favour of smaller farms. Overall, individuals in our study have a strong preference for policies with positive effects on animal welfare, biodiversity, and carbon footprint reduction. For the latter, the footprint per output unit matters much more than the footprint per farm. Organic production standards are valued, but not as highly as the above listed environmental and animal welfare effects. Regarding the farmers’ financial situation, respondents express preferences strongly in line with the need justice principle, i.e. payments to farmers who are struggle financially are more accepted than payments to better-off farmers (Miller 1999).

In what is, to our knowledge, a novel contribution to applications of factorial survey experiments, we illustrate simple ways to make the regression results more accessible to readers and, as we believe, more useful to policy makers. One approach shows how social acceptance scores based on estimated regressions vary across a set of stylized farmers and farms, and across social acceptance outcomes. This can facilitate an understanding of the relative magnitude of the effects of the farm and farmer characteristics on social acceptance, for example, showing the relatively small effect that production type had on evaluations. The approach also facilitates an understanding of relative acceptability. In our scales, a score of six represents a ‘neutral’ evaluation. On average, our results suggest that respondents are relatively accepting of being supplied by all three stylized farmers—including a large, conventional farm that does not perform particularly well with respect to efficiency, maintains only a basic level of animal welfare or product standard, and has a comparatively low level of environmental performance. Arguably, it is more difficult to obtain such insights if the communication of results focuses on the marginal effects of individual attributes rather than on predicted summary scores.

A number of limitations and caveats regarding our study must be raised. First, the fact that respondents clearly differentiate between the social acceptance outcomes shows that it is possible to capture several dimensions in a single survey instrument. However, more research is needed to better understand how evaluations might be affected by the inclusion and order of several outcomes of interest.

Second, we selected attributes based on insights from a review of literature and policy discussions, to form expectations based on past empirical findings (e.g. regarding environmental attributes such as biodiversity and carbon footprint) and theory (e.g. farmer characteristics in the light of justice beliefs). Alternative ways to selecting attributes for factorial survey experiments are desirable that allow derivation of specific hypotheses for attributes to be tested. One option in this regard is to apply a document and discourse analysis on the study context to identify relevant attributes and their levels. Qualitative and mixed methods could be used to facilitate the attribute selection process, for example, Q methodology (Armatas et al. 2014; Jensen 2019; Schulze and Matzdorf 2023).

Third, and related to attribute selection, we cannot deny the possibility that some respondents may have struggled with the complexity of the evaluation tasks. Issues with literacy difficulties in the general population may or may not carry over to online panels such as the one used here. Our model results, which discriminate well across the four social acceptance outcomes, suggest that overall respondents were able to meaningfully relate to and engage with the evaluation tasks. In our survey data, there is no evidence to suggest issues in comprehension: based on 59 per cent of all respondents who answered a voluntary open-ended question about their understanding of the whole questionnaire, only 3 per cent answered ‘no’ (92 per cent ‘yes’; remaining 5 per cent ‘don't know’). Further, when asked whether they had difficulties answering questions, of 60 per cent who answered, only 7 per cent answered ‘yes’ (89 per cent ‘no’; remaining 6 per cent ‘don't know’). Nevertheless, to increase confidence in the validity of the method, it is important to better understand how respondents perceive and react to the complexity of wordy vignettes or alternative displays of the same information. In this respect, an early study comparing display of vignette information via tables versus text found no significant differences in evaluations in a factorial survey experiment with eight attributes (Sauer et al. 2020).

Fourth and possibly related to concerns about complexity, we find that the mode of the distributions of evaluation scores is always the midpoint of the scale. In our study, midpoints have a defined meaning. For example, the midpoint for the fairness scale is defined as views that income support is perceived to be fair based on the vignette description. Hence, evaluations at the midpoint can simply indicate agreement of perception with the definition of the midpoint. An alternative explanation is that respondents might have looked for specific thresholds regarding attributes in vignettes that triggered deviations from an otherwise ‘average’ rating. Higher rates of mid-point ratings could also indicate respondent fatigue. An ‘average’ evaluation may be perceived as an ‘easy way out’ of a cognitively challenging task. However, for none of the four social acceptance outcomes do we find that incidence of a score of six (midpoint) increases for the last three vignettes shown compared to the three vignettes shown first. Another potential explanation is that mid-point evaluations are systematically used by respondents who are not confident about their judgements, or who are not sufficiently engaged with an evaluation. We find that 5 per cent of respondents showed serial selection of mid-point scores (i.e. always giving a rating of six) for acceptance of changes in payments. The corresponding figures are 6 per cent for fairness of payments, 8 per cent for intention to be supplied and 14 per cent for intention to petition. These rates do not suggest that serial selection of mid-points was a prevalent issue. The higher incidence rate for the intention to petition criterion may be due to respondents being uncertain whether to petition or not. Importantly, we find that our general findings are unaffected by serial selection of the same ratings that shift the value of the intercept but do not bias the coefficients of attribute influence.

Fifth, the evaluation tasks are hypothetical, and we cannot exclude the possibility that respondents’ answers differ from actual behaviour, or that they do not fully translate into corresponding behaviour. This is a common challenge for multifactorial survey experiments (including stated preference experiments), and it mainly means that we need to be careful when interpreting absolute acceptance and fairness levels stated. However, the differences in relative attribute effects (i.e. whether certain attributes are more important than others) can be expected to be less affected by this potential bias. Combining hypothetical factorial survey experiments with approaches such as real choice experiments (Liebe et al. 2019) or other incentivised experimental games (Lusk and Briggeman 2009) could shed more light on any differences with actual behaviour.

Our study offers insights on the social acceptance of future agricultural policy in Scotland. Elements that resonate strongly in our findings are recognition of production efficiency and reward for high standards with respect to animal welfare and environmental performance. This overlaps with important elements laid out in the strategic vision of Scottish Government for a post-Brexit agricultural support architecture (Scottish Government 2022a). Scotland is aiming to embed principles of a ‘Just Transition to a net zero economy’ within its sectoral policies, defined as a transition that is fair and leaves no one behind (Scottish Government 2022b). This includes ensuring appropriate support for smaller farmers and crofters, and relates to our finding of strong acceptance of payments for smaller (as opposed to larger) farms. However, it should be noted that payments to smaller and unprofitable farms in line with public preferences reduce competition and may slow down structural change in the industry, an issue that deserves further investigation. The new policy framework also aims at establishing an effective monitoring framework to assess the farms’ performance with respect to environmental targets and agricultural production. Our study shows that members of the public were concerned about a wide range of attributes that are indicative of a farm's performance. This offers opportunities for communicating policy changes and their impact to the public. For example, monitoring information could be prepared in summary form to reflect farm attributes, including the ones used in this study, as a way to maintain policy legitimacy, once the new agricultural policy is implemented.

Regarding views of citizens and consumers, we have shown that factorial survey experiments can be used to study the socio-political acceptance of agricultural policy. The method may also offer an effective way to assess food values, as an alternative to choice-based approaches such as best-worst scaling (Lusk and Briggeman 2009), or to study food-related behaviours, for example, related to food waste (Walter et al. 2023). Factorial survey experiments allow the study of consumer acceptance of new technologies in agriculture, especially if there are potential ethical implications, for example, regarding genetical modification in agriculture. Acceptance and social norms of novel food products, for example, insect-based food, or acceptance of food safety interventions, and effectiveness of risk communication, are other fields of potential application.

The above examples illustrate the potential and flexibility of factorial survey experiments as a methodological tool applied in the context of agriculture and land use. Stated preference methods, including choice experiments as another multi-attribute method, tend to focus on outcomes and are best applied to questions with a focus on allocative efficiency. Factorial survey experiments are highly suitable to study questions of social acceptance that cannot be easily captured through a binary (yes/no, either/or) response. This is often the case when there is a dependency across societal actors that raises questions of distributional and procedural equity, and where the influence of social norms on outcomes cannot be neglected.

Footnotes

Acknowledgements

K.G. and S.T. acknowledge support of the Scottish Government, as part of the Environment, Natural Resources and Agriculture (ENRA) Strategic Research Programme 2022–2027, project SRUC-B3-1: Ensuring positive behavioural change for farmers towards best practice for clean growth: economic and behavioural investigations. J.B. acknowledges UKRI funding for the project BB/W018152/1 – TRAnsforming the DEbate about livestock systems transformation (TRADE), which is part of the Transforming the UK Food System for Health People and a Healthy Environment SPF Programme.

Data Availability

Data and code for the analysis presented in this paper are available from an online repository (https://zenodo.org/doi/10.5281/zenodo.11035264).

Appendix Appendix

References