Extension and management pathways for enhanced farm sustainability: evidence from Irish dairy farms

Abstract

This article examines pathways among farmers’ extension participation, the uptake of recommended farm management practices and economic and environmental sustainability. We explore the ‘win-win’, efficiency-based focus of the Irish hybrid extension programme using an unbalanced panel dataset of dairy farms from 2010 to 2019. We apply two-way fixed effects regression models and sensitivity analyses to ensure the robustness of our results to effect heterogeneity and omitted variable bias. Our findings reveal that extension participation has a limited association with the adoption of recommended practices. These practices might be associated with economic benefits, while their environmental effects are not evident. Additionally, extension participation is not found to have a direct association with sustainability outcomes. These findings have important implications for extension programmes that focus on economic and environmental outcomes.

1. Introduction

Environmental implications of agriculture create a significant public debate (Lipper et al., 2018). As a result, farmers are expected to adapt their farming practices in line with the evolving consumer, policy and public landscapes and produce food in a more environmentally sustainable manner (Dessart et al., 2019; Pannell and Claassen, 2020). Yet, changing farm practices requires knowledge on available technologies and how to implement them (Chavas and Nauges, 2020). This underlines the key role of agricultural extension in disseminating new information to farmers and supporting the transition towards more sustainable production systems (Klerkx and Jansen, 2010; Norton and Alwang, 2020). While funding public extension has been a key channel of governmental interventions since the 1950s (Anderson and Feder, 2004; Takahashi, Muraoka and Otsuka, 2020), significant challenges arise from the more recent dual role of advising farmers in both productive and environmental goals (Feder, Birner and Anderson, 2011; Klerkx, de Grip and Leeuwis, 2006; Klerkx and Jansen, 2010). In addition, the general reach of public extension messages has been increasingly questioned due to changes in extension demand and growing diversity of farmers’ profiles and information needs (Feder, Birner and Anderson, 2011; Läpple, Barham and Chavas, 2020; Norton and Alwang, 2020; Weersink, 2018). Indeed, larger commercial operators may liaise directly with industry information sources, thereby leaving public extension dealing primarily with smaller scale farms (Norton and Alwang, 2020). Hence, better understanding how extension can encourage the transition towards greater farm sustainability is important to improve extension schemes and facilitate successful use of public resources in sustainable agricultural development. In this article, we address this question in the context of a mixed public–private extension system.

Mixed public–private extension systems (hereafter referred to as hybrid extension) are gaining attention for their potential in reconciling objectives across multiple sustainability aspects, while attracting a wide pool of farmer profiles (Feder, Birner and Anderson, 2011; Nordin and Höjgård, 2017; Prager et al., 2016; Sutherland et al., 2013). Specifically, they are viewed as a means to get the best of both public and private extension systems to accompany the global shift in farm structure and environmental focus (Feder, Birner and Anderson, 2011; Klerkx and Jansen, 2010). These hybrid systems share funding and delivery responsibilities between public and private sources to address farmers’ ever-evolving information needs on farm production and sustainability. To date, little empirical work has examined hybrid extension in depth and notably its success in striking a balance between farm economic and environmental sustainability (Feder, Birner and Anderson, 2011; Klerkx, de Grip and Leeuwis, 2006; Klerkx and Jansen, 2010). A potential strategy for such extension schemes is to identify synergies across sustainability aspects and promote ‘win-win’ farming practices (Llonch et al., 2017; Nordin and Höjgård, 2017; Pannell and Claassen, 2020). This requires a combined focus on identifying suitable farming practices that can lead to desired sustainability outcomes and on alleviating information constraints to encourage their widespread uptake.

This article specifically examines pathways between farmers’ participation in a ‘win-win’ hybrid extension programme and farm economic and environmental sustainability outcomes. To this end, we assess the direct and indirect effects of extension participation. We first explore the indirect effect of extension by assessing its association with the adoption of recommended, ‘win-win’ farm management practices. Then, we test the relationship of such practices with farm sustainability across a range of economic and environmental outcomes. In addition, to control for pathways other than the adoption of the recommended, ‘win-win’ practices under examination, we analyse the direct association of extension participation with selected farm sustainability outcomes.

We contribute to the literature in three main ways. First, we widen the focus of extension studies on environmental outcomes, which is necessary as publicly funded programmes increasingly incorporate environmental objectives to their schedules. In general, environmental outcomes are underrepresented relative to economic outcomes in the extension literature. While many studies investigated extension’s effect on economic outcomes, such as productivity, farm incomes and profitability (e.g. Cawley et al., 2018; Davis et al., 2012; Nakano, Tanaka and Otsuka, 2018), only Nordin and Höjgård (2017) analysed the impact of participation on an environmental sustainability indicator, i.e. farm nutrient surplus. Some other articles merely explored extension’s effect on the adoption of ‘sustainable’ practices, without providing further empirical proof of environmental enhancements (e.g. Arslan, Belotti and Lipper, 2017; Knook et al., 2020).

Second, we specifically discuss study findings within the context of a hybrid extension system. Better consideration of the alignment between programme outcomes and funding and delivery mechanisms is needed to improve the use of public resources in agricultural extension for greater sustainability (Norton and Alwang, 2020). This is particularly important as it can be difficult to justify public funding in agricultural development and extension without wider environmental focus and benefits (e.g. Barnes et al., 2019).

Third, by examining in depth the pathways among farmers’ extension participation, recommended farm management practices and economic and environmental sustainability outcomes, we investigate if and where ‘things can go wrong’ in extension. To date, the literature provides limited information as to whether extension sometimes has difficulties in encouraging technology adoption or whether adopted technologies are unsuitable to reach desired farm-level outcomes. Filling this gap in knowledge will provide insights into the success of extension schemes and help better target extension plans.

We focus on the hybrid extension programme of the Republic of Ireland (hereafter referred to as Ireland) from 2010 to 2019, which provides an excellent context for this analysis. The public extension body (Teagasc, the Agriculture and Food Development Authority) operates under a mixed public–private arrangement¹ (Donnellan, Hennessy and Thorne, 2015). Within Teagasc, part of the advisory services are delivered based on the peer-to-peer, participatory learning model of farmers’ discussion groups (Donnellan, Hennessy and Thorne, 2015). Specifically, discussion groups are composed of 15 to 20 farmers and an advisor, who acts as a group facilitator. Over the 2010s, Irish extension focused on economic and environmental challenges associated with changes in European Union (EU) and Irish policies (Department of Agriculture Food and the Marine, 2020; Donnellan, Hennessy and Thorne, 2015; Läpple and Hennessy, 2014). The programme adopted a ‘win-win’, efficiency-based approach to improve farm management practices and thereby enhance farm economic and environmental sustainability.

We use an unbalanced panel dataset of Irish dairy farms from the Teagasc National Farm Survey (NFS), which is part of the EU Farm Accountancy Data Network (FADN). Our sample covers a 10-year time period from 2010 until 2019 and gathers data from 437 farms, including both discussion group members and non-members. The breadth of the Teagasc NFS allows for the development of continuous farm management indicators, as well as distinct economic and environmental sustainability outcomes. The choice of farm management and sustainability indicators is guided by the ‘win-win’ focus of the hybrid extension programme, as well as policy relevance in the Irish context. Farm management is represented by the share of homegrown grass in the diet of dairy cows and herd calving rate. Direct production costs per unit of milk produced and gross margin per cow serve as indicators of economic sustainability, while environmental sustainability is measured through nitrogen surplus per hectare (ha) and greenhouse gas (GHG) emissions per unit of milk produced.

Based on our study context, we develop a conceptual framework to shed light on the pathways among years of participation in discussion groups, recommended farm management practices and economic and environmental sustainability. The econometric analysis is performed with two-way fixed effects (TWFE) regression models (Allison, 2009). We overcome common issues of the TWFE estimator with continuous treatments by assessing robustness to effect heterogeneity (Callaway, Goodman-Bacon and Sant’Anna, 2021; Wooldridge, 2021). Finally, sensitivity to omitted variable bias (OVB) is tested using the Cinelli and Hazlett (2020) methodology. This analysis does not claim to identify causal impacts due to data constraints; however, it credibly signs the effects of interest under investigation.

Overall, our findings reveal that extension participation has a limited association with the adoption of recommended practices under study. These practices might be associated with economic benefits, while their environmental effects are not evident. Additionally, extension participation is not found to have a direct association with sustainability outcomes. The study questions the ‘win-win’, efficiency-based approach for achieving greater farm sustainability in hybrid extension schemes. It also recommends designing more tailored extension messages and better account for the changing structure and heterogeneity of farms.

The remainder of this article is structured as follows: Section 2 outlines the contextual background. In Section 3, the conceptual framework is developed. Section 4 describes the empirical approach. Section 5 presents the data and descriptive statistics. In Section 6, the results are reported and discussed, followed by the conclusion in Section 7.

2. Contextual background

The abolition of EU milk quotas in 2015 triggered significant expansion and intensification of Irish dairy production. Supported by governmental targets, farmers increased milk production by about 54 per cent between 2010 and 2019 (Central Statistics Office, 2020), mainly through expansion of dairy cow numbers and higher use of external inputs (Kelly et al., 2020). Despite positive economic outcomes, dairy sector growth challenged compliance with EU environmental commitments. For example, between 2010 and 2019, Irish agricultural GHG emissions increased by 10.3 per cent (Environmental Protection Agency, 2022). Additionally, nutrient concentrations have been on the rise in Irish waters since 2013, with 32 per cent of rivers exceeding nitrates environmental quality standards in 2020 (Environmental Protection Agency, 2021). In that regard, the Environmental Protection Agency underlined the link among nitrogen response in waters, changes in herd numbers and nitrogen surplus on agricultural land (Environmental Protection Agency Catchments Unit, 2021).

To limit negative environmental externalities of agricultural development, the Irish government focused its policy attention on the ‘sustainable intensification’ concept (Department of Agriculture Food and the Marine, 2015; Kelly et al., 2020). The predominant strategy was to encourage farm-level efficiency gains so that better dairy sustainability would be achieved through an improved conversion of inputs into outputs and a reduction of fixed environmental costs per unit of milk produced. In this context, to provide further support to extension in fostering sustainable growth, the Irish government paid farmers up to EUR 1,000 per year to participate in discussion groups from 2010 to 2012 and again from 2017 to 2019 (Department of Agriculture Food and the Marine, 2020; Läpple and Hennessy, 2014).

Over the studied period, the hybrid extension programme aligned with the ‘win-win’ view of enhanced dairy sustainability by fostering an improved use of farm resources (Department of Agriculture Food and the Marine, 2020; Läpple and Hennessy, 2014; Teagasc, 2015). The programme’s scope covered issues of cost control and profitability, nutrient management and pressure, and GHG emission efficiency. Content guidelines were specified accordingly to address farmers’ information needs in the key management areas of Irish dairying, that is, grassland management, and breeding and herd fertility. In fact, the Irish dairy sector relies on a grass-based production system, where cows graze outdoors from early spring until late autumn through rotational grazing (Kelly et al., 2020; Läpple, Hennessy and O’Donovan, 2012). The temperate climate allows for the growth of a large amount of grass, which represents the main and cheapest source of dairy feed (Hanrahan et al., 2018; Läpple, Hennessy and O’Donovan, 2012). Best practice for Irish dairy farmers is to maximise homegrown grass in the diet of dairy cows with appropriate supplementation from concentrate feed when grass supply and grazing conditions are not adequate (O’Brien, Moran and Shalloo, 2018). Thus, dairy farmers synchronise seasonal grass growth, herd feeding requirements and reproductive cycles (Butler, 2014). In such a production system, high reproductive performance, notably represented by large calving rates, is key for matching spring pasture growth and cow intake demands (O’Sullivan et al., 2020; Shalloo, Cromie and McHugh, 2014).

Embedded into the scope and guidelines of the hybrid extension programme, discussion group members and their advisor decide on the details of how they wish to proceed, i.e. they allocate the amount of learning time dedicated to each key management area (i.e. grassland management, and breeding and herd fertility). More precisely, meetings are usually held on one of the participants’ farms, where host farms rotate on a monthly basis. At their first meeting of the year, groups decide on the year’s meeting calendar and choose which topics will be discussed, in line with seasonal farming tasks, host farmer’s specific conditions and programme guidelines. In this way, each meeting focuses on a specific issue, presented by the advisor and the host farmer, with time dedicated to discussion among peers.

3. Conceptual framework

In this section, we describe the hypothesised pathways among extension participation, improved farm management practices and economic and environmental sustainability.

The overarching aim of extension is to improve farmers’ decision-making and analytical skills, as well as to reduce information constraints associated with the adoption of new technologies (Chavas and Nauges, 2020; Davis et al., 2012; Norton and Alwang, 2020). Given the ‘win-win’ focus of the Irish hybrid extension programme, farmers may have economic and/or environmental motivations for participating in discussion groups. Over the years of discussion group participation, farmers are expected to learn about the economic and environmental benefits of new farming practices, both of which are important in their decision-making. As such, farmers’ motivation to adopt recommended farm management practices may be driven by private (i.e. economic) and public good (i.e. environmental) implications. This motivation may differ across farmers because of heterogeneity in their objectives, profiles and farming conditions (Montes de Oca Munguia and Llewellyn, 2020; Pannell and Claassen, 2020). However, due to the ‘win-win’ nature of recommended practices, either motivation (i.e. private or public) can meet the sustainability objectives of the hybrid extension programme.

To encourage dairy farmers to improve their adoption of recommended management practices, Irish extension has set specific targets (Teagasc, 2016). Ninety-three per cent of dry matter fed to dairy cows should come from homegrown grass, distributed between 74 per cent of grazed grass and 19 per cent from grass silage (Teagasc, 2016). Farmers should achieve a 90 per cent in-calf rate at the final pregnancy diagnosis (Teagasc, 2016), which can be translated into a 90 per cent target for the calving rate. To date, the general consensus in Irish studies is that adoption rates of practices in the areas of grassland management and reproductive performance are relatively low, notably due to information constraints (Hyland et al., 2018; Regan et al., 2021; Shalloo et al., 2021, 2018). Improvements are expected through learning by reducing the uncertainty revolving around new practices and how best to apply them (Chavas and Nauges, 2020). Additionally, while some easily observable knowledge is best acquired through peer-to-peer demonstration, others, more complex by nature, require formal training (e.g. the use of decision support tools) (Chavas and Nauges, 2020; Songsermsawas et al., 2016). Hence, the Irish extension model is well suited to support farmers’ learning in the key management areas of Irish dairying, as it encourages farmers to share information and learn from their peers, while providing additional expertise and structure (Macken-Walsh, 2019).

Specifically, Irish extension recommends increasing the share of homegrown grass in the diet of dairy cows by improving grass production and utilisation (Teagasc, 2016). Enhancements in grass production are achieved through improved soil fertility and grass productivity, notably by adopting practices such as yearly soil testing and periodic sward reseeding. To enhance grass utilisation, farmers are recommended to improve grazing infrastructures and perform regular grass measuring and budgeting. Regarding herd calving rates, extension encourages improvements in herd genetic merit and fertility monitoring (Teagasc, 2016). Higher herd genetic merit can be achieved through improved breeding decisions. As for fertility monitoring, the combination of high submission and conception rates is essential and relies on precise assessment (e.g. timely heat detection, appropriate body condition scores and good animal health), particularly just before and during the breeding season (Butler, 2014; Lane et al., 2013).

Improving the share of homegrown grass and herd calving rate can have positive implications for farm economic and environmental sustainability, which justifies the focus of the Irish hybrid extension programme over the 2010s. In relation to economic benefits, better grass productivity and utilisation can reduce requirements for fertiliser application and brought-in feed (Hanrahan et al., 2018; O’Brien, Moran and Shalloo, 2018). This is expected to have positive cost implications and thus increase farm profitability (Hanrahan et al., 2018). Moreover, improvements in herd genetic merit can increase cow productivity and fertility in the medium to long term. As a result, direct production costs can be reduced, thereby resulting in higher gross margins (Ramsbottom et al., 2012; Ring et al., 2021). Reproductive inefficiency can lead to large economic losses such as increased culling rates, adverse effects on labour costs resulting from suboptimum calving dates and higher artificial insemination (AI) and intervention costs (Lane et al., 2013; Shalloo, Cromie and McHugh, 2014).

In relation to environmental benefits, increasing grass productivity and utilisation can decrease GHG emissions associated with the application of fertilisers and the production and transport of brought-in feed (O’Brien et al., 2016, 2014a). Reducing nitrogen imports, mainly through a decrease in fertiliser application and concentrate feed use, is expected to lower nitrogen pressure at the farm level (Buckley et al., 2016; Foote, Joy and Death, 2015). Over time, productivity gains through improved herd genetic merit are expected to lower GHG emitted per unit of milk produced (Lanigan et al., 2018; O’Brien et al., 2014a). Additionally, increasing fertility can extend the average productive lifespan of the dairy herd and reduce culling due to low reproductive performance (Llonch et al., 2017). This is expected to improve GHG emitted per unit of milk produced, notably by diluting the fixed environmental costs associated with feeding dairy animals before they become productive and reducing the need to generate dairy replacements (Berry et al., 2020; Garnsworthy, 2004; Llonch et al., 2017). Finally, because of efficiency gains and better conversion of inputs into outputs, improved reproductive performance is expected to lower nitrogen pressure at the farm level (Lanigan et al., 2018; O’Brien et al., 2014a).

Overall, based on the key assumption that farmers’ decision to participate in discussion groups is motivated by economic and/or environmental considerations, we expect discussion group participation to help farmers to improve farm sustainability through multiple pathways. These are summarised in Figure 1. Notably, we expect extension participation to foster learning about grass production and utilisation and breeding and herd fertility, thereby improving the share of homegrown grass in the diet of dairy cows and herd calving rate. This is expected to serve as indirect pathways to achieve greater economic and environmental benefits in the form of reduced costs, larger margins, higher emission efficiency and lower nutrient pressure. As extension aims to increase farmers’ decision-making and analytical skills (Davis et al., 2012; Norton and Alwang, 2020), we also expect discussion group participation to directly improve selected farm sustainability outcomes.

4. Empirical approach

Our empirical strategy is based on TWFE regression models, where individual and time fixed effects are controlled for and standard errors are clustered at the farm level (Allison, 2009; McCullagh and Nelder, 1989). To explore the effect of discussion group participation on recommended farm management practices, we estimate models where homegrown grass (⁠|$HG$|⁠) and calving rate (⁠|$CR$|⁠) are dependent variables and the year of discussion group participation (⁠|$DG$|⁠) is an independent variable. To examine the effects of discussion group participation and recommended practices on farm sustainability, we estimate models where the sustainability outcomes (⁠|$S$|⁠) are dependent variables and years of discussion group participation, homegrown grass and calving rate are independent variables. We include a wide set of farm and farmers’ characteristics as control variables in the regression models to reduce OVB. Moreover, to limit the simultaneity between dependent and independent variables, we include years of discussion group participation, as well as some other control variables, as one-year lags (De Mey et al., 2016; Fan, Hazell and Thorat, 2000; Wooldridge, 2012). This is because the simultaneity between current dependent variables and past regressors is likely to be small or even non-existent (Fan, Hazell and Thorat, 2000). This approach also allows for time adjustment lags between learning about and implementing new practices (Fan, Hazell and Thorat, 2000; Wooldridge, 2012), which is relevant in an extension context. One-year lags are chosen to limit loss of information as only 10 time periods are observed.

The following equations are estimated:

where the subscript |$i$| refers to the individual being observed and the subscript |$t$| denotes the time period at which individual |$i$| is observed. |${\left( {{\alpha _{j,i}}} \right)_{j \in \left[ {1;5} \right]}}$| are the individual fixed effects, |${\left( {{\gamma _{j,t}}} \right)_{j \in \left[ {1;5} \right]}}$| are the time fixed effects and |${\left( {{u_{j,it}}} \right)_{j \in \left[ {1;5} \right]}}$| are disturbances. |$X$| is the set of farm and farmers’ characteristics that are included as one-year lags, and |$C$| are the controls included as current levels. |$\beta $| are parameters to be estimated. To explore the indirect pathways between extension and farm sustainability, particular interest lies in |${\beta _{1,1}}$|⁠, |${\beta _{2,1}}$|⁠, |${\beta _{3,2}}$| and |${\beta _{3,3}}$|⁠. Specifically, |${\beta _{1,1}}$| and |${\beta _{2,1}}$| give the effect of discussion group participation on homegrown grass and calving rate, respectively. |${\beta _{3,2}}$| represents the effect of homegrown grass on sustainability outcomes. |${\beta _{3,3}}$| gives the effect of the calving rate on farm sustainability. Finally, the direct effect of discussion group participation on sustainability outcomes is given by |${\beta _{3,1}}$|⁠.

While we exploit the panel nature of the data by controlling for TWFE and clustering standard errors at the farm level (Allison, 2009; McCullagh and Nelder, 1989), this approach is not sufficient to draw credible conclusions about pathways for enhanced farm sustainability. Difficulties arise in interpreting TWFE estimates for two reasons. First, the estimator is sensitive to effect heterogeneity, notably when using continuous treatments (Callaway, Goodman-Bacon and Sant’Anna, 2021). Consequently, TWFE estimates can give a misleading summary of the overall average effect of treatment variables (Callaway, Goodman-Bacon and Sant’Anna, 2021). Second, because extension participation is voluntary and discussion groups may attract more motivated farmers with better performances, concerns remain regarding time-varying unobserved heterogeneity (Imbens and Wooldridge, 2009). To overcome these limitations and assess robustness, two sensitivity analyses are used after the TWFE estimation. While a brief summary of methodological approaches is provided in the main body of this article, please refer to  Appendix B for a more detailed description.

Regarding the issue of effect heterogeneity, we estimate equations (1–3) on subsamples of the dataset, where the ‘intensity’ of our treatments (i.e. |$DG$|⁠, |$HG$| or |$CR$|⁠) varies. Then, we compare the results obtained across the different subsamples and the full sample. If the TWFE-estimated effect remains consistent across all estimations, we conclude that it is not sensitive to effect heterogeneity.

As for the issue of time-varying unobserved heterogeneity, we implement the sensitivity analysis developed by Cinelli and Hazlett (2020). This method simulates the effect of potential confounders with varying levels of association with treatment and outcome to test the stability of the estimated effect of the treatment on the outcome. Levels of association are measured in terms of changes in the partial R-squared of the treatment or outcome model, i.e. changes in the residual variance of the treatment or the outcome associated with the inclusion of simulated confounders. As recommended by Cinelli and Hazlett (2020), we use the robustness value (⁠|$RV)$| and |$R_{Y\sim D|X}^2$| values as sensitivity measures, as well as a bounding approach of confounders’ strength based on observed characteristics.

|$RV$| refers to the proportion of residual variance of the treatment and the outcome explained by confounders so that the TWFE-estimated effect is eliminated. |$R_{Y\sim D|X}^2$| reveals how strongly confounders that explain 100 per cent of the outcome would have to be associated with the treatment to eliminate the TWFE-estimated effect. As for the bounding approach, we bound the partial R-squared values of confounders with the treatment and the outcome (i.e. |$R_{D\sim Z|X}^2$| and |$R_{Y\sim Z|X,\,D}^2$|⁠, respectively) based on information from two observed characteristics. We choose to use the largest predictors of treatment and outcome as bounding variables, identified as the highest contributors to the overall R-squared values when regressing treatment and outcome on all explanatory variables individually.

Assuming that confounders are as strong as these bounding characteristics (in terms of variance explained), we compare the bounds for |$R_{D\sim Z|X}^2$| and |$R_{Y\sim Z|X,\,D}^2$| with the |$RV$| value, as well as the bound for |$R_{D\sim Z|X}^2$| with the |$R_{Y\sim D|X}^2$| value. These comparisons allow us to determine the level of association that would eliminate the TWFE-estimated effect in instances where confounders are as strong as the largest predictors of treatment and outcome. Moreover, we also estimate the adjusted treatment effect (⁠|${\hat \beta _{adj}}$|⁠) and its t-value (⁠|${t_{adj}}$|⁠), if we were to control for confounders as strong as the largest predictors of treatment and outcomes. Consistency between the TWFE-estimated effect and the adjusted effect would indicate robustness to OVB.

5. Data and descriptive statistics

The main source of data is the Teagasc NFS, which is operated as part of the EU FADN in Ireland (Dillon et al., 2016). The data are collected on a yearly basis through face-to-face surveys by a team of professional data recorders. In conjunction with the Central Statistics Office, a sample of approximately 900 farms is selected to represent Irish farms. Respondents are classified into six farming systems according to their main source of gross output: dairy, cattle rearing, cattle other, sheep, arable and mixed livestock.

In this article, we restrict our sample to farms with a dairy enterprise, composed of specialised and non-specialised dairy farms. More explicitly, dairy farms included in the sample earned on average 67.9 per cent of their gross output from dairy production, while the remainder came from not only alternative farming enterprises, mainly beef cattle, but also sheep or to a lesser extent crop production. An unbalanced panel dataset from 2010 until 2019 is compiled by selecting farms that do not have missing values for the key variables of interest. The sample contains 2,848 observations accounting for 437 farms and represents about 87 per cent of the full 2010–2019 Teagasc NFS dairy sample. Farms remain in the panel for an average of 6.5 years. As regressions include one-year lagged variables, the sample size reduces to 2,241 observations for the main analysis to focus on farms present in the sample for at least two consecutive years.

The key variables of interest include years of discussion group participation, the two farm management practices and four sustainability outcomes. Regarding years of discussion group participation, the initial year of participation is observed in the data for members who joined after 2010. For members who joined in or prior to 2010, the initial year of participation is self-declared. For non-members, the variable takes the value of 0.

The share of homegrown grass in the diet of dairy cows is calculated following the methodology developed by O’Brien, Moran and Shalloo (2018) for the Teagasc NFS dataset. Through a back calculation based on dairy cow energy requirements, kilograms (kg) of dry matter fed from homegrown grass can be estimated by subtracting energy supply coming from other feed sources from the total energy demand. This method accounts for differences in energy content by feed type (Jarrige, 1989; O’Brien, Moran and Shalloo, 2018). Energy demand includes maintenance and activity, milk production, pregnancy and body weight change and growth (O’Brien, Moran and Shalloo, 2018). The share of homegrown grass in the diet of dairy cows is then deduced by dividing kg of dry matter fed from homegrown grass by total kg of dry matter fed.

Calving rate, i.e. the proportion of dairy cows calving in the herd, is not directly recorded in the Teagasc NFS. This information is retrieved by dividing the amount of calf births by the opening stock of dairy cows, adjusted for potential still and twin birth rates. As these data are not included in the dataset, national averages for Irish dairy herds are used: the prevalence of still births is about 4 per cent (Mee, Berry and Cromie, 2008) and the average twinning rate is approximately 2 per cent (Fitzgerald et al., 2014).

Economic sustainability is measured through direct production costs per unit of milk produced and gross margin per cow. These are common indicators of farm economic performance (Kassie et al., 2018; Läpple and Hennessy, 2015). Kilograms of milk produced are converted to kg of fat-protein-corrected milk (FPCM) to control for differences in milk solids and energy requirements across farms (International Dairy Federation, 2015).

Environmental sustainability is assessed through nitrogen surplus per hectare (ha) and GHG emitted per unit of milk produced (Buckley et al., 2019). Nitrogen surplus gives a farm-level estimation of nitrogen, potentially available for loss to the atmosphere and water bodies, and hence serves as an indicator of nutrient management and farm environmental pressure (Buckley et al., 2016; Schröder et al., 2003). It is calculated following an input–output accounting methodology, i.e. as imported (e.g. fertiliser and concentrate feed) minus exported (e.g. milk and livestock) quantities of nutrient on a per-hectare basis (Buckley et al., 2019). In this way, the analysis is restricted within the farm gate to include imports and exports of nutrients over which the farmer has direct control (Buckley et al., 2016).

GHG emitted per unit of milk produced reflects milk GHG emission efficiency and is used as an indicator of farm environmental efficiency (Crosson et al., 2011). Agricultural GHG emissions are estimated by using a cradle-to-farm gate life cycle assessment (LCA) approach (O’Brien et al., 2014b, 2010).² On- and off-farm GHG emissions associated with dairy production are modelled using emission factors that follow the Intergovernmental Panel on Climate Change guidelines or come from other resources in the literature (Dong, Mangino and McAllister, 2006; Duffy et al., 2019). Emissions are converted to kg of carbon dioxide equivalent (CO₂e) using the 100-year Global Warming Potential (Forster et al., 2007). They are reported per kg of FPCM (International Dairy Federation, 2015). The data necessary to calculate GHG emissions are available only after 2013 in the dataset. Thus, the sample is reduced to the years 2013 to 2019 for the analysis with this indicator.

We also include other farm and farmers’ characteristics as control variables in the regressions. Due to space constraints, these are defined and presented in Table A1 in  Appendix A.

Table 1 summarises the definition of key variables and reports descriptive statistics for the 2010–2019 pooled sample. Discussion group members represent about 48.9 per cent of the sample and have been involved in discussion groups for about 9.5 years, on average. The proportion of discussion group members gradually increased from 2010 (46.8 per cent) until 2015 (53.7 per cent). After EU quota removal, participation rates started dropping and reached the lowest figures of the panel in 2019 (43.0 per cent). Moreover, differences in farm management and sustainability performance between discussion group members and non-members are formally tested with t-tests, with results also reported in Table 1. They indicate that discussion group members achieve higher shares of homegrown grass and calving rates than non-members. However, both discussion group members and non-members are below advisory targets of 93 per cent of homegrown grass in the diet of dairy cows and 90 per cent in-calf rates (Teagasc, 2016).

Table 1 also shows that farm economic sustainability is higher for members than non-members. In terms of farm environmental sustainability, while discussion group members achieve lower GHG emissions per unit of output than non-members, they have larger nitrogen surpluses per ha. In general, discussion group members are more productive than non-members, and more productive farms achieve lower GHG emissions per unit of output. In relation to differences in nitrogen surplus, this is likely explained by a larger dairy production scale and intensity of discussion group members.

6. Results and discussion

This section provides the estimation results and discussion. We start by presenting the results of the main TWFE models, with a focus on the indirect and direct pathways between extension and farm sustainability. Before interpreting these findings in our discussion, we present the results from both sensitivity analyses, by using the association between discussion group participation and nitrogen surplus as an example in the main body of the article. All results are summarised and discussed in the Section 6.3.

6.1. Results of the main TWFE estimation models

Table 2 reports the main TWFE estimation results, where only the variables of interest are reported. Please refer to Table C1 in  Appendix C for the full estimation results. We first describe the estimation results concerning indirect pathways between extension and farm sustainability and then turn to the direct effect of extension participation.

Regarding the indirect pathways among extension, farm management and sustainability, the results in Table 2 indicate that discussion group participation does not have a significant effect on the share of homegrown grass nor calving rate. As for the subsequent effect of recommended farm management practices on sustainability outcomes, the results reveal that homegrown grass and calving rate have a positive association with farm economic performance. Specifically, both management practices are negatively associated with direct production costs per kg of FPCM and positively associated with gross margin per cow. In addition, Table 2 shows that the effects of recommended practices on environmental outcomes are mixed. While homegrown grass is negatively associated with nitrogen surplus per ha, the calving rate has a positive association with this indicator. Neither management practice has a significant effect on GHG emitted per kg of FPCM.

Regarding the direct effect of extension on farm sustainability, the results in Table 2 show that discussion group participation does not have a significant effect on economic outcomes, i.e. direct production costs per kg of FPCM and gross margin per cow. The effect of discussion group participation on environmental sustainability is mixed. Discussion group participation is positively associated with nitrogen surplus per ha, but it has a negative association with GHG emitted per kg of FPCM.

6.2. Sensitivity analyses

Before further interpreting the results in Table 2, we verify their robustness to effect heterogeneity and OVB with additional sensitivity analyses. In line with our main study focus on extension and farm sustainability, we describe the results of the sensitivity analyses regarding the effect of discussion group participation on nitrogen surplus as an example. Table 3 shows results about the robustness to effect heterogeneity, while Table 4 reports the results of the Cinelli and Hazlett (2020) method. Due to space constraints, the remaining results from both sensitivity analyses can be found in  Appendices D and  E.³

Regarding the sensitivity to effect heterogeneity, as mentioned, we estimate the TWFE model on subsamples depending on the intensity of the treatment variable. Specifically, to test the effect of discussion group participation on nitrogen surplus in Table 3, the TWFE model is estimated on three subsamples: (i) discussion group members only, (ii) discussion group members for whom the years of participation were below the median of members’ pooled sample and (iii) discussion group members for whom the years of participation were above the median of members’ pooled sample. The results suggest that the effect of discussion group participation is not stable across all three estimations. Hence, we cannot draw conclusions regarding the overall effect of discussion group participation on this outcome due to sensitivity to effect heterogeneity. Nonetheless, Table 3 suggests that the effect of discussion group participation is only significant for more established members who have been participating in discussion groups for longer than the median of members’ pooled sample.

In Table 4, we test the sensitivity of the effect of discussion group participation on nitrogen surplus to OVB based on the Cinelli and Hazlett (2020) method. The findings reveal that if confounders explained 100 per cent of the residual variance of nitrogen surplus, they would need to explain 0.68 per cent of the residual variance of discussion group participation to fully account for the effect reported in Table 2 (i.e. |$R_{Y\sim D|X}^2$| = 0.68 per cent). If confounders explained less than 7.92 per cent of the residual variance of both discussion group participation and N surplus, they would not be strong enough to overturn or explain away the effect reported in Table 2 (i.e. |$RV$| = 7.92 per cent).

To facilitate interpretation, we bound the strength of confounders using observed characteristics that are the most important determinants (in terms of variance explained) of the treatment assignment or the outcome. After regressing treatment and outcomes on all explanatory variables individually, we identify household and stocking rate as the largest predictors of discussion group participation and nitrogen surplus, respectively (i.e. they have the largest contribution to the R-squared values in the treatment or outcome equations). Therefore, the bounding approach presented in Table 4 is based on these two observed characteristics.

We find that the strength of association with confounders as strong as household would be |$R_{D\sim Z|X}^2$| = 0.05 per cent for discussion group participation and |$R_{Y\sim Z|X,\,D}^2$| = 0.14 per cent for nitrogen surplus. As the |$RV$| of 7.92 per cent is higher than either quantity, the findings reveal that confounders as strong as household could not fully eliminate the TWFE-estimated effect from Table 2. Moreover, because the bound for |$R_{D\sim Z|X}^2$| = 0.05 per cent is less than |$R_{Y\sim D|X}^2$| = 0.68 per cent, ‘worst-case confounders’ explaining 100 per cent of the residual variance of the outcome and as strongly associated with discussion group participation as household would not eliminate the estimated effect either. Finally, when controlling for confounders as strong as household or stocking rate, the adjusted effects remain significant and of positive sign. Similar results are found for confounders as strong as stocking rate, with |$R_{D\sim Z|X}^2$| and |$R_{Y\sim Z|X,\,D}^2$| values of 0.53 and 0.86 per cent, respectively, and a consistent adjusted effect. Thus, the main estimation result regarding the effect of discussion group participation on nitrogen surplus reported in Table 2 is robust to OVB.

Table 5 provides an overview of the sensitivity to effect heterogeneity and OVB for all coefficient estimates. As can be seen, the results are more sensitive to effect heterogeneity than to OVB. For example, only 6 out of 14 coefficients are robust to effect heterogeneity, while 11 out of 14 coefficients are robust to OVB. Conclusions on the overall effects of discussion group participation and farm management indicators are drawn based on the combination of results from both sensitivity analyses and are summarised in the last column of Table 5. Only four of the analysed relationships are robust to both effect heterogeneity and OVB; these are the insignificant coefficient estimates of discussion group participation on homegrown grass, cost and margin and the positive association of homegrown grass with margin.

6.3. Summary of findings and discussion

Even though we can conclude on robust, overall effects for only four relationships under examination, the joint analysis of the main TWFE estimates and the results from both sensitivity tests can provide some avenues for further reflection about extension and management pathways for enhanced farm sustainability. Therefore, we summarise and discuss main findings by studied pathway.

First, we focus on the pathway among discussion group participation, homegrown grass and farm sustainability. Our findings suggest that improving farmers’ grassland management skills can serve as an indirect pathway towards enhanced farm economic sustainability. Indeed, the results reveal that the share of homegrown grass in the diet of dairy cows has a robust, positive association with gross margin per cow (Tables 2 and 5). However, over the studied period, we find that discussion group participation has a robust, insignificant association with homegrown grass (Tables 2 and 5), which is surprising given the focus of the hybrid extension programme (Department of Agriculture Food and the Marine, 2020; Läpple and Hennessy, 2014; Teagasc, 2015). A potential explanation for this insignificant finding is that many factors influencing grass production and utilisation are outside of the farmer’s direct control, such as weather and agronomic conditions (Hanrahan et al., 2017). Alternatively, recommended practices to improve homegrown grass may not be applicable to all discussion group members because of differences in their profiles and farm conditions.

Three out of the four homegrown grass effects are sensitive to OVB. In this context, unaccounted differences in farm profiles and conditions (e.g. weather and agronomic conditions) might be a substantial source of unobserved heterogeneity. Without more detailed farm information (especially in relation to grass production and utilisation), it is difficult to assess the effect of homegrown grass on sustainability outcomes and thus draw robust conclusions. However, it is worth noting that, despite sensitivity to OVB, the TWFE estimates for cost and nitrogen surplus are in the expected direction (Table 2).

Second, we examine the pathway among discussion group participation, calving rate and farm sustainability. As all effects related to the calving rate are robust to OVB (Table 5), effect heterogeneity is worth exploring in more detail. For example, the TWFE results suggest that the effect of discussion groups on the calving rate is insignificant (Table 2), but the sensitivity analysis to effect heterogeneity shows that participation is beneficial for more established members (Table D1 in  Appendix D). This could be due to time delays in the effect of extension participation as the herd calving rate is determined by complex practices that do not give an immediate and observable return and require more learning time (Foster and Rosenzweig, 2010; Kuehne et al., 2017; Montes de Oca Munguia and Llewellyn, 2020). Improving herd genetic merit for fertility traits is a very slow process and hence a long-term breeding project. In Ireland, age at first calving is generally between 24 and 36 months (Berry and Cromie, 2009), which suggests a significant time lag between current breeding decisions and resulting improvements in cow reproductive efficiency.

The findings also reveal that improving calving rates is associated with positive economic outcomes (Table 2), but these effects might only be observed for certain farmer cohorts because of effect heterogeneity (Table D3 in  Appendix D). Notably, improving calving rates is associated with positive cost implications for farmers with low cow reproductive performance, while the effects are more mixed for gross margin per cow. A potential explanation for these heterogeneous effects is that farmers may have different breeding strategies or varying needs to generate dairy replacements. Finally, it is worthwhile to highlight that in none of the analyses performed in this study were the calving rate effects beneficial from an environmental perspective.

Third, we explore the direct associations between discussion group participation and farm sustainability outcomes. Our findings show that participation has a robust, insignificant association with farm economic performance (i.e. direct production costs per unit of milk produced and gross margin per cow) (Table 5). This is surprising given the scope of the Irish extension programme (Department of Agriculture Food and the Marine, 2020; Läpple and Hennessy, 2014; Teagasc, 2015), but could be a sign of increasing diversity of Irish dairy farms and may warrant more tailored extension messages depending on the specific farm structure.

Moreover, the TWFE results suggest that discussion group participation has a positive association with nitrogen surplus and a negative association with GHG emitted per unit of milk produced (Table 2), but both effects are sensitive to effect heterogeneity (Table 3 and Table D1 in  Appendix D). Specifically, the associations between discussion group participation and environmental outcomes are driven by more established members. This could be because of time delays in learning about new practices, implementing adjustments in farm management and observing changes in farm performance (Chavas and Nauges, 2020; Foster and Rosenzweig, 2010; Weersink and Fulton, 2020). When considering the direction of these associations, it is interesting to note that while discussion group participation may show benefits in terms of GHG emission efficiency, the association with nitrogen surplus is positive. This highlights the need to comprehensively explore and promote sustainability outcomes, as extension programmes and new technologies may have conflicting effects on farm sustainability (e.g. Godfray, 2015; Pretty, 2018).

7. Conclusion

In a context where agricultural sustainability is increasingly at the forefront of public policy, hybrid extension is gaining attention for its potential in meeting the dual role of advising farmers in both productive and environmental goals. Nevertheless, to date, little empirical work has been conducted to assess this type of extension system. In this article, we examined the Irish hybrid extension programme and its ‘win-win’, efficiency-based approach to enhance farm sustainability. Specifically, we explored the pathways among farmers’ discussion group participation, recommended farm management practices and economic and environmental sustainability outcomes. The analysis was based on an unbalanced panel dataset of Irish dairy farmers from 2010 to 2019. We estimated TWFE models and ensured robustness to effect heterogeneity and OVB with additional sensitivity analyses.

Overall, the findings of this study reveal that the Irish hybrid extension programme has a limited association with the uptake of recommended farm management practices. Promoting these practices might serve as an indirect pathway towards some economic benefits, while their effect on environmental sustainability is not evident. The results also suggest that the hybrid extension programme does not have a direct association with farm sustainability outcomes. While it is important to recognise time delays among learning, implementation and subsequent results in farm sustainability, our findings confirm two potential reasons for the lack of extension’s effect. First, extension can fail to improve farmers’ management skills and get them to adopt recommended practices. This could be because the extension message is not relevant to all participants in the context of growing diversity in farm profiles. Second, recommended practices do not always lead to the desired outcome at the farm level. Hence, the results of this study highlight the importance of breaking down pathways between extension participation and farm sustainability to identify where ‘things can go wrong’. This can help to improve the targeting of extension programmes by identifying more suitable strategies to achieve programme goals and/or reviewing the knowledge dissemination approach. An increasingly important strategy is to better account for the changing structure and heterogeneity of farms and design more tailored extension messages.

In developed countries where policy agendas are increasingly focused on sustainability, contributions to the rural economy, without wider environmental benefits, may not be a sufficient justification to invest public funding in agricultural extension (e.g. Barnes et al., 2019). Our study did not provide statistically significant evidence of environmental benefits associated with the Irish hybrid extension programme, which received significant public funding in the 2010s (Department of Agriculture Food and the Marine, 2020; Läpple and Hennessy, 2014). Difficulties in identifying environmental benefits could be due to potential ambiguous effects of the ‘win-win’, efficiency-based approach to enhanced farm sustainability, promoted by Irish extension. In the context of dairy sector growth, farm-level efficiency gains may have provided incentives to further increase the production scale, thereby outweighing environmental efficiency benefits or even resulting in unintended deterioration as a ‘rebound’ effect (Alcott, 2005; Barnes et al., 2019). This issue has been extensively documented in the literature as the Jevons’ paradox, which questions the ‘win-win’, efficiency-oriented view of enhanced agricultural sustainability (Alcott, 2005; Godfray, 2015; Paul et al., 2019).

Over the 2010s, the ‘win-win’ approach of the Irish hybrid extension programme may not have been suitable to achieve the desired balance between economic and environmental objectives. Alternative farming practices may be considered in the future to meet ‘win-win’ goals. In this context, it is worthwhile to mention that the sustainability focus of the hybrid extension programme is evolving away from a purely efficiency-based approach for the dairy industry, notably to support the implementation of the EU ‘Farm to Fork’ strategy and Irish ‘Food Vision 2030’ strategy. Moreover, dairy expansion and intensification have moderated, which puts less pressure on the agricultural system and could potentially shift some of the information needs of Irish dairy farmers. More research will be needed to continue assessing extension efforts and improve the use of public resources in mixed public–private arrangements.

Finally, it is important to recognise the three main limitations of our study. First, we only explored partial effects by focusing on a selected set of farm management and sustainability indicators. The summary statistics of our pooled sample revealed that discussion group members have been participating for 9.5 years on average. While farmers received some incentives to join, they are dedicating time to extension activities on a voluntary basis. Given the time pressures associated with dairy farming, they must perceive some kind of benefits to continue participating. Our analysis did not identify these benefits, which could be the focus of future research.

Second, the results point out the need to better understand and account for heterogeneity in farming conditions across farms. This would not only help better tailor the extension advice to individual farmers but also be beneficial in future analysis of extension’s effect.

Third, the econometric analysis was bound by available data and methods. In the absence of natural experiments or instrumental variables, it is difficult to provide evidence on extension impact with observational data. In that regard, econometric tools, such as the Cinelli and Hazlett (2020) sensitivity analysis, can help applied researchers to expand the level of knowledge on extension or technology effect, especially when detailed information about farmers’ extension participation, technology adoption behaviours and sustainability performance is available over time. Nevertheless, better monitoring and assessment plans are needed when investing public resources into agricultural extension. It is only by evaluating the (cost-)effectiveness of publicly funded extension programmes in the frame of programme objectives that progress can be made to improve the use of public money and achieve greater sustainability (Norton and Alwang, 2020).

Acknowledgements

This research was funded through the Teagasc Walsh Scholarship Scheme. The authors would like to thank the staff involved in the collection and verification of the Teagasc NFS data, as well as farmers who voluntarily participate in the survey. They are also grateful to Donal O’Brien and Laurence Shalloo (Teagasc) for the sharing of their LCA model used in this research.

Footnotes

References

Appendix A: Additional control variables

Appendix B: Detailed description of methods to assess robustness of TWFE estimates

Effect heterogeneity in TWFE models

Two main issues arise in interpreting TWFE estimates because of effect heterogeneity. First, Callaway, Goodman-Bacon and Sant’Anna (2021) explain that when estimating the intensity of a continuous treatment d with the TWFE estimator (e.g. |$DG$| in equations (1 and 2)), the obtained effect mixes both the ‘level effect’ and the ‘slope effect’. The level effect is the treatment effect of d, defined as the difference between an individual’s potential outcome under treatment and its untreated potential outcome (e.g. difference in the farm management performance between a discussion group member and his/her counterfactual). The slope effect is the causal response to an incremental change in the dose (e.g. effect of an additional year of discussion group participation). The latter effect may vary depending on the dose’s level (e.g. the effect of an additional year of discussion group participation may be different for more established members than for newer members).

Second, Callaway, Goodman-Bacon and Sant’Anna (2021) highlight that TWFE estimates can give a misleading summary of the overall average causal response because the estimator puts more weight on doses close to the average than on the tails. This can be particularly problematic in instances where the highest or lowest doses provide an extreme causal response (e.g. if the effect of discussion group participation takes time to kick in or fades away after a few years of participation).

To address these issues, we estimate equations (1–3) on subsamples of the dataset, where the ‘intensity’ of our treatments (i.e. |$DG$|⁠, |$HG$| or |$CR$|⁠) varies. We do not claim to identify causal effects per se nor separating level and slope effects. Rather, we intend to ensure that the associations between variables of interest pointed out through the TWFE estimation are robust to effect heterogeneity, thereby allowing us to conclude on their overall statistical significance and direction. To do so, we compare the results obtained across the different subsamples and the full sample. If the estimated effect remains consistent across all estimations, we conclude that the TWFE-estimated effect is robust to effect heterogeneity. Conversely, a change of significance or sign in the estimated effect across all estimations would indicate a lack of robustness to effect heterogeneity.

Specifically, to test for the robustness of the estimated effect of discussion group participation (i.e. |$\widehat {{\beta _{1,1}}}$|⁠, |$\widehat {{\beta _{2,1}}}$| and |$\widehat {{\beta _{3,1}}}$|⁠), equations (1–3) are estimated on three subsamples: (i) discussion group members only, (ii) discussion group members for whom the years of participation are below the median of members’ pooled sample and (iii) discussion group members for whom the years of participation are above the median of members’ pooled sample. To test for the robustness of the estimated effect of farm management practices (i.e. |$\widehat {{\beta _{3,2}}}$| and |$\widehat {{\beta _{3,3}}}$|⁠), equation (3) is estimated on three subsamples per farm management practice: (i) farmers for whom the farm management performance is below the 33rd percentile of the pooled sample, (ii) farmers for whom farm management performance is included between the 33rd and 66th percentiles of the pooled sample and (iii) farmers for whom farm management performance is above the 66th percentile of the pooled sample.

Sensitivity to OVB

Cinelli and Hazlett (2020) extends the OVB framework to analyse the sensitivity to unobserved confounders. This method allows us to explore the sensitivity of our TWFE estimates by simulating the effect of potential confounders with varying levels of association with treatment and outcomes. Specifically, it is based on a reparameterisation of the bias in the OVB formula in terms of partial R-squared values (i.e. residual variance). That is, levels of association between confounders and the treatment and between confounders and the outcome are measured through changes in partial R-squared, i.e. changes in the residual variance explained by adding the simulated confounders to the treatment or outcome model. The method does not require any assumptions on the functional form of treatment variables nor on the distribution of unobserved confounders. Specifically, Cinelli and Hazlett (2020) introduce the following measures of sensitivity:

• The proportion of variation in the outcome explained uniquely by the treatment (also called partial R-squared of the treatment with the outcome), |$R_{Y\sim D|X}^2$|⁠: it reveals how strongly confounders that explain 100 per cent of the residual variance of the outcome would have to be associated with the treatment to eliminate the TWFE-estimated effect. In other words, this measure gives the minimum level of association needed between the treatment and confounders that explain 100 per cent of the residual variance of the outcome so that the TWFE-estimated effect is eliminated.

• The robustness value for the point estimate, |$RV$|⁠: it gives the minimum residual variance of the outcome and the treatment needed to ‘explain away’ the effect estimate; that is, if the confounders’ association with both the treatment and the outcome are assumed to be less than |$RV$|⁠, then such confounders cannot explain away the TWFE-estimated effect.

While these two measures are useful to examine the overall sensitivity of our estimates, they do not provide information as to whether obtained values are problematic in our study context. In other words, we must consider whether it is plausible to expect confounders of such strength in our analysis or not. Moreover, it is worthwhile to highlight how unlikely it is to have confounders equally affect both treatment and outcome, which is the assumption taken to estimate the |$RV$| value. Therefore, Cinelli and Hazlett (2020) suggest complementing the sensitivity analysis with a bounding approach of confounders’ strength based on observed characteristics. Confounders’ strength is defined by the proportion of variation explained in the treatment assignment or outcome. Assuming that confounders are x times as strong as a chosen bounding characteristic (where the value x and the bounding characteristic are chosen by the investigator), Cinelli and Hazlett (2020) propose the following, additional sensitivity measures:

• The proportion of variation in the treatment explained by the simulation of confounders that are x times as strong as the chosen bounding characteristic (also called partial R-squared of confounders with the treatment), |$R_{D\sim Z|X}^2$|⁠: it gives the maximum level of association between confounders and the treatment if confounders are x times as strong as the bounding characteristic.

• The proportion of variation in the outcome explained by the simulation of confounders that are x times as strong as the chosen bounding characteristic (also called partial R-squared of confounders with the outcome), |$R_{Y\sim Z|X,\,D}^2$|⁠: it gives the maximum level of association between confounders and the outcome if confounders are x times as strong as the bounding characteristic.

• The adjusted estimate, |${\hat \beta _{adj}}$|⁠, and adjusted t-value, |${t_{adj}}$|⁠: they give an estimation of the treatment effect and its t-value in the instance that bias is introduced by confounders x times as strong as the bounding characteristic. In other words, these values reveal if the estimated effect would remain consistent if we were to control for confounders x times as strong as the bounding characteristic.

As recommended by Cinelli and Hazlett (2020), while we use the |$RV$| and |$R_{Y\sim D|X}^2$| values in our sensitivity analysis, we also report |$R_{D\sim Z|X}^2$|⁠, |$R_{Y\sim Z|X,\,D}^2$|⁠, |${\hat \beta _{adj}}$| and |${t_{adj}}$| values. Following the example outlined in Cinelli and Hazlett (2020), we choose an x value of 1 and the largest predictors in the treatment model and the outcome model as bounding characteristics. Thus, we assume that it is not plausible to expect time-varying confounders to be more than as strong as the largest predictors of our treatments or outcomes. By the largest predictor, we mean the explanatory variable that explains the largest proportion of variation in the treatment or outcome. Cinelli and Hazlett (2020) explain that bounds can be built from meaningful covariates (i.e. variables that explain a large share of the variation in treatment assignment or outcome) based on the research context, theory or previous literature. In our study, we also establish a ‘rule of thumb’ to identify these bounding characteristics by regressing the treatments and outcomes on explanatory variables individually and choosing those with the highest contribution to the overall R-squared value.

While the Cinelli and Hazlett (2020) method does not allow us to identify causal effects per se, it allows us to conclude on the robustness of associations between our variables of interest and OVB. Therefore, this sensitivity analysis is performed to test for the robustness of the estimated effect of discussion group participation in equations (1–3) (i.e. |$\widehat {{\beta _{1,1}}}$|⁠, |$\widehat {{\beta _{2,1}}}$| and |$\widehat {{\beta _{3,1}}}$|⁠) and of farm management practices in equation (3) (i.e. |$\widehat {{\beta _{3,2}}}$| and |$\widehat {{\beta _{3,3}}}$|⁠). In each case, the combination of three robustness checks is used to determine sensitivity to OVB, as follows:

• Bounds for both |$R_{D\sim Z|X}^2$| and |$R_{Y\sim Z|X,\,D}^2$| for bounding characteristics (i.e. the largest predictors of treatment and outcome) are compared to |$RV$|⁠. A higher robustness value indicates that confounders as strong as the bounding characteristics (in terms of variance explained) could not fully eliminate the TWFE-estimated effect.

• The bound for |$R_{D\sim Z|X}^2$| for both bounding characteristics is compared to the |$R_{Y\sim D|X}^2$| value. This comparison establishes a ‘worst-case scenario’, where confounders would explain 100 per cent of the residual variance of the outcome and be as strongly associated with the treatment as the bounding characteristics. In the instance that |$R_{D\sim Z|X}^2$| is less than |$R_{Y\sim D|X}^2$|⁠, this would indicate that such confounders would not eliminate the TWFE-estimated effect.

• We examine if the adjusted effect is consistent with the TWFE-estimated effect when accounting for confounders as strong as our bounding characteristics.

The combined result of these three checks determines if the TWFE-estimated effect is robust to OVB. However, it is worthwhile to mention that if the TWFE-estimated effect is insignificant, we consider that the comparisons between |$R_{D\sim Z|X}^2$| and |$R_{Y\sim Z|X,\,D}^2\,$|bounds and |$RV$| and between the |$R_{D\sim Z|X}^2$| bound and |$R_{Y\sim D|X}^2$| are not very meaningful. Indeed, these provide information as regard to what strength of association would eliminate the estimated effect. Hence, in such a case, we assess predominantly the consistency in the adjusted effect.

Appendix C: Full TWFE estimation results

Appendix D: Additional results of the sensitivity analysis to effect heterogeneity

Appendix E: Additional results of the sensitivity analysis to OVB