Does presentation matter? An analysis of images and text in a choice experiment of green roofs

Abstract

The use of images in choice experiment surveys has been increasing over time. Research on the impact of complex graphical displays of information on respondent comprehension and the quality of preference estimates yields mixed results. We contribute to this literature by leveraging a split-sample design for a choice experiment concerning green roofs in Portland, Oregon, to investigate the effects of including high-quality static images in the survey instrument and in the choice cards. We find that respondents who completed the ‘image’ version of our survey had a significantly higher total willingness to pay (TWTP) to support a new green roof program than respondents who completed the ‘text only’ version of the survey. We explore the relationship between respondent characteristics and TWTP and find that respondents with little knowledge about green roofs who completed the image survey have a TWTP that is over three times larger than text survey respondents. Our findings support the trend in the literature of using images in choice experiments but also highlight the importance of paying attention to image quality in survey design, using focus groups with mixed previous knowledge for survey refinement, and gathering information in surveys themselves about respondents’ prior knowledge about the valuation scenario.

1. Introduction

The increasing use of choice experiments, and their role in informing policies, has raised awareness about the importance of using best practices for survey design and model estimation (Holmes, Adamowicz, and Carlsson 2017; Johnston et al. 2017). The information provided to respondents in stated preference surveys has been a long-standing concern (Green and Tunstall 1999; Munro and Hanley 1999) with studies exploring the effect of different information sets (Munro and Hanley 1999; Czajkowski, Hanley, and LaRiviere 2015; Welling, Zawojska, and Sagebiel 2022), presentation formats (Bateman et al. 2009; Fiore et al. 2009; Jansen et al. 2009; Matthews, Scarpa, and Marsh 2017; Patterson et al. 2017; Rid et al. 2018; Eppink, Hanley, and Tucker 2019; Shr et al. 2019), and framing effects (Faccioli and Glenk 2021). Our paper contributes to this literature by using high-quality static attribute images in our choice experiment survey to investigate if estimates differ based on whether respondents completed an image- or text-only survey and if effects differ by respondent characteristics.

Eppink et al. (2019) compare, in an experimental setting, a traditional text presentation to others that used color maps, histograms, or radar graphs to describe complex ecosystem services. The authors find that the visual representations negatively impacted choice consistency and cautioned practitioners to be careful when using graphics in stated preference surveys. Shr et al.’s (2019) study of landscape attributes of green infrastructure uses three different presentation formats: text only, images only, and both images and text. Their choice cards included neighborhood images from three different viewpoints that were generated using visualization software to ‘give respondents an idea of the place as a whole’ (Shr et al. 2019: 377). The authors find that static images influence respondents’ marginal willingness to pay (MWTP) estimates and decrease attribute non-attendance, but that using images results in less choice consistency, which they attribute to cognitive overload. Rid et al. (2018) find that simpler visuals (static 3D images) outperform a more sophisticated 3D film presentation in their choice experiment about housing development, except for respondents who had more experience with the good being valued. Other studies, for example, Bateman et al. (2009) and Matthews et al. (2017) find that using virtual reality has beneficial effects, including reduced choice errors and increased respondent engagement.

Given the increasing use of static images, typically in the form of clip art, in choice experiments on environmental and natural resource issues, our study extends this literature by exploring the effect of high-quality static images in a split-sample design for a choice experiment of green roofs in Portland, Oregon.^1,2 Respondents saw the identical survey text, but some respondents completed a survey that included attribute images in the background information and choice cards. Great care was taken in creating the images; we hired a graphic designer and devoted a majority of two focus group meetings to improving the images to ensure they clarified the alternatives being offered yet did not overload or confuse our respondents.

Our estimates show that including images significantly increases the MWTP for every attribute and produces total willingness to pay (TWTP) estimates that are 126 and 178 per cent higher, depending on the new green roof program scenario, for respondents who were shown images. The difference in TWTP is driven by two factors. The first is the much higher estimated marginal utility for one particular attribute—reducing combined sewer overflows—among image survey respondents. The second is the significantly lower estimated coefficient for the cost attribute, that is, the marginal utility of money, for image survey respondents. We also find evidence that images have more impact on preference estimates for respondents who were relatively unfamiliar with the subject of the valuation study—green roofs—before the survey began.

Many respondents were familiar with green roofs, with almost 69 per cent reporting that they had visited, seen, heard, or read about green roofs prior to taking the survey. Previous work finds that respondent familiarity with the good tends to be a highly significant predictor of response reliability (Loomis and Ekstrand 1998), and familiarity can also help limit attribute non-attendance (Hanley and Czajkowski 2019).³ We find, for the combined sewer overflow attribute, significantly lower non-attendance for image survey respondents. Although respondents who completed the image survey took, on average, somewhat longer, which is often seen as an indication of higher-quality responses (Börger 2016; Campbell, Mørkbak, and Olsen 2018), the mean total completion time was not statistically different across survey versions.

We estimate models that explore differences in preferences depending on each of a set of respondent's characteristics—education, age, income, and prior experience with green roofs, that is, if a respondent had (‘high knowledge’) or had not (‘low knowledge’) seen, visited, heard, or read about green roofs prior to taking the survey. We find higher TWTP estimates for respondents who completed the image compared to the text version of the survey. Interestingly, the TWTP for respondents with low knowledge about green roofs was over three times higher for those who completed the image survey than those who completed the text survey. When respondent characteristics are considered together in a latent class model, only high knowledge significantly determines differences in preferences. Respondents in the group that tend to have high knowledge are more likely to prefer any green roof project to the status quo and less likely to have a significantly larger TWTP when responding to the image versus the text survey.

This paper is structured as follows. The next section includes a discussion of the geographic area for our study, details on how we developed the survey instrument and images, and the statistical models used in our analysis. Results are presented in the third section, which also includes formal hypothesis tests comparing outcomes from the text and image surveys. This is followed by a discussion of key findings and TWTP estimates for two new green roof programs. The final section summarizes key findings and provides suggestions for future research.

2. Methods

2.1 Study area

The geographic area for our study is Portland, Oregon. Like many cities in the USA, Portland's stormwater system includes an area where sewage and stormwater are combined in one pipe. The combined-sewer part of the system conveys wastewater to a centralized treatment facility, but when its design capacity is exceeded during storm events it discharges untreated wastewater into nearby water bodies (City of Portland Bureau of Planning and Sustainability 2010). A lawsuit filed in 1991 claimed that these discharges (combined sewer overflows) violated the Clean Water Act. In response, the city invested $1.4 billion in gray infrastructure (pipes) to reduce the number of overflow events (Environmental Services City of Portland n.d.).

Green infrastructure—a decentralized approach for managing stormwater that attempts to mimic the natural environment—includes bioswales, tree planting, rain gardens, green streets, and green roofs (Ando and Netusil 2018). These alternatives to gray infrastructure have the potential to change the quantity and timing of stormwater runoff, improve water quality, reduce summer temperatures, provide pollinator habitat, reduce air pollution, and enhance aesthetics (Ando and Netusil 2018); numerous papers have valued the multiple benefits of green infrastructure (Schäffler and Swilling 2013; Mazzotta, Besedin, and Speers 2014; Netusil et al. 2014, 2022; Irwin, Klaiber, and Irwin 2017; Lamond 2017; Ando et al. 2020). A 2007 memo from the Environmental Protection Agency allowed cities to use green infrastructure to meet permit conditions and consent decrees (U.S. EPA 2007). Portland has embraced the use of green infrastructure and is acknowledged as a world leader (O'Donnell et al. 2021).

In 2018, the city updated its zoning code to require green roofs for all new construction having a net building area exceeding 20,000 square feet or larger in the Central City area, which has the greatest density of jobs and residents in the state (Bureau of Environmental Services 2018). As of 2019, there were more than 400 green roofs in the city of Portland, collectively covering almost 1.4 million square feet (Netusil and Thomas 2019).

2.2 Survey instrument

Numerous studies have quantified the private benefits to building residents from green roofs, for example, decreased energy costs and increased roof lifespan (Ando and Netusil 2018). However, the ‘public benefits’ of green roofs, that is, the multiple benefits to non-building residents, are less well known (Zhang, Fukuda, and Liu 2019; Netusil et al. 2022). Including public benefits can tip the scales in favor of a green roof program when conducting a cost-benefit analysis (Teotónio, Silva, and Cruz 2021), so accurate estimates are essential for sound policy decisions.

The survey instrument was developed using best practices detailed in Johnston et al. (2017). The project team visited numerous green roofs in the study area and interviewed experts as part of the survey design process. These conversations helped inform our initial selection of green roof attributes, which were further refined during three focus-group meetings. Meetings were facilitated by a professional moderator and included Portland residents who were recruited via Craigslist and compensated with a gift card to a local grocery store.⁴

Our study was designed from the beginning to test whether the image survey had a different effect on estimated coefficients, non-attendance, and time for survey completion compared to the text survey. Thus, we devoted the majority of the second and third focus-group meetings to refining the images used in our survey and invited the graphic designer who helped create the survey images to attend the final focus-group meeting. Focus-group participants were confused by images that were too realistic. For example, an early image used a realistic map of the study area to show where new green roofs would be located under different scenarios (Table 1). This caused focus-group members to look for where they lived and compare that to the locations of new green roofs instead of thinking about the general distribution of green roofs. The attribute levels used in our survey (Table 1) were also emphasized in our focus-group meetings with the selection of cost levels informed by recent increases in sewer and stormwater fees in our study area.

Our survey versions were identical except for the use of images in the section of the survey where attributes were described (Fig. 1 and Online Appendix A) and in the choice cards (Fig. 2). Individual choice cards were generated using experimental design techniques described in Kuhfeld (2010). An orthogonal fractional factorial experimental design for the five attributes and their different levels in Table 1 produced 96 unique choice cards. The experimental design was 100 per cent D-efficient.⁵ Respondents answered eight randomly selected choice cards with two-thirds of respondents randomly assigned to the image version of the survey and one-third to the text version. Respondents were presented with a choice of either a new green roof program or no program (‘status quo’). This two-option design is considered to be one feature of choice experiment design that promotes incentive compatibility (Collins and Vossler 2009), along with other features of survey and question construction that ensure respondents believe that their answers will have some impact on a decision or outcome that they care about (Carson and Groves 2007).

2.3 Data

We used a Qualtrics panel of respondents who live in Portland, Oregon. Project researchers specified demographic targets for age: 18–34 years old (≥25 per cent), 35–55 years old (≥25 per cent), 55+ years old (≥25 per cent), race (≤80 per cent White), and education (≤60 per cent bachelor’s or higher). The survey was administered in 2020, from September through early December.

Respondents who were determined to be speeding through the survey, or identified as a low-quality response based on failed attention checks, were deleted from the sample.⁶ After screening for protests and duplicate responses, we had 391 completed surveys for respondents who saw the image version and 200 for respondents who saw the text version of the survey. Each respondent was presented with eight randomly selected choice cards that included a new green roof program compared to a no-program (‘status quo’) option. Our final data set includes 6,172 choice tasks for respondents who saw the image version (after accounting for eighty-four skipped choice tasks) and 3,168 choice tasks for respondents who saw the text version of the survey (after accounting for thirty-two skipped choice tasks). Thus, we have 9,340 total choice tasks for analysis.

Respondents who completed the text survey selected the status quo option 28.22 per cent of the time; this decreases to 22.26 per cent for image version respondents. When we analyze the selection of status quo or new green roof program for our 391 respondents, 66.5 per cent of text survey and 58.57 per cent of image survey respondents selected the status quo option at least once.

Differences in respondent characteristics across survey types could account for differences in estimated willingness to pay. Our text and image respondents were extremely well matched (Table 2) with only two demographic characteristics, the percentage with a bachelor's degree or above (P-value = 0.0091) and employment categorized as ‘other’ (P-value = 0.0413), determined to be statistically different. City of Portland data are from the 5-year (2014–9) American Community Survey (U.S. Census Bureau 2019); some text and image survey respondent characteristics are statistically different from our study area (Table 2, columns 5 and 6), so care might be taken when generalizing our results. On the other hand, we do not find that demographic features (income, education, and age) are significant predictors of which category of preferences an individual belongs to, so these differences between our sample and the population of Portland may not compromise the external validity of our results. Our survey respondents were quite knowledgeable about green roofs with 71.5 per cent of text survey respondents and 67.5 per cent of image survey respondents reporting that they had seen, visited, heard, or read about green roofs prior to taking the survey.

2.4 Data analysis

2.4.1 Statistical models

Our statistical models are based on the linear random utility model, which assumes a respondent chooses the option that yields the highest utility when presented with a set of alternative options (Train 2009; Holmes, Adamowicz, and Carlsson 2017). We estimate mixed multinomial logit (MMNL) models, which allow for heterogeneous preferences. As is typical of MMNL models, we specify the utility individual i derives from alternative j in choice occasion t as

where |${p}_{ijt}$| is the cost of the alternative, |${x}_{ijt}$| is a vector of other alternative specific variables, |${\alpha }_i$| is the individual-varying price coefficient, |${\beta }_i$| is a vector of individual-varying coefficients, and |${\epsilon }_{ijt}$| is the unobserved part of utility. We apply the standard assumption that the error terms, |${\epsilon }_{ijt}$|⁠, are independent and identically distributed (IID) extreme value with |${\rm{Var}}\ ( {{\epsilon }_{ijt}} ) = k_i^2\ ( {\frac{{{\pi }^2}}{6}} ),$| where |${k}_i$| is the scale parameter. Dividing equation (1) by the scale parameter produces

where |${\lambda }_i = \frac{{{\alpha }_i}}{{{k}_i}}$|⁠, |${c}_i = \frac{{{\beta }_i}}{{{k}_i}}$|⁠, and the errors are IID Type I extreme value with |${\rm{Var}}\ ( {{\varepsilon }_{ijt}} ) = \frac{{{\pi }^2}}{6}$|⁠. We normalize |${k}_i$| to 1 so that the estimated coefficients are uncorrelated (Train and Weeks 2005).⁷ Given the distribution of |${\varepsilon }_{ijt}$|⁠, the conditional probability that any individual makes a series of choices |${\boldsymbol{n\ }} = \{ {{n}_1,\ \ldots ,{n}_T} \}\ $| over T choice occasions is

and the unconditional probability of the individual making the same series of choices is

where |$f( {\lambda ,c|\theta } )$| is the PDF of the parameter distribution conditional on estimable parameters describing the distribution, |$\theta $| (Train 2009; Hess and Train 2017). The parameters in c are assumed to follow a normal distribution, and |$\lambda $| is assumed to follow a log-normal distribution. We use the mixlogit command in STATA to estimate |$\theta $| (Hole 2007).

The specification of the MMNL model described in equations (1)–(4) is in preference space. Estimating MWTP for each attribute requires taking the ratio of the attribute's estimated coefficient and the normally distributed price coefficient implied by the distribution of |$\lambda .$| We use the delta method to obtain standard errors for these estimates. This estimation procedure can yield extreme MWTP values when a respondent's estimated cost coefficient is close to zero (Scarpa, Thiene, and Train 2008). An alternative to this approach is to estimate the model directly in WTP space, which uses simulations to estimate the attribute coefficients in dollar terms. However, it can be difficult for the WTP-space model to converge at a high enough number of simulation draws to ensure precise estimates. This was true for our model, which converged only at an unacceptably low level of simulation draws in WTP space.

2.4.2 Econometric specification

Our models include an alternative-specific constant (ASC) to capture any intrinsic preference that respondents may have for any new green roof program (regardless of its characteristics) and the effects of unobserved attribute preferences on choice (Rolfe, Bennett, and Louviere 2000; Kamakura, Haaijer, and Wedel 2001). To test whether our estimated coefficients vary by survey type, the five attributes of our green-roof programs (spatial distribution of green roofs; reduced summer temperatures; reduction in combined sewer overflows; increase in birds, bees, and butterflies; and cost) were interacted with whether a respondent completed the text or image version of the survey. The estimated coefficients, therefore, represent marginal utilities for the text and image treatments separately rather than baseline marginal utilities for one group and a differential for the other group.

The spatial distribution attribute is a set of indicator variables, with new green roofs ‘fully concentrated in the Central City’ used as the omitted category (Table 1). Reductions in yearly sewer overflows were presented in the survey as 0 (status quo), 1, 2, or 3, and are coded as such. The coefficient on this variable is the marginal utility of reducing sewer overflows by 1 per year. Increases in birds, bees, and butterflies are coded as 0 (status quo, no change), 1 (a 50-percentage point increase), 2 (a 100-percentage point increase), or 3 (a 150-percentage point increase). The coefficient on this variable represents the marginal utility of a 50-percentage point increase in pollinators. We coded reduced summer temperatures as levels with 0 (status quo, no change in summer temperatures), 1 (one-level change, corresponding to a <0.5°F decrease), 2 (two-level change, corresponding to a 0.5°F–1°F decrease), or 3 (three-level change, corresponding to a >1°F decrease). The coefficient on this variable approximates the marginal utility of a half-degree decrease in summer temperatures.^8,9All non-monetary attributes and the ASC are assumed normal, and cost is assumed log-normal.

3. Results

We present preference coefficient and MWTP estimates from the MMNL model in Table 3.¹⁰

3.1 Marginal WTP

Interacting the survey type (text or image) with each attribute allows us to see whether survey type has a statistically discernible effect on the magnitude of respondents’ MWTP estimates.¹¹ The marginal utility associated with the ‘concentrated mainly in Central City’ attribute was statistically significant at the 5 per cent level for respondents who answered the image survey but was not significant for respondents who took the text survey. The marginal utility for the other spatial distribution indicator, ‘distributed equally across Portland’ was not significant for either set of respondents. The estimated marginal utilities for the other attributes were individually statistically significant for both survey versions (Table 3).¹² Results are from a pooled sample with interaction dummy variables for the image and text treatments. This allows us to directly obtain the marginal utilities for each of the text and image treatments; results for separate text and image models are in Online Appendix Table B2.

We test if the estimates for the image and text treatments in Table 3 are equal to each other;¹³ the hypothesis of equal means was rejected at the 5 per cent level for all attributes (Table 4, column 3). Four of the estimated coefficients were significantly larger for the image version of the survey, and three were significantly smaller (Table 4, column 4).

The means in Table 4 were calculated using the coefficient (beta) distributions in Fig. 3. In each density plot, the blue area and line represent the text coefficient distribution, and the green area and line represent the image coefficient distribution. The image and text distributions overlap, except for ‘concentrated mainly in Central City’, although their different shapes, and especially their tail lengths, resulted in means that were statistically different for each attribute (Table 4). Interestingly, there is no consistent pattern in the variances across survey types.¹⁴

By far the most striking example of tail-influenced means is the coefficient on the cost attribute. The smallest values for the cost coefficient distributions are around −200 (text) and −4 (image) (Fig. 3, top left). However, the peaks of the cost coefficient distributions (Fig. 3, top right) largely coincide, but the long tail for the text survey results in a mean that is significantly lower than the image survey (Table 4).

3.2 Attribute non-attendance, certainty, and completion time

After selecting their preferred option (new green roof program or no program) for the eight randomly selected choice cards they were shown, respondents were asked, for each attribute, whether they never ignored, sometimes ignored, or always ignored that attribute. Around 4 per cent of text survey respondents always ignored the reduction in sewer overflow attributes compared to 3.6 per cent of image survey respondents (Table 5); 29.5 per cent of text survey and 22 per cent of image survey respondents reported sometimes ignoring that attribute.

A test of total attribute non-attendance, which equals the sum of ‘sometimes’ and ‘always’ ignoring that attribute, fails to reject, for all but the reduction in combined sewer overflows attribute, equal non-attendance for text survey and image survey respondents (Table 6). Other statistically significant differences across survey types include ‘sometimes’ ignoring the reduction in sewer overflows attribute (29.5 per cent for text and 22 per cent for image) and ‘always’ ignoring the cost attribute (8.5 per cent for text and 4.1 per cent for image).

Respondents were also asked to rank, on a scale from 1 (very uncertain) to 10 (very certain), how certain they were about their answers to all of the choice questions if they thought their answers would be used to decide whether to start a new green roof program and whether they thought they would have to pay for a new green roof program. There was no statistical difference in responses to these questions across survey types (Table 7, column 3). Perceived consequentiality—that is, the belief that a respondent's answer in the survey will be used to decide whether to start a new green roof program—was equal across survey treatments, and therefore not a primary source of the difference between TWTP values (Vossler and Watson 2013; Sandorf, Aanesen, and Navrud 2016; Welling, Zawojska, and Sagebiel 2022).

Although respondents who completed the image version of the survey took longer (average total completion time of 13.2 min for image and 12.2 min for text), which is often seen as an indication of higher-quality responses (Börger 2016; Campbell, Mørkbak, and Olsen 2018), the mean total completion time was not significantly different across our two survey versions (P-value = 0.4788).

The analysis was extended to account for stated attribute non-attendance, certainty, and consequentiality. Attribute non-attendance was incorporated by assigning a marginal utility of zero to an attribute if a respondent said they always ignored that attribute. We dropped respondents if they selected a value of 3 or lower, on a scale of 1 (very uncertain) to 10 (very certain), for their answers to ‘Certain their answers will be used to decide whether to start a new green roof program’ and/or ‘Certain that they will have to pay for a new green roof program.’ We placed no restrictions on the values for respondents’ answers to ‘Certain about their answers to the choice questions’ (Table 7). The estimated preference coefficients and MWTP values are very similar to those in Table 3; TWTP results are in Online Appendix Tables B10 and B12.¹⁵ We also test for significant differences in average TWTP between the model correcting for attribute non-attendance and the model correcting for both attribute non-attendance and consequentiality and find no difference (Online Appendix Tables B11 and B13).

4. Discussion

Respondents’ TWTP for a new green roof program was calculated for two scenarios. The first example is a ‘lowest effects’ program, where new green roofs are concentrated fully in the Central City, summer temperatures are reduced by <0.5°F, there is one fewer combined sewer overflow, and the number of birds, bees, and butterflies increases by 50 per cent. The second example is a ‘highest effects’ program, where new green roofs are concentrated mainly in the Central City, summer temperatures are reduced by >1°F, there are three fewer combined sewer overflows, and the number of birds, bees, and butterflies increases by 150 per cent. Table 8 includes TWTP estimates for these programs at the household level and scaled to the citywide level for text and image survey respondents.

In both scenarios, the estimated effect for each attribute is higher for respondents who saw the image survey compared to respondents who saw the text survey. Estimated TWTP per household for the lowest effects program is $86.32 (text) and $194.80 (image), a difference of 126 per cent. The difference is even larger (178 per cent) for the highest effects program, which has an estimated TWTP per household of $153.48 (text) and $426.73 (image). Our results are consistent with Shr et al. (2019), who find that the use of images and text in their choice experiment survey generated WTP values for some attribute levels that were much higher than when they used only text. Importantly, when scaled to the citywide level (Table 8), the difference between survey versions is likely large enough to influence program outcomes.

This difference in our TWTP results seems to be driven by two factors. The largest difference in attribute values is for reduced combined sewer overflows, which has significantly lower non-attendance for respondents who took the image version of the survey (Table 4). The second factor is the difference in the cost attribute (Table 4 and Fig. 3). Recall that in the MMNL model, the MWTP value is calculated by dividing the estimated coefficient of an attribute by the cost coefficient, which represents the marginal utility of money. The lower estimated coefficient on cost (i.e. the denominator) for respondents who saw the image version of the survey produces MWTP values (Table 3) and TWTP values at the household and city level (Table 7), which are much higher than for those who were shown the text version of the survey. This implies something about the use of images overall, that is, images appear to have caused respondents to have a smaller disutility from paying for a new green roof program.

Images convey information and have more impact on survey responses if a respondent is unfamiliar with the program being valued (Welling, Zawojska, and Sagebiel 2022). We explore this effect by separating our respondents into a ‘high knowledge’ group, which includes respondents who said they had visited, seen, heard, or read about green roofs, and a ‘low knowledge’ group that selected ‘No’ or ‘Maybe’ when asked if they had visited, seen, heard, or read about green roofs.¹⁶ We expect the difference in TWTP between the image and text versions to be larger for the low-knowledge group than the high-knowledge group if images are conveying additional knowledge about the attributes and program; the results in Table 9 are consistent with this expectation.

At the household level, the difference in TWTP between the image and text versions is 223.04 per cent for low-knowledge respondents and 78.33 per cent for high-knowledge respondents for the low effects program and 288.60 per cent (low knowledge) and 127.05 per cent (high knowledge) for the high effects program (Table 9).¹⁷ Another interesting outcome is that image survey respondents with low knowledge have, on average, a higher TWTP than high-knowledge respondents for both scenarios. The opposite occurs for text version respondents, that is, TWTP values are lower, on average, in both program scenarios for low-knowledge respondents compared to high-knowledge respondents.

We also explored whether a respondent's demographic characteristics, such as education, age, and income, influence a respondent's TWTP (Online Appendix Tables B17–B19).¹⁸ Consistent with our other findings, image survey respondents have a higher TWTP than text survey respondents. Findings include a higher TWTP for respondents in Education Category 1 (some high school, high school diploma or GED, some college, associates degree, technical school, and other) compared to Education Category 2 (bachelor's, master's, professional degree beyond bachelor's, and doctorate). The largest difference between image and text survey respondents in our demographic models is for Education Category 1 (Online Appendix Table B17).

We ran a latent class model with two classes that included respondents’ prior knowledge about green roofs and demographic characteristics as class participation predictors (Online Appendix Table B20). Respondents with high knowledge were significantly more likely to belong to Class 1; respondents’ demographic characteristics were not significant class participation predictors (Online Appendix Table B21). For both the text and image respondents, Class 1 had significantly larger preference coefficients for the ASC (the dummy for there being a green roof project rather than the status quo) and cost attribute (because it is less negative), which drives the difference in TWTP across classes (Online Appendix Tables B22–B23). Given the relationship between assignment to Class 1 and prior knowledge, it seems high-knowledge respondents are more likely to choose any green roof project over the status quo regardless of the project's attributes. They are also more likely to be willing to pay more for the project. The latent class model offers little evidence of a significant difference in TWTP between text and image survey respondents apart from TWTP for the highest effects program among respondents in Class 2. Because high-knowledge respondents are less likely to be in Class 2, this may suggest that differences in TWTP between the image and text surveys are driven by the effect of images on respondents who had not visited, seen, heard, or read about green roofs prior to taking the survey.

5. Conclusion

This paper explored, using a split-sample design, whether the inclusion of images, instead of just text, in a choice experiment survey of green roofs in Portland, Oregon, affects estimated coefficients, attribute non-attendance, and elapsed time for survey completion.

While the existing literature on information presentation effects is mixed across studies, our study's findings are very clear. Including images in the background section of our survey and in the choice cards increased the number of statistically significant coefficients and significantly reduced attribute non-attendance for one of our green roof attributes. Elapsed time for completing the image version of the survey was longer, which is often seen as an indication of higher-quality responses, though the difference in elapsed time was not statistically significant. The TWTP for respondents with little prior knowledge about green roofs was over three times higher when they were assigned to complete the image survey than the text survey. A latent class model provides more evidence that images increase mean estimated TWTP for a green roof project by increasing the estimated values of people with less prior familiarity with the good. Because images also reduce attribute non-attendance for one of the features of the green roof projects, the effects estimated by the image version of the survey may be closer to respondents’ true values.

Future research should explore whether static images systematically influence the choices, certainty, consequentiality, and non-attendance of respondents who are relatively less knowledgeable about the study's subject matter, thereby extending research by Sandorf et al. (2016) that explores the effects of survey mode for unfamiliar and complex environmental goods. Images may also be important for respondents with different demographic characteristics, for example, age or language abilities. Other potentially important factors could include differences in respondents’ opportunity costs of time spent on the survey or if respondents are not part of a survey panel. The significant difference in the cost attribute and its effect on MTWP and TWTP deserve further investigation. The marginal disutility of spending money on a project was significantly smaller for respondents who saw the image version of the survey, but the factors that account for this difference are not clear.

Ultimately, the goal of survey design for choice experiments is to provide respondents with enough information to allow them to express, in an informed way, their true preferences so that accurate estimates of willingness to pay can be calculated to guide policies. Our findings support the trend observed in the literature of incorporating static images into choice experiments, although we encourage researchers to focus on the quality of those images. Future research using choice experiments for non-market valuation might take care to ensure that focus groups include people with varied levels of knowledge about the good to be valued and gather information about knowledge in the survey instrument itself. Additionally, meta-analyses that carry out the important work of combining results from multiple choice experiment studies to derive robust values for a good might control for whether images are included in the survey instrument, since our findings suggest that images can improve the validity of value estimates for a good that is not well known to the general public.

Acknowledgments

We are grateful to Paul Manson, Issac Wimer, Ben Thomas, Olyssa Starry, Tom Liptan, Trudy Cameron, and Bryan Parthum for their help with this project. Helpful feedback was received from participants at 2021 W-4133 and 2022 AERE-WEA meetings.

Funding

Support for this research comes from the Bernard Goldhammer Grant for Research on Economics and Natural Resources, Reed College and the Stendal Fund for Economics, Reed College. This paper is also based in part on work funded by the USDA-NIFA W4133 Multistate Research Grant 1008843.

Data availability

The data and code that support the findings of this study are available in a zipped supplementary file.

Footnotes

References