The effect of conditional incentives in an email–web valuation survey of recreational anglers

Abstract

Web surveys often suffer from low response rates when invitations are sent via email. We conducted an experiment testing the effects of a $5 Starbucks gift card incentive on response rates and survey results from an email–web survey of Gulf of Mexico recreational anglers. Half the sample was randomly assigned to be offered the incentive for completion of the survey. Despite this, response rates were nearly identical between the incentivized (12 percent) and non-incentivized (13 percent) groups. Respondent demographics, fishing trip characteristics, and econometric estimates were highly consistent across groups. Statistical tests could not reject the null of no difference between the incentive treatments. Our results suggest foregoing small conditional non-monetary incentives can be cost-effective for this type of specialized recreation survey using an email protocol. However, generalizability to other populations, incentives, and protocols requires further research. The non-monetary incentive did not meaningfully impact response rates or estimates in this context.

1. Introduction

Incentives are often used to increase response rates and improve the quality of responses in surveys. There is considerable evidence suggesting that incentives work well when, for example, they are included in mail invitations to complete a survey on paper or on the internet as part of a mail-push strategy (Edwards et al. 2023). Less is known about the effectiveness of conditional incentives on response rates and quality in online-only surveys that make all contacts via email. Conditional incentives are popular because they are paid out after the completion of the survey task and are, therefore, less expensive than up-front unconditional incentive payments.¹

We are focusing on a very specific, yet common, type of survey protocol that makes all contacts via email, including the initial invitation to complete the survey on the internet. This approach is a relatively fast and inexpensive way to collect data. However, the response rates tend to be low, especially when contacting people who are not expecting the email. A common way to incentivize participation is to offer a voucher or gift card upon completion of the survey.² There is little published research examining the relative efficacy of this popular survey recruitment approach.

A recent review found evidence that monetary and non-monetary incentives increased response rates to online epidemiological surveys based on around twenty field experiment trials (Edwards et al. 2023).³ According to the review, however, it did not matter whether the incentive was delivered before (unconditional) or after (conditional) completing the survey. Looking more closely at the experiments included in Edwards et al. (2023), there is only one study that follows the email-only, conditional incentive strategy. Hathaway et al. (2021) found that cancer patients who received an email invitation promising a $10 gift card for completing an online survey were about twice as likely to respond than those who received an email invitation without the gift card promise.

There are a few field experiments not included in the above review that also followed a variant of the email invite, conditional-incentive protocol. Brown et al. (2016) found that a post-paid (conditional) incentive increased the response rate by about 14 percent in a sample of health care system clients.⁴ Note that the incentive was mailed and the respondent was given the option of a cash payment or gift card. Most respondents chose the cash. For the most part, the characteristics of study participants and the response to survey items were not statistically different between the treatment groups.⁵ In another field experiment, DeCamp and Manierre (2016) found that a low incentive ($2) did not change response rates, but that a higher incentive ($5) did increase response rates by around 40 percent in a sample of college students. The different incentive treatment groups were similarly representative of the population, consistent with administrative records at the college.

In contrast to the results of the field experiments, a recent meta-analysis model including more than 1,000 online education-related surveys found that, in general, incentives were not significantly correlated with response rates (Wu, Zhao, and Fils-Aime 2022). This result of no incentive effect on response rates was consistent with results reported in an earlier meta-analytical study of factors explaining response rate variation in online surveys (Cook, Heath, and Thompson 2000). Note, however, that the meta-analysis modeling approach cannot say anything regarding the causal effect of incentives on response rates.

We consider the implications of incentives in a valuation survey of recreational anglers, which is an important real-world setting because estimates of the value of recreational fishing are frequently used in policy analysis. For example, NOAA Fisheries analyzes the economic effects of every proposed change in saltwater fishing regulations and includes predictions of the change in economic value when the requisite data are available (National Marine Fisheries Service 2007). Studies to generate new estimates of economic value are costly and time consuming. It is important to find the most cost-effective survey (or other) approach to generate estimates of value that are relevant for policy analysis. Furthermore, we need to understand whether incentives change survey responses or, more crucially, the resulting estimates of sportfishing demand parameters and surplus measures.

Other studies have investigated the role of incentives in angler surveys, but none, to our knowledge, consider the effect of conditional incentives in email-only surveys. For example, NOAA's Fishing Effort Survey includes $2 along with the survey instrument in the invitation mailing because experiments found that the incentive increased the odds of responding by more than 90 percent (National Marine Fisheries Service Office of Science and Technology 2023, Andrews 2014). Similarly, Anderson et al. (2022) randomly assigned licensed anglers in the western United States to receive various cash incentives in a mail invitation to complete an online screener survey. They found that a $2 incentive increased response rates by more than 50 percent and a $5 incentive nearly doubled the response rate relative to no incentive. In an earlier analysis using the same survey form used in the present study, we found a similar doubling-of-response-rate result with a $2 mailed incentive in a mail-push strategy compared with an email-only survey with no incentive (Carter et al. 2024).^5,6

We present the results of a random trial of the effect of conditional incentives in an email–web survey on response rates, sample characteristics, and angler demand estimates. Importantly, our trial considers a “cold” survey invitation with a conditional incentive, i.e. paid after the completion of the survey. Our results suggest that the surveys with and without incentives achieve similar response rates, and similar results for sample characteristics, demand model parameters, and measures of economic value. Therefore, an email–web survey that forgoes such incentives appears to be a more cost-effective solution.

2. Methods

2.1 Survey background

The fall 2023 version of the Florida Boating and Fishing Survey (FBFS) was conducted by the US National Atmospheric and Oceanic Administration (NOAA) to obtain information about Florida anglers’ fishing activity in the Gulf of Mexico (GOM).⁷ We administered the survey in November of 2023 to collect information about recreational fishing in the GOM during September and October of 2023. The target population for the FBFS was any person who might have used a private boat to fish offshore, specifically those who fished for Gag Grouper.⁸ We focused on Gag Grouper anglers because the policy-based contingent behavior (CB) questions in our survey (described below) concerned Gag Grouper. Around 6 percent of all trips by private boat anglers fishing from the west coast of Florida in 2022 targeted or caught Gag Grouper. In federal waters, the share of private boat angler trips fishing for Gag Grouper was even higher at around 21 percent.⁹

The recreational harvest of Gag Grouper in the GOM is managed with fixed seasons and bag limits. The bag limit has been two fish per angler since 2009, but the seasons varied considerably until 2016 when the season in federal waters was set to open in June and continue through the end of the year.¹⁰ In 2023, the Gag season closed early on October 19th. These regulations and related regulations in the commercial sector were implemented to protect the Gag Grouper stock, which was in decline during the early 2000s.

2.2 Survey questions

The FBFS was programmed in Qualtrics with questions designed to improve the prediction of changes in angler effort and economic value anticipated with changes in Gag Grouper bag limits and seasons. There were two main sections of the survey following a question confirming boat ownership and questions regarding the type of boat usage during September and October. For the respondents who used their boat for fishing, the first section asks a series of questions related to fishing activity during September and October. Specifically, respondents were asked to report the number of trips taken in September and October and the total cost paid by all anglers on a typical trip. We also asked for the duration of and the number of anglers on board a typical trip.

The second section of the survey contained two types of CB questions that asked respondents to report the number of trips they would have taken in September and October if fishing costs or Gag Grouper regulations were different. This is a type of reassessed contingent behavior trip question format that asks anglers to reassess how many trips they would have taken if hypothetical trip costs or Gag Grouper regulations had been in place (Simoes, Barata, and Cruz 2013). The full set of CB question scenarios is summarized in Table 1, where the first row represents actual conditions in September and October, the second two rows represent the cost (price) scenarios, and the last three rows represent the Gag Grouper bag limit scenarios. There are two sources of variation in the scenarios when collected for a set of anglers: (1) across anglers and (2) across scenarios within one angler.

The first CB question in the survey (row 2 of Table 1) asks for the number of trips that would have been taken if the cost had been double the cost of a typical trip and the second CB question (row 3) asks for the number of trips if the cost were half that of a typical trip. Note that we used piping on the internet survey to present respondents with trip cost alternatives that were double or half their actual reported costs.

The other three CB questions (rows 4 through 6) ask for the number of trips that would have been taken if the bag limit were three fish, one fish, or zero fish (closed season). These questions were only shown to those who reported fishing for Gag Grouper during September or October and stated that they might have taken a different number of trips if Gag Grouper regulations had been different. Note that the hypothetical regulation questions ask the angler to consider changes in the number of all of their trips, not just their trips that targeted Gag Grouper.¹¹ For the analysis, we set the trips in the Gag Grouper regulation scenarios to the actual trips for those who stated that they would not have changed their trips under different Gag Grouper regulations.

2.3 Survey sampling and incentive experiment

The FBFS made all contacts (invitations, reminders, etc.) via email. There is no specific email list of Florida anglers fishing in the GOM for Gag Grouper. Therefore, we focused on anglers fishing from the west coast of Florida because nearly all of the recreational harvest of Gag Grouper originates from this area. We further narrowed our interest to boat-based anglers because Gag Grouper are primarily located around offshore reefs, which can only be reached by boat.

The State of Florida administers the State Reef Fish (FSRF) license, which is required to fish for reef fish, even for those who are over 65 and would not otherwise require a Florida saltwater fishing license. The FSRF license designation allowed us to more directly target the reef fish angling population of interest.

The State of Florida categorizes each record in the FSRF database based on county of residence and boat ownership. We narrowed the FSRF to boat owners with emails from the six strata likely to contain anglers fishing in the GOM.¹² This subgroup comprises about 10 percent of the more than 800,000 FSFS licenses outstanding. A map of the sampling strata is shown in Fig. 1. We sampled 7,600 records, with the number of records for each stratum drawn in proportion to the relative share within the population subgroup. The 7,600 sample amount was selected given a target of 400 respondents who could fill out the Gag Grouper portion of the survey, an assumed response rate of 21 percent, and a Gag Grouper fishing prevalence rate of 25 percent based on previous work with the FSRF database and a similar sampling frame. Note that previous experience with the FSRF database actually achieved a response rate of around 16 percent. However, in planning for this survey we assumed that incentive (discussed below) would achieve an additional five percentage points in the overall response rate based on the literature reviewed in the introduction. Also, our target of 400 surveys completed (200 in each group) was based on a simple power analysis for a comparison of the incentive treatment group means using a t-test with a significance level of .05 to detect an effect size of 0.3 with a power of 0.8 (Champely 2020).

We randomly assigned the 7,600 records in the FSRF sample to the incentive or no-incentive treatment groups (3,800 each).¹³ The text in the invitation email was identical between the two groups, with the exception of the second sentence, which read “(w)e will send you a $5 Starbucks gift card for carefully completing the 5 minute survey” for the incentive group and read “(t)he questions should take about 5 minutes” for the no incentive group.

Both groups received three reminder emails, approximately two days apart following the initial invitation. The text of the reminder emails was also identical except for the language about the gift card incentive.

2.4 Data analysis

2.4.1 Trip demand model and evaluation measures

Details regarding the derivation of the recreational fishing trip demand model can be found in Carter et al. (2022). For the purposes of this analysis, we assume that the trips follow a Poisson distribution and estimate the following fixed-effect trip demand model for the number of trips d_ijselected by angler i for scenario j:

where p_ijis the trip cost and associated parameter γ, r_ijis the bag limit with parameters δ and λ, and α_iis an angler-specific fixed effect.¹⁴ We include an indicator, h_ij, for the hypothetical scenarios and interact the indicator with the trip cost variable.¹⁵ The parameters on the hypothetical indicator, θ and ϕ, are meant to capture the differences in the unmodeled factors that affect trips reported in the hypothetical scenarios (Englin and Cameron 1996; Haab, Sun, and Whitehead 2012). For example, the hypothetical indicator could measure errors on the part of the respondent. The internet survey reminded the respondent how many trips they took in the base case before each hypothetical scenario question. However, respondents could have made an error (e.g. recording or recall) such that the expected trips over the hypothetical scenarios at the baseline cost and bag limit do not equal the actual trips. The parameters associated with the dummy variable designating the hypothetical scenarios should capture this error.

With the Poisson fixed-effects estimator, the unobserved factors represented by the fixed effects can be correlated with p, r, or h without biasing the corresponding parameters. This is important because the angler response to changes in trip costs and bag limits is likely to be related to angler characteristics or fish stock conditions not included in the model.¹⁶

There are several other important measures that we calculate to compare between the two incentive treatments. The first type of measure is a semi-elasticity, which measures the percent change in trips expected with a unit change in trip cost or bag limit, all else equal. The average trip cost semi-elasticity is simply the parameter γ on this variable. The average semi-elasticity for the bag limit in scenario j is slightly more complicated because there is a squared term: ϵ_j= δ + 2λr_j.

The other measures we compare between the incentive treatment groups are the value of a fishing trip and the change in value of fishing expected with a change in the Gag Grouper bag limit (Haab and McConnell 2002). The negative of the inverse of the trip cost parameter gives the expected value of a fishing trip, i.e. CS = −1/γ.¹⁷ Similarly, the change in value per trip associated with a change in the bag limit is expressed as MWTP_j= −ϵ_j/γ, which measures the marginal willingness-to-pay (MWTP) for each bag limit increment.

2.4.2 Incentive treatment comparison

We consider several ways to formally compare the results between the two data collection treatments. First, we use χ² tests to compare sample distribution (opened, started, finished, etc.) proportions and geographic stratification proportions between the treatment groups. Second, we use standard t-tests to compare the means of the key variables for both the sample overall and the Gag Grouper angler subset.¹⁸ Then, also on the Gag Grouper subset, we use a bootstrap-based approach to compare the trip demand model parameters (γ, δ, λ, ϕ), elasticities, ϵ, the value per trip (CS), and the MWTP for bag limit changes per trip estimates, MWTP.¹⁹

Specifically, we focus on the Gag Grouper angler subset and use a cluster bootstrap procedure (Cheng, Yu, and Huang 2013; Cameron and Miller 2015) in combination with the method of convolutions (Aizaki 2015; Poe, Giraud, and Loomis 2005) and an overlap index (Pastore 2018). First, we use the cluster bootstrap to obtain empirical distributions for the results of interest, denoted β_t= (γ_t, δ_t, λ_t, ϕ_t, ϵ_t, CS_t, MWTP_t) for t = no − incentive, incentive, as follows:

1. Sample anglers (respondents) with replacement N_ttimes from the original sample of anglers.

2. For the sampled N_tanglers, retain the observations for all six scenarios.

3. Obtain estimates of β_twith the fixed-effects Poisson regression using the 6 × N_tobservations.

4. Repeat steps 1, 2, and 3 B times to obtain B bootstrap estimates of β_t.

Note that the resampling is done over anglers, rather than over scenarios. In this way, some anglers may not appear in bootstrap samples at all while other anglers will appear multiple times. The result of the bootstrap simulation gives vectors of length B for each element in β_tfor each incentive treatment group t. We then use the method of convolutions to test the null hypothesis of estimate equality (Aizaki 2015; Poe, Giraud, and Loomis 2005). Specifically, we apply the method of convolutions to corresponding vectors in β for each treatment group. For example, to evaluate the null of equality of the value per trip estimates for the no incentive and incentive groups, we apply the method of convolutions on the vectors CS_{no−incentive}and CS_incentive. Lastly, we evaluate the overlap of the two bootstrap distributions for each result of interest using plots and an overlap index (Pastore 2018). The overlap plots and indices offer another way to visualize and compare the distributions of the economic measures estimated using the two incentive treatment groups.

3. Results

3.1 Sampling results and sample characteristics

The final disposition of the samples is shown in Table 2. Of the 3,800 emails sent to each treatment group, the general return profile is very close between the two treatment groups. However, a test of equality for the status proportions in each group can be rejected at the .05 level using a χ² test, which is unsurprising given the large sample size. It is notable that more than a third of emails were not even opened and that roughly half of those who opened the email did not start the survey. This was true whether or not the incentive was offered. Most people who started the survey finished. The response rates are consistent with other angler surveys employing similar sampling strategies (Wallen et al. 2016). Note, however, that we were not able to obtain a higher response rate using the incentive, with roughly 15 percent of delivered surveys at least started in both treatments. The distribution of the responses is shown by strata in Table 3 by treatment along with the disposition of the original sample, which was drawn to be representative of the population. Both treatment groups are similarly distributed among the strata and match the sample/population distribution relatively well.

Table 4 shows the means of respondent characteristics by treatment along with the P values for t-tests of mean differences.²⁰ Nearly 60 percent of respondents used their boat during the study period and roughly half used their boat to fish in the GOM. Furthermore, around 20 percent of respondents stated that they “fished for Gag Grouper” in the GOM during the study period. Note, however, that we did not obtain the target number (400) of Gag Grouper anglers because incentives did not provide the five percentage-point increase in the response rate overall that we had assumed.²¹

The results in Table 4 are very similar between the two groups, with only the difference in the mean number of trips showing as statistically different from zero according to the P values. The difference in trips suggests that those who were not offered the incentive reported taking slightly more trips during the study period. However, the standard error of the mean estimated number of trips was considerably larger for those who were not offered the incentive. This result could also reflect the idea that relatively less avid anglers were induced to participate by the incentive.

Table 5 presents the means for respondents who said that they fished for Gag Grouper in the GOM during the open season. This is the population of interest for the analysis of the effect of changes in Gag Grouper regulations. With the exception of trip hours, all angler and trip characteristics are similar between the treatment groups with no statistically significant differences. Anglers who fished for Gag Grouper in the incentive group reported slightly shorter trips on average than anglers who fished for Gag Grouper in the group that did not receive the incentive.

Table 5 also shows the summary statistics for all of the hypothetical trip scenarios shown in Table 1. In general, the results are as expected with higher trips reported at lower costs, lower trips at higher costs, and trips increasing in the Gag Grouper bag limit offered. We cannot reject the null of equality of the hypothetical trips between the treatment groups for any scenario.²²

In Fig. 2, we illustrate the similarity in trip demand between the incentive treatment groups by plotting the piecewise linear trip demand relationships based on the mean trips reported in the actual, cost doubling and cost halving scenarios for the Gag Grouper anglers. Consistent with the results in Table 5, the curves for the two treatment groups are relatively close together.

3.2 Fishing trip demand and value

The estimated parameters of the trip demand regressions are shown in Table 6 for the sample of Gag Grouper anglers. Note that the number of observations is given by the number of Gag Grouper anglers reported in Table 5 times 6 for the number of scenarios shown in Table 1. We use cluster-robust standard errors to adjust for the fact that the multiple observations from the same individual are likely to be correlated. These adjusted standard errors account for both overdispersion and correlation over choices for a given angler (Bergé 2018).

A detailed discussion of the regression model parameters and outputs shown in Table 6 is beyond the scope of this analysis. We focus here on the key model outputs that are important for policy analysis and how these estimates compare between the incentive and no incentive treatment groups. The trip cost parameters, γ, are identical up to three decimal points for the two treatment groups.²³ These parameters represent the percent change in trips with a unit change in trip cost for the average angler who targeted Gag Grouper. The negative of the reciprocal of the travel cost parameter measures the CS or value per trip for the average angler (Haab and McConnell 2002), which is around $250 as shown in Table 6. The CS per trip estimates are relatively close between the treatment groups. We consider more formal comparisons below.

The trip response of anglers to bag limit changes is formally measured in Table 6 as bag limit semi-elasticities, i.e. the percent change in trips with a unit change in the bag limit. We use the semi-elasticity expressions for the bag limit changes presented earlier to calculate the bag limit semi-elasticity starting from zero, one, and two fish. Generally, the percent change in trips with a unit change in the bag limit increases at a decreasing rate with each bag limit increment for the group that received the incentive, but seems relatively stable overall and for the group that received the incentive.

The last set of measures shown in Table 6 is the estimates of the marginal MWTP or CS_jfor a change in the bag limit. Similar to the semi-elasticities, the MWTP is relatively stable at around $26 over different bag limits overall and for the group that did not receive the incentive. The MWTP appears to be linearly increasing for the group that received the incentive.

3.3 Survey mode comparison analysis

The results of the bootstrap analysis are shown in Fig. 3. The plots show the bootstrapped distributions by incentive treatment for each parameter to visualize the extent of the overlap. The distributions for most of the parameters and measures overlap. However, the distribution of the trip cost parameter in the incentive group shows relatively more variance. This higher variance is also seen in the CS per trip and, to a lesser extent, the MWTP for bag limit estimates for the incentive group.

The overlap for each parameter is numerically evaluated in Table 7 where we give a formal measure of the percentage of overlap (Pastore 2018) and the P values based on the method of convolutions for the null hypothesis that the means for each treatment group are equal. We also show the average of the ratio (mean ratio) for the estimates between the two modes. Consistent with the overlap plots, the trip cost parameter has a high level of overlap, which is also apparent in the CS per trip because CS is simply the negative of the reciprocal of the trip cost parameter. Furthermore, we cannot reject the null hypothesis that the CS per trip is equal between modes at a .05 significance level.

There is also a relatively high level of agreement in the plots and overlap measures for the bag limit parameters and the bag limit semi-elasticities and MWTP measures. None of the parameters and measures of interest are statistically different between modes at the .05 significance level. All of the measures in the table demonstrate more than 60 percent overlap between treatment groups.

4. Discussion

We collected data on recreational fishing trip behavior from anglers via email invitations to a web survey. Half of the sample was randomly assigned to be offered a conditional, non-monetary incentive (gift card) for completing the survey.

There was little difference in the response rate between the incentive treatment groups, and summary statistics for more than ten different variables were very close between the treatment groups. The estimated parameters of the same recreational fishing trip demand model specification estimated using data from each treatment group were also very similar according to a bootstrap analysis that compared distribution plots, formal measures of overlap, and P values.

Both treatment groups directed respondents to complete the survey on the internet via email and the survey protocol was identical between the groups. Therefore, the costs of the web survey development and administration were the same for both survey strategies. The only differences in costs were the expense to administer the gift cards and the total face value of the cards. Based on our results, these additional costs, amounting to nearly $4,000 in our case, do not appear warranted.

More research is necessary to determine the extent to which our findings can be generalized. For example, there may be other domains where conditional incentives to complete web surveys offered via email work to increase response rates and the quality of survey data. Our results could also depend on the type and amount of incentives offered, such as the $5 Starbucks gift card used in our study.²⁴ Research with other modes (e.g. mail) has found that unconditional cash incentives perform better than “non-monetary” offers and that higher incentives tend to generate higher response rates (Edwards et al. 2023). Conditional cash payments may work better than conditional non-monetary incentives. Unconditional gift cards could be explored as well. We had success in improving response rates with an unconditional cash incentive as part of a mail-push strategy, albeit with little difference in response quality (Carter et al. 2024). Others have found similar results with a mail-push-incentive strategy (Anderson et al. 2022). However, unconditional incentives for email-to-web surveys could be expensive and even wasteful if the improvements in response rates and survey quality are not significant.

Acknowledgments

We would like to acknowledge the help from the Florida State Reef Fish Survey Program in securing the sample for this study and thank the anglers who responded to the survey.

Conflict of interest

None declared.

Funding

None declared.

Data availability

The data underlying this article will be shared on reasonable request to the corresponding author.

Footnotes

Appendix A. Sample frame counties

Appendix B. Brunner–Munzell permutation tests

Results from the permutation version of the Brunner–Munzel (BM) test are reported in Table B1 as another way to compare the distributions of the key variables between the incentive and no incentive treatments for the respondents who fished for Gag Grouper. According to Noguchi et al. (2021), the permutation version of the BM test is robust and powerful for small sample sizes, under a variety of assumptions. In our case, the BM test evaluates the null hypothesis that the probability of observing a given variable level is the same regardless of the survey mode. Specifically, the null hypothesis is based on the relative effect: π = P(X < Y)+ |${\textstyle{1 \over 2}}P( {X = Y} )$|⁠, where X and Y represent observations from two independent groups. If π = 0.5, then the two groups are considered stochastically comparable, which is the null hypothesis of the BM test. Rejecting the null with π > 0.5, for example, suggests that a randomly drawn individual from the Y group will have a higher value for the variable of interest than a randomly drawn individual from the X group. We use the BM test to compare the following variables among the email-only, mail-push-incentive, and non-response groups: respondent age, respondent income, and the number of days fished (trips), trips with costs doubled and halved, and trips with a three-, one-, and zero-fish bag limit (relative to a two-fish bag limit). All BM tests are conducted with the rank.two.samples function in the rankFD R package (Konietschke et al. 2022). We use 10,000 permutations to calculate the relative effect along with its standard error and 0.95 confidence interval. Range-preserving confidence intervals are obtained using the logit transformation. The studentized permutation test is used to test the equality of the two distribution functions of the two groups with a two-sided alternative.

References