COPAFS
 

The User of Incentives in Surveys Supported by Federal Grants[1]

Sandra H. Berry, Jennifer S. Pevar, and Megan Zander-Cotugno
RAND Corporation, Santa Monica, CA

Introduction

In 2001 the U.S. federal government provided $42.87 million in funding for basic and applied research, of which approximately half was provided through the Department of Health and Human Services (DHHS). Almost all the remaining funds were awarded by the National Science Foundation (NSF), the Department of Agriculture, the Department of Defense, the Department of Energy, and the National Aeronautical and Space Administration. Within DHHS, 99% of research funding is awarded through the National Institutes of Health (NIH).

Research grants are highly sought after and very competitive. At NIH and NSF grant proposals go through internal review and peer review by panels of 10-20 experts. Reviewers are not blind to the identities of principal investigators or institutions, since investigator qualifications and institutional capabilities are an important part of the review criteria. In addition, grants are evaluated on whether they advance knowledge within and across fields, their creativity or innovativeness, and the soundness of their approach or methods. Receiving awards is important for investigators in terms of tenure decisions and support for “soft money” positions. In 2001-2003 NIH received 132,368 proposals from 61,147 investigators who came from for-profit organizations, research institutions, hospitals and medical schools, and institutes of higher education. Excluding a small number of very large awards, the average award size was about $400,000 and fewer than 30% of the applicants received awards. Nearly all applicants for NIH funding hold Ph.D. or M.D. degrees (Hosek et al., 2005). All of the agencies that provide basic and applied research funding undertake survey research, although it would be very difficult to determine how much funding is allocated specifically to this purpose within each agency.

In the following sections we first discuss some background factors that affect federal grants with respect to their use of incentives in surveys, including regulation, expectation for journal publication, budgets, and staffing arrangements. Investigators working on federal grants find themselves under competing pressures and constraints from their Institutional Review Boards (IRBs), the journals where they want to publish, their budgets, and the capabilities of their project staff. We then discuss survey researchers’ perspectives on use of incentives and our findings from a recent Internet survey of NIH grant recipients about their use of incentives in surveys.

Background Factors Affecting Use of Incentives in Grant Funded Surveys

Federal Regulations

Survey research funded through grants is not covered under the Paperwork Reduction Act and surveys conducted under federal grants are therefore not subject to the requirement for Office of Management and Budget clearance, so OMB regulations or guidelines on appropriate methods are not directly applicable (Proposed Standards and Guidelines for Statistical Surveys). However, institutions that receive federal funding are covered by Title 45 CFR (Code of Federal Regulations) Part 46 which requires that research on human subjects conducted at institutions that receive federal funding be reviewed and approved by an IRB established for the protection of human subjects from research risks. In order to approve research covered by this policy the IRB must determine that all of the following requirements are satisfied (Code of Federal Regulations, 2005):

  1. Risks to subjects are minimized: (i) By using procedures which are consistent with sound research design and which do not unnecessarily expose subjects to risk, and (ii) whenever appropriate, by using procedures already being performed on the subjects for diagnostic or treatment purposes.

  2. Risks to subjects are reasonable in relation to anticipated benefits, if any, to subjects, and the importance of the knowledge that may reasonably be expected to result.

  3. Selection of subjects is equitable. In making this assessment the IRB should take into account the purposes of the research and the setting in which the research will be conducted and should be particularly cognizant of the special problems of research involving vulnerable populations, such as children, prisoners, pregnant women, mentally disabled persons, or economically or educationally disadvantaged persons.

  4. Informed consent will be sought from each prospective subject or the subject's legally authorized representative, in accordance with, and to the extent required by §46.116.

  5. Informed consent will be appropriately documented, in accordance with, and to the extent required by §46.117.

  6. When appropriate, the research plan makes adequate provision for monitoring the data collected to ensure the safety of subjects.

  7. When appropriate, there are adequate provisions to protect the privacy of subjects and to maintain the confidentiality of data.

While the DHHS Office for Human Research Protections provides guidelines and regulations governing the review and approval of research, there is considerable room for interpretation of the guidelines and how they apply to specific research efforts. Individual IRBs and, in some cases, individual IRB members, make their own interpretations and apply varying standards. In some cases survey research may be determined to be exempt from review; however, if the survey involves children or collects identifiable information that, if disclosed, could place the subjects at risk of criminal or civil liability or be damaging to financial standing, employability, or reputation it will not be exempt. If the research is determined to be of no more than minimal risk the review may be done by expedited procedures, normally by a single IRB member or a subcommittee of the IRB. Otherwise, it will go to the full IRB for review. Research involving prisoners is subject to especially stringent review standards.

IRBs consider the risks and benefits of research in a variety of ways and one area where they differ is in their view of the risks posed by survey research in relation to risks posed by other kinds of research participation. For example, some IRBs that are accustomed to considering the risks to subjects posed by studies of invasive procedures or experimental drugs tend to view all survey research as very low risk. Others see risks to subjects from survey participation as indistinguishable from risks from other kinds of research or possibly more serious. Moreover, survey participation in research funded by federal grants often includes a variety of activities, including answering questions about private, stigmatized, or illegal behaviors; providing biological specimens, having physical measurements taken, or having various kinds of medical and other tests; and allowing access to personal medical records or links to records in other databases, such as Medicare data or insurance claims.

IRBs wrestle with the role of incentives for research participation and have yet to come up with a universally accepted framework. For example, Ackerman (1989) laid out his view of the ethical issues posed by incentives from a human subjects perspective. While incentives can be seen as consistent with the goal of encouraging social cooperation, which is valuable (so perhaps payment should be unrestricted), it may also undermine the requirement that research participation be voluntary. The prospect of economic reward may lead people to undertake risks that are not consistent with their true values and preferences. In addition, providing incentives may undermine the principle of fair treatment if it leads to disadvantaged persons participating in research at a higher rate than wealthier subjects. Ackerman prefers recruitment of altruistically motivated subjects and advises that payment be limited to reimbursement for expenses, but allows that this may make it impossible to conduct research so he also proposes that IRBs view research subjects as “wage laborers”. In this framework, the level of incentives would be based on the time commitment required and the nature and number of procedures involved or the amount of risk subjects undertake. Alternatively, wage payments could be viewed as “pure inducements” – the amount geared to the level required to recruit an adequate number of subjects within the required period of time. His framework assumes that subjects are not exposed to excessive risk and that incentive levels are guided by the economic value of their time as “unskilled labor” on a part-time basis and increased in relation to the arduousness and the social value of the work.

Others propose that the requirement for ethical treatment of participants can be satisfied in terms of an economic model in which researchers are willing to compensate subjects up to the amount they expect the individual’s participation to benefit the research (Dunn and Gordon, 2005). Rational subjects would make decisions based on expected costs (time, discomfort) and benefits of participation (altruistic or intellectual gratification and cash or in-kind compensation). This assumes meaningful informed consent and that subjects are protected by IRBs from being placed at excessive risk. The authors also call on IRBs to collect data on levels of incentives in relation to demographic groups and research protocols and that researchers provide more information about this from their studies. While not addressed directly, the economic perspective and the discussion of the effects that payment levels have on participants at different income levels seems to imply the possibility of flexibility in payment levels across subjects, a feature that is currently problematic with most IRBs.

The degree of discomfort that some ethicists have with paying incentives is illustrated by VanderWalde (2005). While he echoes the concern that incentives may have undue influence on research participants, he also proposes that they have undue influence on the IRBs. He asserts that treating payment as an intrinsic benefit of research will result in IRBs allowing higher levels of risk than they should allow. One form of payment has become a particular concern for some IRBs: participation in lotteries. Brown et al. (2006) argue that lotteries should never be approved as research incentives because they provide unequal reward and undermine the goal of balancing risks and benefits (one person receives high compensation and everyone else receives none) and they undermine making an informed decision to participate (because people overvalue their chances of winning).

Grant funded research often involves so-called “special populations”, such as HIV positive individuals or drug users, and Fisher (2005) reviewed the literature and outlined the issues in relation to incentives for these populations. An issue is the “coercive” incentive, that which prompts participants to lie or conceal information in order to appear eligible for the study or lures those who would otherwise not participate into doing so. Fisher points out that paying an incentive for research participation may also have the beneficial consequence of clarifying the relationship between subject and researcher and avoiding confusion with a patient and therapist or physician relationship. She concludes that there is no consensus on what constitutes ethical compensation in this kind of research and suggests that prospective participants and community advisory boards can inform ethical decisions in this area.

In their chapter entitled “Paying Research Subjects” in Institutional Review Board: Management and Function, Brown et al. (2006) note that “Few institutions have coherent policies [on payment of incentives]”. However, they recommend that:

compensation should not be of a nature that it interferes with the ability of potential subjects to give informed consent, without the possibility of coercion or undue influence (understanding, of course, that the meaning of this is nebulous at best). IRB guidelines should also stress the need for prorating compensation and define limited situations under which such proration is not needed. Finally, guidelines should address the particularly sensitive issues related to direct compensation of minors.

Why go into all this? Researchers conducting surveys under federal grants are required to have their proposed procedures for paying incentives reviewed and approved by IRBs that are far from clear about what is and is not appropriate. This can have a considerable impact on how IRBs consider incentives in relation to other factors that impact survey research. For example, some IRBs are also concerned with whether repeated attempts to reach and interview respondents and refusal conversion attempts are consistent with the principle of voluntariness of research participation. Concerns about equity may be raised in terms of paying higher incentives to respondents who initially refuse to be interviewed or paying different incentive levels to participants from lower or higher income groups. Grant funded researchers are faced with the need to maximize response, within budgets, and in a fashion that is acceptable to their IRBs.

Publication of Research Findings

Investigators conducting grant funded research are generally planning to publish their results in peer-reviewed professional journals, preferably high impact journals such as the Journal of the American Medical Association, the New England Journal of Medicine, first tier medical specialty journals, or the key professional journals in other fields. The reviewers for these journals are typically not themselves survey researchers and often apply standards for response rates that are based on earlier periods. Thus, current grant funded investigators feel considerable pressure to demonstrate high response rates to ensure professional acceptance of their work.

Research Budgets

NIH has been increasingly stringent about funding levels. For the typical grant, the total annual budget (directs and indirects) must be kept below $500,000. These funds need to support the investigators, travel, and other expenses as well as data collection. In the past, it was possible to obtain a waiver of this requirement, but this has been less true in recent years. Of course, there are some kinds of projects that are funded by larger grants, but these typically involve considerably more effort. Money on grant funded projects is always very tight and payment of incentives trades off against other needs.

Organization and Staffing

The principal investigators (PI) of projects supported with federal grants rarely come from a survey research background. Many are physicians conducting surveys as part of biomedical research projects or behavioral scientists of various kinds. While some researchers in these fields are very interested in survey methods and they read the methodological research articles that are published in their professional journals, they are often less exposed to the traditional survey research literature (it is frequently not cited in methodological articles in medical journals). In addition, the survey research they carry out may be managed not by a survey research organization, but by an ad hoc staff reporting to the PI. These individuals, while often very dedicated and innovative, may or may not have traditional survey training and their levels of experience vary widely. Thus, grantees carrying out surveys may approach this task from a somewhat different perspective than the traditional survey organization.

Recent Survey Researcher Perspectives About Survey Incentives

How survey researchers view incentives differs considerably from the way federal grantees view the problem.

Ethical Issues

From our reading of the survey literature, survey researchers, being the practical souls they are, tend to be more interested in whether and how incentives work and how best to use them than whether it is ethical to provide them. Looking over the papers from the 1992 COPAFS incentives conference we see no concern with potential harm to participants from providing incentives. That situation does not appear to have changed. The excellent 2004 book titled Survey Methodology (Groves et al.) never refers to ethical issues in the discussion of the use of incentives. Rather, the concern is with how incentives figure into the attributes of survey participation in relation to “leverage-salience theory” – what features of a survey potential respondents evaluate as having negative or positive value in deciding whether to respond. The American Association for Public Opinion Research (AAPOR) Code of Professional Ethics and Practices (2006) requires participation in most surveys (except those required by law such as the Census) to be voluntary, requires that participants receive sufficient information to allow a free and informed decision, and requires avoidance of practices that may harm, humiliate, or seriously mislead survey respondents – but doesn’t mention paying incentives in this context. The survey methods disclosure requirements don’t include payment of incentives as a feature to disclose. Section 9 of the “Best Practices” includes incentives among the procedures that “should be considered… (where possible)” to stimulate survey cooperation or participation. Payment of incentives is certainly not among the practices that AAPOR condemns (AAPOR, 2006). The AAPOR website guidance on dealing with IRBs does not address the question of payment of incentives.

Effects of Incentives on Response and Data Quality

Eleanor Singer laid out the issues and summarized the evidence on incentives from the perspective of the survey literature in the use of incentives in her (2002) book chapter. Very briefly, she found that incentives improve response rates across all modes. The effect appears to be linear, larger incentives have bigger effects on response rates. Prepaid incentives, where possible, seem more effective in increasing response rates than promised incentives that are contingent on survey participation and money is more effective than gifts. Lotteries tend to be less effective than payments. Incentives are effective in improving response rates in early waves of panel surveys.

In terms of response and coverage effects, there is some evidence that respondents who receive incentives may be in a more optimistic mood and provide more positive responses than unpaid respondents. While all respondents are more likely to participate in surveys if incentives are used, lower income respondents are more affected than higher income respondents. Used as one of several different motivational techniques (salience, altruism, sponsorship), incentives help to increase the representativeness of the sample. The effect of incentives appears to be due to their effects on respondents and not through interviewer expectations of greater respondent willingness to participate when incentives are used.

How Grant Funded Researchers Use Incentives

Although we do grant funded survey research and see a good deal of it go through the RAND Survey Research Group and the RAND IRB, there is no reason to think that our view of this subject is comprehensive. Therefore, to obtain a broader perspective on this topic we designed and conducted a brief web survey of NIH grant recipients who are carrying out surveys.

Methods for Web Survey of Grantees

In an effort to learn about the factors that have influenced decision-making on use of incentives for survey respondents in recent studies, we designed a web-based survey of NIH grantees conducting surveys. To obtain a sample, we searched the NIH Computer Retrieval of Information on Scientific Projects (CRISP) database for grants awarded in fiscal years 2004, 2005 and 2006. Since grants normally cover multiple years and survey design is often done in the first year, we sampled those years to obtain projects that had already carried out their surveys. We included only grant proposals that listed “survey” or “questionnaire” in their abstract.

We reviewed the CRISP “hits” and included grants that planned to field a primary data collection (that is, funding was not for analysis of existing data). We excluded grants for work conducted outside the United States and research involving non-human subjects. Grants were also removed from our sample if the primary research activities were questionnaire development or methodological testing or if the funding was for a research center or conference support. After also removing any projects that did not include an email address for the grantee (nearly all did), we identified 145 eligible projects.

We sent a personalized email invitation to the principal investigator listed in the grant proposal on CRISP to request participation, describing the upcoming Council of Associations on Federal Statistics (COPAFS) conference. PIs were provided with a link to the web survey and a unique PIN to access the questionnaire. We requested that PIs either complete the survey themselves or pass the URL and PIN to another person on the project who could provide the information.

The use of PINs allowed us to track completed surveys, partial surveys, and non-responders. We sent two personalized email reminders to non-responders and those with partial completes and we sent a final, personalized request for participation, along with a short list of frequently asked questions and answers, to non-responders and those with partial completes. The survey was available on line for a total of 19 days.

The web survey asked PIs or their designated respondent to describe the features of the main survey that was conducted under this grant. To control respondent burden, if more than one survey was conducted we asked respondents to think about their most important survey, in terms of their research goals. Specifically, respondents were asked to describe the sample population and sample size (including planned number of completed surveys and actual number of completed surveys), survey mode(s), form of initial contact, number of follow-up attempts, survey content, and average survey length.

First, we asked respondents whether incentives were provided for survey participation. If no incentives were provided, respondents described why incentives were not used and if any IRB or other reviewing organization raised any questions about the lack of payment to survey respondents. If incentives were provided for survey participation, we asked about the type, value and timing of incentives. We also collected information about the factors that influenced the decision to provide incentives, including the influence of an IRB or other reviewing organization.

Results

We sent out e-mails asking for survey participation to 145 principal investigators and, after three e-mail reminders, 92 provided usable responses, a response rate of 63.4%. Over 80% were completed by the PI. We compared responders and non-responders based on information about the survey in the abstract included in the CRISP database for most grants: sample population and sample size. In general, the responders look very much like all of the studies in terms of sample population. The three most common sample populations across all studies (patients, “other,” and minorities) accounted for nearly 60% of all studies that we included and nearly 60% of responders. The “other” category includes all populations that did not fit into any of the existing categories. Three sample populations were slightly under-represented in the responding pool: adolescents/students (13.8% of all studies, 12.0% of responders), stigmatized groups (4.1% of all studies, 1.1% of responders), and low income (3.4% of all studies, 1.1% of responders). Three sample populations were slightly over-represented in the responding pool: physicians (11.7% of all studies, 15.2% of responders), general adult population (6.9% of all studies, 8.7% of responders), and elderly (3.4% of all studies, 4.3% of responders).

Over one-third of all studies did not provide any information about sample size, a finding that was true for responders and non-responders as well. Surveys with a sample size of less than 1,000 were slightly under-represented among responders (24.8% of all studies, 20.7% of responders) and surveys with a sample size of 1,000-4,999 were slightly over-represented among responders (26.9% of all studies, 29.3% of responders.) Larger surveys were also over-represented among responders: surveys with a sample size of 5,000-9,999 made up 6.9% of all studies on our sample but 8.7% of responders. Surveys with a sample size of 10,000 or more were 6.2% of all studies that we included but 7.6% of responders. In general, the surveys described in the abstracts of responders and non-responders were quite similar and we made no attempt to weight the data.

How often are incentives used?

Of the 87 respondents who provided information on incentive use in their only survey or most important survey, 82.8% indicated that incentives were given to their survey respondents while only 17.2% reported that an incentive was not offered. Most researchers who provided an incentive to participants did so only if the participant took part in their study in some capacity: 44.4% gave participants an incentive for full completion, 31.9% gave participants an incentive if they completed part of the survey, 12.5% of the respondents reported that they gave incentives if a participant considered taking part in the study, and 11.1% provided a prepaid incentive.

Respondents who reported providing an incentive were also asked whether participants were asked to do anything in addition to completing a survey. Of the 72 studies that provided an incentive, 52.8% also asked participants to complete an additional task. While 50.0% of these studies provided a separate payment to participants for completing the additional task, 28.9% lumped the payment for this task and the survey together and 21.1% of these studies did not provide an incentive to participants for completing the additional task. Most often the additional tasks involved participating in follow-up surveys/interviews or providing biological specimens. A few studies asked participants to undergo a physical exam, keep a diary of certain events, or attend a class. One study requested participants’ written consent to access their medical records.

We examined the characteristics of the research studies in which incentives were not offered and the characteristics of research studies in which incentives were offered.

What kinds of surveys don’t use incentives and why not?

The 15 research studies that did not offer incentives varied in the average length of time it took participants to complete the survey: 40% took less than 15 minutes to complete, 13.3% took between 16-29 minutes to complete, and 40% took between 30-59 minutes to complete. Only 1 respondent who did not provide incentives reported that it took participants 60 minutes or more to complete the survey.

Thirteen of the 15 respondents who did not provide incentives responded to the question about the topics covered in their survey. Of these respondents, 73% indicated that questions about health status or health conditions were included in their survey. Respondents also collected information on personal financial information (27%), drug use or drug history (13%), immigration status (13%), sexual behavior (13%), and other sensitive behavior (33%). Two respondents indicated that they requested participants’ permission to link to a database to gain more information about them.

The non-incentive surveys were conducted with a variety of populations: general adult household, adolescents or students, children age 18 or under, minorities, low income, older adults/elderly, and patients. However, none of the studies that lacked an incentive involved physicians/medical providers or special populations (such as people who are HIV positive, drug users, or sex workers). (See Table 1 for breakdown by population.)

Of the 15 research studies not providing an incentive, 40% mailed surveys to participants. Other researchers had participants complete their survey by in-person interview (27%), telephone interview (13%), Internet (13%), or a combination of data collection modes (7%). Sixty percent first contacted participants by mail, 13% by telephone, 7% by email, 13% in-person, and 7% by first enrolling participants at a health care setting, work site, or other location. Follow-up attempts were carried out in 68% of these studies.

Also of interest was the number of surveys collected by these projects. These projects were collecting large numbers of surveys, with 27% of respondents reporting that they originally planned to collect between 1001-5000 surveys and 33% who planned to collect over 5000 surveys. While these may seem like lofty ambitions since no incentive was being offered to survey participants, 67% of the researchers who did not provide an incentive to participants reported collecting 100% or more of their planned sample. In contrast, however, studies that attempted to collect a greater number of surveys reported lower completion rates than studies that attempted to collect fewer surveys.

While the topics, population, project size, and completion time varied for many of the projects that did not provide an incentive, the reasons for forgoing an incentive were not so diverse. More than half of the researchers (60%) who did not provide an incentive for completing their survey reported that they expected a good response rate without the use of incentives. More than half of these projects (60%) said they did not have the budget to provide incentives to their participants. Only 27% of respondents cited that the preference of the survey team was a factor in deciding not to pay participants. None of the respondents reported that the preference of the IRB was a factor in not paying incentives.

What kinds of surveys do use incentives and how?

The 72 research studies that offered incentives varied in the average length of time it took participants to complete the survey, but were generally more time consuming than the non-incentive surveys. Of the surveys offering an incentive, 20% took less than 15 minutes to complete, 18% took between 16-29 minutes to complete, and 35% took between 30-59 minutes to complete. About a quarter of the studies that offered incentives took participants an hour or longer to complete the survey. The “no incentive” studies took a mean of 27.2 minutes to complete (SD = 17.6) while the studies with incentive took a mean of 45.5 minutes (SD = 44.7) to complete.

As with the studies that did not provide incentives, questions about health status or health conditions were included in most of the studies that gave incentives to participants (74%). Respondents also asked participants questions about their personal financial information (44%), drug use or drug history (25%), immigration status (14%), sexual behavior (31%), and other sensitive behavior (40%). Several of the studies asked participants for information from personal records (10%) and requested participants’ permission to link to a database to gain more information about them (7%).

The studies that provided incentives were conducted with many different populations including general adult household (18%), adolescents or students (17%), children age 18 or under (4%), minorities (25%), low income (11%), older adults/elderly (15%), patients (21%), physicians/medical providers (21%), and special populations (3%).

The most frequently used modes for collecting data when an incentive was provided was by mail (28%), through an in-person interview (25%), or through a telephone interview (15%). Other modes of data collection included computer assisted self interview (7%), paper and pencil self administration (8%), over the Internet (10%), group administration (1%), or a combination of modes (6%). Eighty-six percent of studies that provided an incentive attempted follow-ups with non-responders.

On average, studies that provided incentives were smaller than studies that did not provide an incentive. Of the respondents that provided incentives to their survey participants, 38% reported that they originally planned to collect between 301-1000 surveys and 38% planned to collect between 1001-5000 surveys. Only 10% of the researchers who provided incentives originally planned to collect over 5000 surveys. Sixty-nine of the respondents who provided an incentive supplied information about the number of surveys they originally planned to collect and the number that was actually obtained, so that the completion rate could be calculated. Of these cases, almost all (99%) of the studies that provided incentives collected at least 50% of their survey goal, with 83% collecting over 75% of their survey goal and 51% collecting 100% or more of their goal (a remarkable achievement).

The most popular forms of incentives were cash (31%), gift card/certificate (27%), and check (25%). Almost half (48%) of the studies that provided incentives paid participants between $20-49. Thirty-four percent gave participants an incentive worth $10 or less while 14% provided respondents with $50 or more. Some participants offered unusual incentives such a lottery for an iPod, gift certificates, or a health club membership for a year. Others offered the more common mugs, bags, water bottles, and in a few cases, copies of the research results.

Surprisingly, payments to physicians and medical providers were, on average, much less than the amounts paid to other populations. Physicians and medical providers were paid an average of $15.00 while studies that surveyed general adult households paid those participants an average of $35.77. This may be due to the fact that surveys for medical providers took less time to complete (average = 25.3 minutes, SD = 15.6) than surveys for general adult households (average = 80.0 minutes, SD = 83.3). (See Table 3 for average incentive amount and time to complete for different populations.)

Why are incentives used?

The main reason most often cited for offering an incentive was to increase response rate. Seventy-three percent of respondents who used an incentive in their study ranked increasing response rate as a main reason for using an incentive. Other major reasons for offering an incentive included reducing non-response bias (rated as being a “main reason” or “very important” by 71% of respondents) and rewarding participants for research participation (rated as being a “main reason” or “very important” by 56% of respondents). Reducing data collection costs, reducing time spent on follow-ups, and helping interviewers feel comfortable were not rated by most respondents as being important factors in the decision to offer an incentive (see Table 4). Several respondents commented that they wanted to maximize response from particular kinds of subjects – lower income or lower SES individuals, hourly wage workers, busy people, older people, teenagers, subjects in randomized control trials, and physicians. Others based the need for incentives on their prior experience with the type of participant or the research setting; for example, school-based research that indicated a need for incentives or previous randomized trials of incentives that they had done. Very few saw incentives as trading off costs for follow-up either by endorsing this response to the survey question or in their comments. Some cited research evidence of the value of incentives and others expressed the desire to offer a token of appreciation or to reimburse for time and transportation expenses or to empower interviewers to ask for study participation. One commented, “We get paid to do this work – why shouldn’t the subjects?”

How are incentives types and amounts determined?

While we did not specifically ask how the incentive amounts and types were determined, participants offered open-ended comments about this. Some tied it to actual costs of participation, such as transportation, childcare, lost wages, or cell phone charges for an RDD interview and others referred more generally to compensation for time and contribution of personal information. A few mentioned having conducted experiments to determine the amount of incentive that was effective. Several mentioned consideration of what other research projects that included the same kinds of subjects were paying, wishing to avoid an “arms race” of escalating incentives or referring to a “community standard.” Quite a few respondents commented that the amount of the incentive was constrained by their budget and they wished they could have offered more. A few were concerned with accounting issues (the need for receipts) or safety issues for interviewers or respondents.

What is the role of IRBs?

Of the researchers who did not provide an incentive for survey respondents, all reported that the IRB did not raise questions about lack of payment and that IRBs were not a factor in the decision not to provide incentives. Only one respondent who provided incentives cited that IRB preferences were a main reason for giving incentives to survey participants. Most other participants (88%) rated IRB preferences as being not important or not applicable to their decision to provide incentives. However, 23% of respondents who provided an incentive did report that a reviewing organization such as an IRB raised questions, placed limitations, or provided advice about their payment to participants. Most of these respondents explained that their IRB took steps to make sure that the incentive being offered to participants was not coercive. For example, several researchers reported that their IRB had placed a cap on the amount of money that a study can offer participants or required that the same incentive amount be offered to all research participants. Other IRBs required researchers to provide an explanation for why the incentive is being offered in order to make sure that the amount of the incentive is appropriate to what is being asked of the participant. In order to make sure that participants are not being coerced into taking part in a study, some IRBs even had rules about when the incentive may be disclosed to participants, with one respondent reporting that he was not allowed to list the incentive on recruitment materials. A few commented that their IRBs were very strict about the use of incentives or that their IRB was “anti-incentive.” Another commented, “Our IRB recommends a participant stipend equal or ‘comparative’ to the amount of time it takes for participation in a study, but not too much that it acts as an incentive.”

Conclusion

Researchers who are funded by grants that include surveys are conducting survey research with a variety of kinds of participants, including minors, minorities, and other groups that are of special interest in terms of payment of incentives. Their survey research often covers sensitive topics and is paired with additional participation requests to subjects, such as additional survey participation, physical measurements, providing physical samples for lab tests, and allowing links to additional databases. The researchers are driven by the need to achieve standards of quality that will allow publication of results in major journals, often in fields where journal reviewers are not familiar with conducting surveys, and they face tight budget constraints. Their work is reviewed by IRBs that have differing or no specific standards for judging the appropriateness of incentive payments.

These researchers seem generally convinced that incentives should be used and are using them in a variety of ways. While some are conducting experiments to determine incentive amounts, others are making practical judgments about how to use them, informed by what others are doing, the influence of IRBs, community standards, and their budget constraints. Some of their approaches are not in line with what is recommended practice based on survey research evidence. Survey researchers who conduct methodological research on incentives may wish to attempt to publish in the non-survey journals when their subject matter is related or in the form of review articles.

The survey field could also inform IRBs about research on incentives and, possibly, conduct research that IRBs would find useful, given their uncertainty about how to make judgments about the appropriateness of incentives. Considering the ethical perspectives along with the practical issues would make research more informative to IRBs and having IRBs develop some standard approaches would make it easier for grant funded researchers.

Table 1. Populations surveyed by whether an incentive was offered

Population Did Not Offer Incentive Did Offer Incentive
General Adult 1 13
  6.7% 18.1%
Adolescents or Students 1 12
  6.7% 16.7%
Older Adults/Elderly 4 11
  26.7% 15.3%
Children age 18 and under 1 3
  6.7% 4.2%
Low income 1 8
  6.7% 11.1%
Physician or Medical Provider 0 15
  0.0% 20.8%
Patient 3 15
  20.0% 20.8%
Special Population 0 2
  0.0% 2.8%
Minority 1 18
  6.7% 25.0%
Other 7 22
  46.7% 30.6%
  n = 15 n = 72

Table 2. Comparison of studies in which incentive was provided and studies in which incentive was not provided

  Provided Incentive Did Not Provide Incentive Total
Time to Complete      
Mean 45.51 27.20 42.31
N 71 15 86
SD 44.703 17.559 41.777
Minimum Number of Follow-Ups      
Mean 2.95 3.44 3.01
N 60 9 69
SD 3.239 2.603 3.151
Maximum Number of Follow-Ups      
Mean 18.82 6.29 17.40
N 55 7 62
SD 68.121 4.309 64.232
Number of Surveys Planning to Collect      
Mean 2307.90 8642.00 3412.69
N 71 15 86
SD 3713.813 18898.566 8719.481
% of Goal Collected      
Mean 90.16% 92.32% 90.55%
N 69 15 84
SD 18.860 23.981 19.726

Table 3. Average incentive amount and time to complete survey for different populations for studies offering an incentive (note that same population may be classified in more than one way)

Population Amount of Incentive ($) Time to Complete Survey (minutes)
General Adult    
Mean 35.77 80.00
N 13 13
SD 30.403 83.292
Adolescents or Students    
Mean 19.18 44.33
N 11 12
SD 15.012 31.431
Older Adults/Elderly    
Mean 24.09 49.55
N 11 11
SD 15.300 23.817
Children age 18 and under    
Mean 18.33 28.33
N 3 3
SD 2.887 11.547
Low income    
Mean 30.88 84.38
N 8 8
SD 37.038 99.371
Physician or Medical Provider    
Mean 15.00 25.29
N 15 14
SD 10.637 15.657
Patient    
Mean 25.80 60.00
N 15 15
SD 28.340 76.649
Special Population    
Mean 45.00 52.50
N 2 2
SD 28.284 10.607
Minority    
Mean 32.33 61.94
N 18 18
SD 33.242 72.358
Other    
Mean 19.86 38.73
N 22 22
SD 12.438 32.884

Table 4. Reasons for providing survey participants with an incentive

  Main reason Very important Somewhat important Not important N/A Total
Increase response rate 52 13 4 2 0 71
 e 73.2% 18.3% 5.6% 2.8% 0.0% 100.0%
Reduce non-response bias 12 36 15 2 3 68
  17.6% 52.9% 22.1% 2.9% 4.4% 100.0%
Reduce follow-up costs 2 11 22 22 11 68
  2.9% 16.2% 32.4% 32.4% 16.2% 100.0%
Reduce time for data collection 3 12 19 29 5 68
  4.4% 17.6% 27.9% 42.6% 7.4% 100.0%
Help interviewers feel comfortable requesting participation 3 9 14 27 16 69
  4.3% 13.0% 20.3% 39.1% 23.2% 100.0%
Survey organization wanted to pay incentives 0 1 6 23 38 68
  0.0% 1.5% 8.8% 33.8% 55.9% 100.0%
IRB wanted incentives in this study 1 0 4 23 40 68
  1.5% 0.0% 5.9% 33.8% 58.8% 100.0%

References

Ackerman TF, “An Ethical Framework for the Practice of Paying Research Subjects”, IRB: A Review of Human Subjects Research 11:4, pp 1-4, 1989.

American Association for Public Opinion Research, “Code of Professional Ethics and Practices & Best Practices for Survey and Public Opinion Research and Survey Practices that AAPOR Condemns” 2006.

Brown GG, Brown JS, Kratochvil C, Schonfeld TL, and Prentice EP, “Paying Research Subjects” in Institutional Review Board Management and Function edited by Bankert EA and Amdur RJ, Jones and Bartlett, Sudbury MA (2006).

Brown JS, Schonfeld TL, and Gordon BG, “ “You May Have Already Won…”: An Examination of the Use of Lottery Payments in Research”, IRB: Ethics and Human Research 28:1 (2006), Pp 12-16.

Code of Federal Regulations, TITLE 45?PUBLIC WELFARE DEPARTMENT OF HEALTH AND HUMAN SERVICES, PART 46 -PROTECTION OF HUMAN SUBJECTS, Revised June 23, 2005?Effective June 23, 2005 http://www.hhs.gov/ohrp/humansubjects/guidance/45cfr46.htm#46.11105

Dunn LB, Gordon NE, “Improving Informed Consent and Enhancing Recruitment for Research by Understanding Economic Behavior”, Journal of the American Medical Association 293:5, pp. 609-612, 2005.

Fisher CB, “Ethics in Drug Abuse and Related HIV Risk Research”, Applied Developmental Science 8:2, pp 91-103 (2004). Groves RM, Fowler FJ, Couper MP, Lepkowski JM, Singer E, and Tourangeau R, Survey Methodology, John Wiley and Sons, Inc. New York (2004).

Hosek SD, Cox AG, Ghosh-Dastidar B, Kofner A, Ramphal N, Scott J, and Berry SH, Gender Differences in Major Federal Grant Programs, RAND Corporation, Santa Monica, CA (2005) RB-9147.

Proposed Standards and Guidelines for Statistical Surveys, http://www.whitehouse.gov/omb/inforeg/proposed_standards_for_statistical_surveys.pdf, undated.

Singer E, “The Use of Incentives to Reduce Nonresponse in Household Surveys” Chapter 11 in Survey Nonresponse, John Wiley and Sons, Inc. New York (2002).

VanderWalde A, “Undue Inducement: The Only Objection to Payment?”, American Journal of Bioethics, September/October 2005, pp.25-27

[1] Funding for preparation of this paper was provided by the RAND Corporation Survey Research Group, Santa Monica, CA.