week5-increasing-response-rates-reminders

Up: survey-nonresponse Prev: week4-increasing-response-rates-advanced-letters-and-contact-attempts Next: week6-increasing-response-rates-incentives

Reading:

• Schouten, B., van der Laan, J., & Cobben, F. (2014). The impact of contact effort on mode-specific selection and measurement bias. Survey Methods: Insights from the Field, 1-8.

Reading Notes:

• The article focuses on a large-scale mixed-mode experiment with the Crime Victimisation Survey. The experiment involved randomly assigning sample individuals to four survey modes (Web, mail, telephone, and face-to-face) and a follow-up using only interviewer-administrated modes. The goal was to disentangle mode-specific selection and measurement biases. The article examines the impact of contact effort (number of telephone calls, face-to-face visits, and reminders in Web/mail) on these biases.

• Key findings:

  • Contact Effort and Biases: Analyses show that contact effort has little impact on measurement-bias and modest impact on selection-bias.
    • 🚨 Thus, increasing the effort to contact people is not a major reason of messed up accuracy or respondent selection.
  • Mode-Specific Effects: Contact effort affects nonresponse-bias and measurement-bias differently depending on the survey mode.
    • But there is no cases where low contact effort (just one call or reminder) caused significantly problems in both selection and measurement biases at the same time.
  • Implications for Survey Design:
    • Making more calls or sending more reminders doesn’t make people answer questions less accurately.
    • Adaptive survey design: Surveys can be designed to target specific groups to improve who responds, without worrying about contact effort ruining answer quality.
    • I mixed-mode surveys, keeping the balance of responses from each mode consistent over time helps make result reliable.

• Results:

  • For employment, more contact effort increased the number of employed people responding, mainly due to selection bias.
  • For education, face-to-face surveys initially overrepresented people with primary education, but this decreased with more visits, due to both selection and measurement biases.
  • In Web surveys, more reminders led to more missing education data (who need multiple reminders often skip the education question).
  • For crime-related questions, phone and mail surveys needed more effort to reach crime victims or those feeling unsafe, but Web showed no clear pattern.
  • Overall, contact effort affected biases but didn’t significantly impact both selection and measurement biases at the same time.

• Discussion:

  • Contact effort has a small effect on measurement-bias but a larger effect on selection-bias.
    • If you’re designing a survey about neighborhood safety, choosing between Web and face-to-face matters more than how many reminders or visits you make, because the mode affects both who responds and how they answer.

Lecture Notes:

• Advance mail and reminders serve a different purpose.
  • the former is done in order to attempt a contact — noncontact problem
  • the latter assumes that the contact has been established, but the people refused it — refusal problem

Mode	Paper Based	CAI
telephone	PATI	CATI
face-to-face	PAPI	CAPI
self-administrated	P&P, Mail, etc.	eMail, Web, DbM

• The biggest difference between these modes: whether there is an interviewer or not
  • Interviewers reduce nonresponse error
  • Interview error is also something that should be avoided
  • There is a trade-off between the two

• Reminders:

  • Reminders are effective.
    • The first reminder is more powerful than the follow ups.
      • This is because the first reminder convinces people who are easy to. So the rest are actually harder to convince.
  • Some reminders can chase away the people who didn’t yet decide whether they want to answer or not.
  • Reminders and Non-response Bias: They do not help when it comes to non-response bias.
    • Reminders bring us the people groups who are already answering our questions.
    • No-possitive effect when it comes to sociodemographic levels in terms of income or education.
  • Usually, because reminders only bring the people who are already responding, reminders do not effect, or worsen the nonresponse bias
    • Criminal records as an example: People who do not have criminal record history will more likely to respond, and if we send reminders, we worsen the situation

Response rates and non-response bias

• Increased response rates decreases the “risk of non-response bias”
  • But when it comes to real non-response bias, it is

Hard-to-get respondents are more or less in the same characteristics as non-respondents. So, we usually compare easy-to-get and hard-to-get respondents in order to understand what non-respondents are thinking about.