Survey Design and Implementation with Qualtrics

21 January 2025

Bella Ratmelia - Senior Librarian, Research & Data Services, SMU Libraries

Learning Objectives

By the end of today’s workshop, you should be able to:

1. Create a survey from scratch in Qualtrics that suit your survey objectives

2. Learn 3 ways to distribute your survey

3. Generate a simple reporting for your survey

4. Describe at least 3 recommended practices

Overview

1. How Qualtrics fit in the Survey Research Workflow

2. Hands-on sessions:

   1. Creating a survey from scratch

   2. Distribute a survey

   3. Generate a report

3. Recommended practices

4. Q&A

This workshop will not be covering the statistical details in creating survey, e.g. Cronbach’s Alpha, how to do PCA / EFA, etc.

Overview of the survey workflow

Figure 2.1 from Exploring Complex Survey Data Analysis using R

Zimmer, S. A., Powell, R. J., & Velásquez, I. C. (2024). Exploring Complex Survey Data Analysis Using R: A Tidy Introduction with {srvyr} and {survey}. Chapman & Hall: CRC Press.

Things to consider as you draft your survey

1. Check if there is already a publicly available data on your topic.

2. Check if there is already a standardized or validated survey questions set available to use.

3. Keep your target demographic in mind.

4. Be mindful of survey length.

5. Consider the complexity of questions and how it can affect your data quality vs respondent’s fatigue.

6. Make sure the questions flow logically.

7. Use sensitive questions sparingly – provide “prefer not to answer” option.

How do I access Qualtrics?

SMU has an institutional subscription to Qualtrics, available here: https://smusg.au1.qualtrics.com/

As it uses SSO login, it is best to access it via Microsoft Edge. Let’s try to access it now.

Hands-on Session #1

Let’s recreate this survey from scratch: https://smusg.asia.qualtrics.com/jfe/form/SV_570663ILMyP7JY2

Looking back at the survey…

Is there anything that can be improved from our survey? What do you think?

How many responses is enough?

1. Use the Qualtrics sampling calculator to determine the ideal number of survey respondents: https://www.qualtrics.com/blog/calculating-sample-size/

2. Ensure your responses are representative.

   • For example, if you’re targeting postgraduates at SMU and stratifying by part-time vs. full-time students, aim to collect sufficient responses from both groups to maintain balance and accuracy.

What are the available distribution methods?

• Anonymous links or anonymous QR codes

• Personalized links

• Generate different links to track which of your survey ad posters get the most response.

• Email campaigns - Send your survey directly to participants via email function in Qualtrics

• Use third party survey recruiter

• Embed survey in blogs or websites

• Share in forums and relevant online communities such as Discord, Reddit, or even Telegram

• Referral program / snowball sampling

• Depending on your target audience: Social media ads on Facebook or Twitter.

Hands-on Session #2 - Distribution

Let’s try the 3 distribution methods in Qualtrics: Anonymous link, Personal links, and Emails

How can I get more responses?

• If it’s a feedback survey on workshop, event, or tools, it’s best to distribute it immediately after the session.

• Highlight the compensation, if you have any

  • IRB Note: Some exceptions exist for offering “a chance to win a prize” as compensation. Check with the IRB for guidelines if you plan to use this method

  • Free surveys can serve as a filter, reducing the likelihood of responses from trolls or individuals only seeking compensation.

Simple analysis and reporting

• Qualtrics’s have two useful tools: StatsIQ and TextIQ

  • Having said that, I personally find it easier to conduct my analysis outside of Qualtrics (I use R)

• Reporting - Easily generate visualizations and data tables based on data gathered so far.

  • Report is updated as new responses are coming in

  • Great for sharing results with team members or stakeholders where you don’t necessarily want to share the survey with them.

Hands-on Session #3

Let’s try create a simple report that we can share with other collaborators!

Recommended practices

1. Just because you can put complicated question types e.g., ranking, allocating percentages, etc. doesn’t mean you should put those questions

2. Always pilot test your survey!

3. Give your questions a nickname – this will be handy when exporting to R, SPSS, or Stata!

4. Analysis shouldn’t be done in silos. Do it with a partner or a team - important to mitigate bias.

5. Qualtrics have a lot of resources e.g. https://www.qualtrics.com/blog/how-to-create-a-survey/

Further Reading

Zimmer, S. A., Powell, R. J., & Velásquez, I. C. (2024). Exploring Complex Survey Data Analysis Using R: A Tidy Introduction with {srvyr} and {survey}. Chapman & Hall: CRC Press. https://tidy-survey-r.github.io/tidy-survey-book/index.html

• Chapter 2 provides more detailed advice when developing surveys.

• Chapter 3 has a good overview of how to write a codebook / documentation for your survey – important if you are planning to deposit your data to into a repository

Thank you!

Thank you for your active participation!

Please scan the QR code on the left or click here to provide your feedback about this workshop.

Questions? email me at bellar@smu.edu.sg