Research Guide: Survey Design

If your research involves understanding the opinions, beliefs, or attributes of a large population, surveys are often the right tool. This guide covers the core design principles for producing valid, reliable survey data.

Define Your Variables First

Before writing a single question, define the variables your survey needs to measure. List each variable, then decide how to operationalize it: that is, what observable indicators will stand in for the underlying concept you care about.

In professionally designed surveys, the validity of multi-question operationalizations is tested statistically through techniques such as confirmatory factor analysis. These questions are then grouped and reported as a scale for regression or other quantitative analysis. For shorter or more informal surveys, at minimum ensure that each question maps clearly to a specific variable.

Open and Closed Questions

You will likely want both open and closed questions. In open questions, the respondent answers in their own words. In closed questions, they choose from a pre-defined set of options.

Example of a closed question: "How much do you agree or disagree with the following statement: BanklessDAO is awesome? (a) Strongly agree (b) Agree (c) Neutral (d) Disagree (e) Strongly Disagree"

In closed questions, the range of answers must be exhaustive (covering all possible responses) and mutually exclusive (no response fits more than one category). If you cannot design a closed question that meets both criteria, consider using an open question instead.

Generally, open questions should appear before closed questions on the same topic. This is because of anchoring: early questions prime respondents to think in certain ways, and closed questions are especially prone to creating anchoring effects.

That said, closed questions are easier to answer. Starting a survey with a few simple closed questions can improve completion rates before moving into more demanding open-ended territory.

Presentation

Everything should be designed to make it easy for your respondent. Specific guidance:

• ·Avoid jargon. Use simple, familiar words. In standard survey design, it is generally advised to aim for a grade 4 reading level — keep things simple, avoid acronyms, and aim for the broadest audience.
• ·Standardize presentation: consistent font, format, and visual hierarchy across all questions. Consider using darker or larger print for the questions themselves, and lighter print for answer options.
• ·Use formatting (bold, underline) to draw attention to key words: but use it sparingly. Too much formatting is as confusing as none.
• ·Example of good formatting use: "How often do YOU go to the doctor?" vs. "How often does YOUR SPOUSE go to the doctor?": emphasis clarifies who the referent is.

Common Errors and Tips

• ·Ask about one thing at a time. Avoid double-barreled questions: "How much do you enjoy cooking and eating?" should be split into two questions.
• ·Avoid leading or loaded questions. Ask about levels of satisfaction AND dissatisfaction, not just satisfaction. Framing shapes responses.
• ·Make questions relevant. Use skip logic in survey software to bypass questions that do not apply to a given respondent.
• ·Place sensitive questions (income, personal beliefs, demographics) at the end of the survey. Respondents are more likely to complete and share accurate data if you have already established rapport through easier questions.
• ·Start with easy, pleasant questions. Early engagement increases investment in completing the survey.
• ·Group questions on the same topic together. This reduces cognitive load and makes the survey feel more coherent.

References

• ·Rethink Research: 2018 SSRL Research Methods Summer Institute — Basic Survey Design. youtube.com/watch?v=IarEwuUP1oQ
• ·Dillman, Don A., Jolene D. Smyth, and Leah Melani Christian. Internet, Phone, Mail, and Mixed-Mode Surveys: The Tailored Design Method. Wiley, 2014.
• ·Krosnick, Jon A. "Survey research." Annual Review of Psychology 50.1 (1999): 537–567.
• ·Pew Research Center: Writing Survey Questions. pewresearch.org/our-methods/u-s-surveys/writing-survey-questions/