12 Common Questionnaire Design Mistakes That Can Ruin Your Research

Questionnaire design mistakes are the single biggest reason good research ends up with data nobody can trust. After twelve years of pulling dissertations back from the edge of unusable data, I can tell you most of these mistakes are invisible to the person who wrote the questionnaire.

I’m Siddharth. I hold an MBA in Finance and an M.Tech. I’ve spent the last 12 years working as a research and statistics consultant.

Most of that time goes into fixing questionnaire design mistakes inside dissertation and thesis data before they turn into failed defences. Everything below comes from questionnaires I have personally reviewed, fixed, or had to explain to a supervisor on someone else’s behalf. You can find more of my work on LinkedIn.

Why Questionnaire Design Mistakes Matter More Than You Think

Here is a scene I have sat through more times than I can count. A student finishes data collection, runs the analysis, and the Cronbach’s alpha comes back at 0.41. The supervisor asks what went wrong.

Nobody can point to a single cause, because the cause was never one big error. It was four or five common mistakes in questionnaire design, made months earlier, before a single response had been collected.

A questionnaire design mistake is any flaw in how a question, scale, or response option is written that stops respondents from answering accurately or consistently. It does not announce itself while you are writing the survey. It shows up later, as low reliability scores, constructs that will not load together the way they should, or results that quietly contradict common sense.

Most guides on this topic, including well-regarded ones like Nielsen Norman Group’s survey design guidance, stop at “avoid bias” and move on. That advice is not wrong, it is just incomplete. What actually matters for your survey research design is knowing which mistake breaks which part of your analysis, and catching it before you are staring at a dataset you cannot defend in your methodology chapter.

The cost of these mistakes is rarely just a few confusing questions. It is weeks of re-collecting primary data. It is a defence where your supervisor asks why your constructs do not correlate the way the literature says they should.

Or it is a dissertation where the analysis stops matching the research question. If you are measuring something complex such as attitude or satisfaction, deciding between single-item and multi-item measures matters right here, before you write a single item.

Bad Survey Questions Examples (Quick Before-and-After Table)

If you only have two minutes, this table previews the questionnaire design mistakes covered in the rest of the article.

Mistake	Bad Example	Fixed Example
Leading question	How satisfied are you with our excellent support?	How satisfied are you with our support?
Loaded question	What do you like most about our app?	Have you used our app? If yes, what do you like most?
Double-barreled	Rate the price and quality of our product.	Rate the price. Rate the quality, as a separate item.
Ambiguous wording	Do you exercise often?	How many of the past 7 days did you exercise 30+ minutes?
Wrong Likert points	3-point scale for job satisfaction	5 or 7-point scale, labelled at every point
Incomplete options	Employment status – Full-time / Part-time / Unemployed	Add Student, Retired, Self-employed, Other
Unbalanced scale	Excellent / Very Good / Good / Fair	Strongly Disagree to Strongly Agree
Dichotomous question	Are you satisfied? Yes / No	How satisfied are you? (5-point scale)
Jargon	Rate our synergistic collaboration framework.	How well do teams work together?
No neutral option	Forced Agree / Disagree only	Add a Neutral / Not Applicable choice
Poor question order	Economy question right before personal finance question	Separate the two with unrelated items between them
No pilot testing	Full survey sent with zero pretesting	10-15 person pilot run before launch

Wording Mistakes That Bias Your Answers – Leading and Biased Questionnaire Questions

The four questionnaire design mistakes below do the most damage to how honestly respondents answer, starting with the one I see most often in dissertation work.

Mistake 1 – Leading and Biased Questionnaire Questions

A leading question nudges the respondent toward the answer you already wanted. The wording does the persuading, so the respondent feels like they are answering honestly even while the sentence structure is steering them.

Bad example – “How satisfied are you with our excellent customer support?” Good example – “How satisfied are you with our customer support?”

The word “excellent” plants an opinion inside the question. Strip out adjectives that praise or criticise the thing you are asking about, and most of the bias disappears with it.

This is also where leading and biased questionnaire questions do the most quiet damage to your findings. If every respondent is nudged toward “satisfied,” your mean score inflates, your variance shrinks, and any correlation you run against that variable looks artificially clean.

Two named patterns sit behind a lot of this damage. Acquiescence bias is the tendency to agree with whatever is asked regardless of the actual content, and it gets worse the longer and more tiring your questionnaire feels. Social desirability bias is the tendency to answer in the way that looks best rather than what is actually true, and leading questions make both worse at the same time.

Mistake 2 – Loaded or Assumptive Questions

A loaded question assumes something about the respondent that may not be true, then asks them to answer anyway.

Bad example – “What do you like most about using our mobile app?” Good example – “Have you used our mobile app? If yes, what do you like most about it?”

The fix is almost always a filter question before the loaded one. Ask whether the condition applies first, then route only the relevant respondents to the follow-up with skip logic.

Mistake 3 – Double-Barreled Questions

A double-barreled question packs two separate ideas into one sentence and asks for a single answer.

Bad example – “How satisfied are you with the price and quality of our product?” Good example – “How satisfied are you with the price of our product?” as one item, with a separate item for quality.

I see this mistake more than any other in dissertation questionnaires, and it is the one that causes the most damage once you reach analysis. When two concepts share one item, the respondent is really answering an average of two different opinions. That noise shows up later as weak factor loadings, or as items that refuse to group together cleanly in your factor analysis.

A student I worked with last year had six double-barreled items buried inside a 40-question survey. Her exploratory factor analysis would not converge into clean components no matter how we rotated it, and tracing the problem back to those six items cost two full days we did not need to lose.

Mistake 4 – Ambiguous or Vague Wording

Vague wording leaves too much room for interpretation, so two respondents can read the same question completely differently.

Bad example – “Do you exercise often?” Good example – “In the past seven days, how many days did you exercise for at least 30 minutes?”

Words like “often,” “regularly,” “a lot,” and “usually” mean different things to different people. Replace them with a specific timeframe, a specific count, or a specific unit wherever you can.

Scale and Response-Option Mistakes

These four questionnaire design mistakes live inside the scale itself, not the wording, and they are just as damaging to your statistics.

Mistake 5 – Wrong Number of Likert Scale Points

Too few points on a Likert scale flatten real differences between respondents. Too many points add noise, because nobody can reliably tell the difference between a 7 and an 8 on a 10-point scale.

Bad example – a 3-point scale (Disagree, Neutral, Agree) measuring something nuanced like job satisfaction. Good example – a 5 or 7-point scale, labelled at every point, running from Strongly Disagree to Strongly Agree.

Five to seven points is the range most methodology sources settle on for general use, and it is the range I default to unless the literature for your specific construct says otherwise. Before you commit to a scale, check whether a validated version already exists, since building your own from scratch is one of the quietest ways to lose validity. Once your scale is set, the next decision is how you plan to design Likert scale questions for analysis, so the data actually supports the test you intend to run.

Mistake 6 – Missing or Incomplete Response Options

Response options need to cover every realistic answer, including the ones you did not expect.

Bad example – “What is your employment status?” with options Full-time, Part-time, Unemployed. Good example – the same question with Student, Retired, Self-employed, and Other (please specify) added.

When options are not exhaustive, respondents pick whatever is closest instead of what is actually true. That single substitution quietly skews every percentage calculated from that variable later.

Mistake 7 – Unbalanced or Inconsistent Scales

A scale is unbalanced when it offers more positive options than negative ones, or when the gaps between points are not equal.

Bad example – Excellent, Very Good, Good, Fair, with no negative option at all. Good example – Strongly Disagree, Disagree, Neutral, Agree, Strongly Agree.

Switching scale lengths or directions partway through a questionnaire is just as damaging. If respondents answer a 5-point scale on page one and a 7-point scale on page three, they have to re-orient every time, and that mental friction adds noise to both sets of answers.

Mistake 8 – Dichotomous (Yes or No) Questions That Needed a Scale

A yes or no option forces respondents into two boxes when their actual opinion sits somewhere in between.

Bad example – “Are you satisfied with our service?” (Yes / No) Good example – “How satisfied are you with our service?” on a 5-point scale from Very Dissatisfied to Very Satisfied.

Yes/no data also limits what you can do statistically later. A continuous or ordinal scale lets you run correlations, regressions, and group comparisons that a binary variable simply cannot support.

Structural and Process Mistakes

The last four questionnaire design mistakes on this list are structural rather than verbal, and they are the easiest to miss because nothing about them looks wrong on the page.

Mistake 9 – Jargon and Overly Complex Language

Technical terms and formal phrasing push respondents to guess at meaning instead of answering honestly.

Bad example – “Rate the synergistic efficacy of our cross-functional collaboration framework.” Good example – “How well do different teams work together on shared projects?”

If you would not say the sentence out loud to a friend, rewrite it. Plain language gets you more honest answers and far fewer blank fields.

Mistake 10 – No Neutral or Opt-Out Option

Forcing an opinion on every question pushes respondents who genuinely have no view toward a random guess, which adds noise you cannot separate from real signal later.

Bad example – a satisfaction scale with no Neutral or Not Applicable option. Good example – the same scale with a clearly labelled Neutral or Not Applicable choice included.

A neutral option is not a weakness in your design. It is what lets you tell the difference between someone who genuinely disagrees and someone who simply has no basis to answer.

Mistake 11 – Poor Question Order (Order Bias)

Earlier questions can prime how respondents answer later ones, even when the two topics seem unrelated on the surface.

Bad example – asking “How would you rate the national economy?” right before “How would you rate your own financial situation?” Good example – separating the two topics with unrelated questions in between, or randomising order where sequence does not matter.

Put sensitive or identity questions near the end of the questionnaire, not the start. Respondents tend to answer more honestly once they are already invested in finishing.

Mistake 12 – Skipping Pilot Testing

A questionnaire that reads perfectly to you can still confuse the first ten people who actually fill it out, and there is no way to know that without testing it first.

Bad example – sending the full questionnaire straight to your sample with zero pretesting. Good example – running it past 10 to 15 people from your target population first, timing it, and fixing whatever confuses them.

Before you decide how large that pilot or main sample needs to be, run the numbers through a sample size calculator rather than guessing. If your questionnaire draws on existing records alongside fresh responses, settle how you are deciding between primary and secondary data before the pilot, not after. Once pilot data is in, this is also the point where you should be planning questionnaire validation, rather than waiting until the full dataset is collected.

Another student I worked with, this one studying consumer behaviour, skipped piloting a 60-item survey entirely. Fourteen respondents in, we discovered question 23 was being read two different ways depending on the respondent’s age group. We caught it only because she agreed to pause and review the first batch of live responses early, which is effectively what a pilot test does anyway, at a much higher cost in time.

Questionnaire Validity and Reliability Problems Caused By These Mistakes

Validity vs Reliability, Explained Plainly

Reliability asks whether your questionnaire gives consistent results if you used it again under the same conditions. Validity asks whether it is actually measuring the construct you think it is measuring. A questionnaire can be reliable without being valid, consistently measuring the wrong thing, but it cannot be valid without being reliable first.

Validity itself splits into a few named types worth knowing. Content validity asks whether your items actually cover the full concept you claim to measure, not just a slice of it. Face validity asks whether the questionnaire looks right to someone glancing at it for the first time, which matters more for respondent trust than for statistics.

Construct validity matters most for dissertation work. It asks whether your items measure the underlying idea itself, rather than something merely related to it. Cronbach’s alpha, in turn, is really measuring internal consistency, which is only one ingredient of reliability, not the whole picture.

Every one of these questionnaire design mistakes feeds into one or both of these ideas. Leading questions damage validity, because they measure your bias more than the respondent’s real opinion. Double-barreled and ambiguous items damage reliability, because the same respondent could plausibly answer differently on a different day, simply because the question itself is unstable.

Most blog content on this topic never goes past defining the mistake. In twelve years of reviewing dissertation data, the single question I get asked most is why a Cronbach’s alpha score came back so low. The honest answer is almost always traceable back to three or four of the mistakes covered in this article. Alpha is a useful diagnostic, but it is not the only reliability measure worth knowing, and it is worth understanding its limits before you lean on it too heavily.

The scale of measurement you end up with also decides which statistical test is appropriate in the first place. Likert data is ordinal, not interval, no matter how many articles quietly treat it as a continuous number, and that single fact decides whether you should be choosing between Pearson and Spearman correlation for your analysis later.

How to Improve Survey Questionnaire Design – Quick Pre-Launch Checklist

Run this checklist against your draft before you send it to a single respondent, since it catches most questionnaire design mistakes in minutes.

1. Read every question out loud. If it sounds like it is nudging an answer, rewrite it.
2. Check every question for two ideas hiding inside one sentence.
3. Replace vague words like “often” and “regularly” with specific numbers or timeframes.
4. Confirm your Likert scale has 5 to 7 points, labelled consistently throughout.
5. Make sure response options cover every realistic answer, including “Other.”
6. Check that your scale is balanced, with equal positive and negative options and equal spacing.
7. Replace any yes/no question measuring an opinion with a proper scale.
8. Cut jargon. Read it back as if you are asking a friend, not a committee.
9. Add a neutral or not applicable option wherever a genuine neutral opinion is possible.
10. Review your question order for anything that could prime a later answer.
11. Pilot the questionnaire with 10 to 15 people from your actual target population.
12. Check your sample size before launch, not after you start seeing low response counts.

A supervisor’s sign-off on your questionnaire is not the same as a statistical check. Approval usually means the questions look sound and cover the right ground, not that every item will survive a reliability and validity test once real data comes in.

If you have run through this list and you are still unsure whether your questionnaire will hold up under a supervisor’s or examiner’s scrutiny, that uncertainty is worth taking seriously rather than hoping it resolves itself on its own. This is exactly the point where a second, trained set of eyes on your instrument helps most, before data collection rather than after. How to know if you need a dissertation expert is a reasonable next read if that is where you are stuck.

Frequently Asked Questions