You can’t read marketing blogs these days without hearing about “data-driven decisions.” Marketers are expected to have transitioned from the “Don Draper” era (in which gut feel and instinct ruled) to the “Data-driven” era (in which analytics, benchmarks, and consumer feedback should be in heavy supply).
The good news is this appetite for feedback is driving more online surveys being run across every organization. The bad news is most professionals don’t know the best practices for conducting a survey. Seemingly minor mistakes can throw an entire dataset into question. Below are some common mistakes I’ve seen survey rookies make, and how to fix them.
1. Reaching biased respondents
You have a powerful survey written; now what? Time to find respondents! Most marketers will trend towards free options: Your friends & family, your Twitter followers, your customers. But, all respondents are not created equal. Your family & friends won’t be honest about the shortcomings of your new product. Your Twitter followers aren’t a true cross-section of the population. And, it’s essential to reach non-customers for many survey goals, such as understanding why people don’t buy your product. Plus, you need to avoid the “survey fatigue” caused by hitting your customer list too often.Fortunately, new services (like my company Survata) let you reach highly-targeted respondents, starting at only $1 per respondent. You can also find affordable respondents on Amazon’s Mechanical Turk (for surveys in which it’s acceptable to over-index on certain demographic groups and income bands.)
2. Writing biased questions
It’s probably unsurprising to hear some surveys contain poorly written question. But, is it surprising to hear that essentially all surveys initially contain some fatal flaws? There are a myriad of traps you need to avoid – leading questions, non-MECE responses, double-barreled questions, etc. You can read more about specific errors in this blog post.
To be safe, have an experienced researcher review your questions before you launch your survey. Many survey services include this feature. Alternatively, find a professional researcher, or academic with polling experience, to take a quick look.
3. Treating online surveys like “pen & paper surveys”
I’ve always wondered why most survey software treat mouse & screen like pen & paper – i.e., most online surveys rely on question types (like closed-ended multiple choice) that existed decades ago. This is a shame, since online interactions allow for much richer data collection.
Instead, you should always be adding new question types to your repertoire. You can have respondents click on an image and then view the heat map of their click – this is especially useful for surveys involving shelf-set testing or mobile app testing.
Here’s a good rule of thumb: When you select a survey software company, ask them, “What’s your newest question type?”
4. Ignoring margin of error
Isn’t it exciting when you see the “winner” from a survey result? You may be testing creative, new product names, packaging, etc. Package A got 55% of votes, and Package B only got 45%! Well, that result is less insightful if you only surveyed 100 respondents, since that equates to almost a 10% margin of error (depending on a few factors).
You can use free calculators to play with the margin of error math. Or, check out an online stats package, like Statwing, which will display error bars on your survey results so you can quickly visualize statistical significance.
5. Missing insightful crosstabs
You may be tempted to quickly glean the main takeaways from your survey data’s “top lines.” But often, powerful clues are hidden deeper in the data. How did Question 5’s answers vary by females, or females over 40? (Note that your margin or error will change if you’re only viewing a subset of data.) How did people who answered “Yes” to Question #4 answer Question #2?
If you hire a market research firm, they’ll often prepare cross-tabs reports for you. If you’re running a survey on your own, you can work magic with pivot tables in programs like Excel. Before you even view your survey results, prepare a list of relevant questions you’d like to answer with crosstabs. Then, run additional ad hoc analyses based on what catches your eye in the data.
Best of luck with your survey efforts. Avoid these mistakes, and you can proudly declare your next marketing decision is properly data-driven.
To learn how surveys and other market research techniques can help your organization achieve success, download our free eBook.
Editor's Note:
This post was written by Chris Kelly, CEO of Survata.
About Survata
Survata is the simplest way to conduct high quality survey research. Survata serves clients ranging from Fortune 100 companies to small businesses, and everyone in between. They're a small team in San Francisco, but they serve clients across the globe. Survata promises fast, accurate, and affordable surveys. Chris started Survata in 2012, having been a frustrated user of outdated and expensive survey solutions. He has a decade of experience building and working with startups. He has also previously worked in venture capital and management consulting. Chris graduated from the University of Notre Dame.
