That’s what preference testing is for. Don’t worry if you don’t know how to conduct one. We’ve put together this guide to help you out, together with questions you can use and example preference tests.
Key Takeaways
➡️ A preference test involves presenting testers with 2 or more design options and asking them which one they prefer
➡️ It is a quick and easy way to gather user feedback
➡️ For best results don’t include too many options as it will make deciding harder (2-3 is best)
➡️ Test with a minimum of 20 respondents to ensure statistical significance
➡️ Include follow-up questions to learn more about participants’ decisions
➡️For more in-depth research combine it with other research methods, such as surveys, task-based usability testing or A/B testing
Table of contents
- What is preference testing?
- When should you conduct the preference testing?
- Advantages of preference testing
- Limitations of preference testing
- How to conduct a preference test?
- Statistical significance of the data
- Preference tests questions
- Preference tests example
- Preference test vs A/B testing: What is the difference?
- Merging preference tests with other techniques in hybrid research
- What is stopping you from trying preference testing?
What is preference testing?
Preference testing is a research method for establishing a preference between a variety of options. Participants in the study are presented with 2 or more options and asked to choose which one they prefer. These tests are commonly used to determine which designs are the most aesthetically pleasing, understandable, or generate the most trust.
A preference test can be used to determine logo, icon, palette, design, mockup, content, copy, and text style preferences. This type of user research helps create a better understanding of what consumers want, which helps in designing the best possible product for them.
When doing logo design, for example, you can present consumers with two different logo styles. The comments from target users will not only tell you which designs they like, but also why they choose them. The feedback from consumers will offer you useful information about your design, allowing you to make modifications until you find the correct option.
We also recommend you check out how preference testing works at UXtweak:
When should you conduct the preference testing?
Preference testing is most effective in the early design phase and early stages of prototyping before you spend too much time and effort on the development. It allows you to decide where to focus your resources, which choice is the most practical, and how you may improve it.
Advantages of preference testing
- A quick and effective way to get feedback from your target audience
- Simulates what users do when purchasing products or services – choosing from a variety of options
- Simple and easy to understand, making them appropriate for even younger audiences
- Ability to get both qualitative and quantitative user feedback
Limitations of preference testing
While preference testing is amazingly simple yet successful, it has its drawbacks:
- When you ask your users to pick a favorite design, for example, you don’t get any information on the size of the preference. It’s possible that they like both designs equally well, but one slightly more. Or they choose the one they loathe slightly less because they had to, but they still find it repulsive on its own.
- Follow-up questions can help you learn more about why your users chose one choice over another, but individuals aren’t always great at explaining why they prefer something.
- You might not get a significant difference between the number of participants that choose one design vs the other (or multiple designs)
How to conduct a preference test?
1.) Set your objectives
Firstly, make sure you understand what kind of data you want to collect from your users and why. Are you developing a new product or redesigning one that already exists? Do you have two design variants or more in mind?
Whatever your study goal is, write it at the top of your research board so that your test participants can see it.
2.) Gather materials
Whether it’s logos, symbols, color palettes, or website designs, gather the items you wish to use for your preference testing.
In most preference tests two or three options are usually utilized, but you can use more if you wish. However, remember that the more designs you include in a task, the more difficult it will be for your participants to choose and then justify their choice.
Your preference test design options don’t have to have the same size or shape or be in away way similar. On the contrary the more similar your design concepts are, the harder it will be for the user to choose.
3.) Choose a tool
There are many great tools online that allow you to conduct preference tests easily. Take a look at our list of the best preference testing tools and you will find one that suits your needs.
You can also use the Preference Testing tool from UXtweak that has all of the features you’ll need to conduct thorough research. It’s simple to set up and use, and creating a UX preference test takes no more than 5 minutes.
4.) Set up your study
For every tool the setup of a preference test will be slightly different, however at its core it always means gathering all your prepared material (like tasks, questions, and pictures or videos) and pouring it into the tool.
Here is how you can set up your preference test with UXtweak
- Name your study and choose a language.
- Create tasks, where the respondents are supposed to express their preference for one of your designs.
- Design a variety of follow-up questions to collect qualitative data from your respondents based on your research objectives and the type and amount of elements you’re comparing.
- Create your personalized messages to be included during the study, such as the welcome message, study instructions, thank you message or you can use the default message.
- Add your unique branding and you’re all set!
5.) Gather respondents
The respondents should reflect your customers, who will eventually be the ones to interact with the studied designs. The best approach is to speak directly to those who use – or are expected to use – the product or service and are aware of the context in which the designs are presented.
Don’t worry, you won’t have any difficulties with recruitment if you use UXtweak. You can send a link to your study via email or social media, or add an Onsite Recruiting Widget to your website to collect respondents right away. Alternatively, just use our User Panel to acquire responders for your UX study!
6.) Analyze the results
A preference test will provide 2 types of data:
Quantitative feedback
The number (and percentage) of participants who selected each design is the quantitative data preference tests will provide. You will be able to see it on the results page and determine which option was preferred at a glance.
Qualitative feedback
Your qualitative data comes from follow-up questions and interviewing test participants about why they chose the design they did. You can divide their input into smaller groups to obtain a clearer idea of how many people had similar responses.
We’ve put together a helpful guide to make the analyzing process easier for you!
Statistical significance of the data
Statistical significance is the likelihood that the best design was actively chosen and determined randomly by chance. The amount of significance will vary according to the sample size. The larger the sample size, the more relevant the results.
It will also depend on the degree of difference between the designs, as two identical designs will make distinguishing one version from another difficult.
To be statistically relevant, preference tests should be carried out with at least 20 users. The panel should not solely include people from your typical intended audience. It should also consider the specific circumstances and viewpoints relevant to the design at hand. If you’re testing a design to attract potential leads, you could also try to draw in non-customers who fit your standard user’s profile in some way.
Preference tests questions
It’s generally a good idea to provide participants with some background information when writing questions.
When comparing two different icons, for example, don’t just question, “Which icon do you prefer?” Include details like “Which icon do you prefer for incoming messages?” or “Which icon do you think is more reliable?” This level of specificity allows you to get more relevant results.
However, avoid being leading in your questions. People are often trying to be accommodating, so if they suspect you like one of the designs, it may influence them.
The questions you will ask will always be highly tailored to your situation, however, here are some example questions you can ask:
-
Which version of the homepage evokes more trust in you?
-
Which font type and size do you find more comfortable to read?
-
Choose an icon that in your opinion more closely represents “volunteering”.
-
Which color scheme do you prefer for this app?
-
Which version of the notification design do you find more attention-grabbing?
-
Which option would you choose to access this feature – a single button on the homepage or a dropdown menu from the navigation bar? (Provide design prototypes for the user)
-
Which menu style do you prefer for this e-commerce website?
-
Which version of the button copy do you find more compelling?
-
Which layout do you find more intuitive for navigating through the app’s features?
-
Which product listing format do you prefer – a grid or a list view?
Preference tests example
There are many situations in which testing to see what users prefer is beneficial. It can answer questions like:
-
Which logo design and font type is best for your product?
-
Which theme do users prefer for your app?
-
What video clip to use for your new marketing campaign?
-
Which sound files do your listeners like more for a podcast intro jingle?
-
Which icon clearly communicates the intended message?
-
What landing page layout is more visually appealing to your users?
To see how these preference studies might look, check out these demos⬇️
Preference test vs A/B testing: What is the difference?
Sometimes people mix up A/B testing and preference testing. It is understandable as these methods for user and market research have the same goal: to determine which option to choose, based on quantitative data rather than your own preferences.
However, the process of these two methods is quite different. In preference tests, participants are asked to express their preference between 2 or more designs. A/B testing involves randomly assigning users to different versions of a design to determine which performs better. Preference testing can also involve qualitative feedback, in the form of follow-up questions, while A/B testing is a more quantitative method.
To run a preference test is significantly easier and quicker.
Merging preference tests with other techniques in hybrid research:
Because preference testing is constitutionally subjective and can only provide surface-level perspectives, it is best used in conjunction with other methods such as interviews and/or surveys to collect deeper insights and data that can help define the reasons behind participant decisions. It can also be used in tandem with A/B tests and behavioral analytics to gather more quantitative results.
What is stopping you from trying preference testing?
When it comes to design research, preference tests are one the easiest ways to get quick and actionable feedback from your customers. It’s important to test your ideas with your prospective users and not to rely solely on your own assumptions when it comes to designing a stellar product.
UXtweak is here to help you to get started. So, don’t wait around – register for free or log in to your UXtweak account and try your first preference test right away!
