That’s what preference testing is for. Don’t worry if you don’t know how to conduct one. We’ve put together this guide to help you out, together with questions you can use and example preference tests.
Key Takeaways
➡️ A preference test involves presenting testers with 2 or more design options and asking them which one they prefer
➡️ It is a quick and easy way to gather user feedback
➡️ For best results don’t include too many options as it will make deciding harder (2-3 is best)
➡️ Test with a minimum of 20 respondents to ensure statistical significance
➡️ Include follow-up questions to learn more about participants’ decisions
➡️For more in-depth research combine it with other research methods, such as surveys, task-based usability testing or A/B testing
What is preference testing?
Preference testing is a research method for establishing a preference between a variety of options. Participants in the study are presented with 2 or more options and asked to choose which one they prefer. These tests are commonly used to determine which designs are the most aesthetically pleasing, understandable, or generate the most trust.
A preference test can be used to determine logo, icon, palette, design, mockup, content, copy, and text style preferences. This type of user research helps create a better understanding of what consumers want, which helps in designing the best possible product for them.
When doing logo design, for example, you can present consumers with two different logo styles. The comments from target users will not only tell you which designs they like, but also why they choose them. The feedback from consumers will offer you useful information about your design, allowing you to make modifications until you find the correct option.
We also recommend you check out how preference testing works at UXtweak:
When should you conduct the preference testing?
Preference testing is most effective in the early design phase and early stages of prototyping before you spend too much time and effort on the development. It allows you to decide where to focus your resources, which choice is the most practical, and how you may improve it.
Advantages of preference testing
- A quick and effective way to get feedback from your target audience
- Simulates what users do when purchasing products or services – choosing from a variety of options
- Simple and easy to understand, making them appropriate for even younger audiences
- Ability to get both qualitative and quantitative user feedback
Limitations of preference testing
While preference testing is amazingly simple yet successful, it has its drawbacks:
- When you ask your users to pick a favorite design, for example, you don’t get any information on the size of the preference. It’s possible that they like both designs equally well, but one slightly more. Or they choose the one they loathe slightly less because they had to, but they still find it repulsive on its own.
- Follow-up questions can help you learn more about why your users chose one choice over another, but individuals aren’t always great at explaining why they prefer something.
- You might not get a significant difference between the number of participants that choose one design vs the other (or multiple designs)
How to conduct a preference test?
1.) Set your objectives
Firstly, make sure you understand what kind of data you want to collect from your users and why. Are you developing a new product or redesigning one that already exists? Do you have two design variants or more in mind?
Whatever your study goal is, write it at the top of your research board so that your test participants can see it.
2.) Gather materials
Whether it’s creative logos, symbols, color palettes, or website designs, gather the items you wish to use for your preference testing.
In most preference tests two or three options are usually utilized, but you can use more if you wish. However, remember that the more designs you include in a task, the more difficult it will be for your participants to choose and then justify their choice.
Your preference test design options don’t have to have the same size or shape or be in away way similar. On the contrary the more similar your design concepts are, the harder it will be for the user to choose.
3.) Choose a tool
There are many great tools online that allow you to conduct preference tests easily. Take a look at our list of the best preference testing tools and you will find one that suits your needs.
You can also use the Preference Testing tool from UXtweak that has all of the features you’ll need to conduct thorough research. It’s simple to set up and use, and creating a UX preference test takes no more than 5 minutes.
4.) Set up your study
For every tool the setup of a preference test will be slightly different, however at its core it always means gathering all your prepared material (like tasks, questions, and pictures or videos) and pouring it into the tool.
Here is how you can set up your preference test with UXtweak
- Name your study and choose a language.
- Create tasks, where the respondents are supposed to express their preference for one of your designs.
- Design a variety of follow-up questions to collect qualitative data from your respondents based on your research objectives and the type and amount of elements you’re comparing.
- Create your personalized messages to be included during the study, such as the welcome message, study instructions, thank you message or you can use the default message.
- Add your unique branding and you’re all set!
5.) Gather respondents
The respondents should reflect your customers, who will eventually be the ones to interact with the studied designs. The best approach is to speak directly to those who use – or are expected to use – the product or service and are aware of the context in which the designs are presented.
Don’t worry, you won’t have any difficulties with recruitment if you use UXtweak. You can send a link to your study via email or social media, or add an Onsite Recruiting Widget to your website to collect respondents right away. Alternatively, just use our User Panel to acquire responders for your UX study!
6.) Analyze the results
A preference test will provide 2 types of data:
Quantitative feedback
The number (and percentage) of participants who selected each design is the quantitative data preference tests will provide. You will be able to see it on the results page and determine which option was preferred at a glance.
Qualitative feedback
Your qualitative data comes from follow-up questions and interviewing test participants about why they chose the design they did. You can divide their input into smaller groups to obtain a clearer idea of how many people had similar responses.
We’ve put together a helpful guide to make the analyzing process easier for you!
Best online Preference Testing Tools list:
For each tool in this article, you will find:
- A short description
- What features for preference testing are available
- Recruitment options
- Pricing
If you want to learn more about when and how to conduct preference testing, check out the Guide to Preference Testing in UX.
1. UXtweak
UXtweak is an all-in-one UX research platform with all kinds of tools for online user research and user testing in one place. Preference Test Tool is one of them. UXtweak specializes in remote usability testing and it is a great option for unmoderated preference tests which allows you to run your studies quickly, effectively, and with real users, even when you are on a budget. The platform’s UI is very simple and intuitive, where all functions can be easily found in the menu.
With UXtweak you can:
- Use preference testing to test images, text, and audio
- Set up the tasks
- Write messages and instructions for participants
- Create pre-study and post-study questionnaires with any text, images, or links
- Customize the test and Use your own brand colors and logo
Try UXtweak user testing tools as a respondent in these demos ⬇️
Recruitment Options
When it comes to recruiting participants for your preference tests, UXtweak offers three primary options:
- Share a study link with your own participants.
- Recruit testers from our cost-effective User Panel of over 155 million users with 2000 profiling attributes.
- Use our handy Onsite Recruiting widget to turn your website visitors into participants i your preference test
UXtweak pricing
There is a free forever plan for small-scale projects, and the Pro plan starts at 80€ / month, where you get unlimited tasks and participants, access to all tools, custom branding, and PDF reports. For more information see UXtweak pricing.
2. UsabilityHub (now Lyssna)
UsabilityHub is a remote user research platform with different testing tools to validate any design decision on videos, logos, packaging, or animations. Preference testing by UsabilityHub offers a simple interface with a few basic functions.
With UsabilityHub you can:
- Add welcome and final screen
- Add designs to be tested
- Write follow-up questions
It can also automatically calculate statistical significance when comparing two designs to let the tester know the relevance of the results.
Recruitment Options
UsabilityHub allows you to recruit respondents in two ways:
- Share a study link with your own testers (at no additional cost)
- Recruit participants from their 340K+ user panel
UsabilityHub Pricing
A free version is available only for testing that lasts a maximum of 2 minutes. The basic plan starts at $89/month, where you can test up to 5 minutes, and the Pro plan costs $199/month with multi-user plans, unlimited tests, and custom branding.
3. Useberry
A user testing service that offers several tools for user testing, such as a preference test. With their tool, Useberry recommends testing microcopies, CTAs, variation performance, and design variations.
With their tool, you are able to:
- Select a preference type (image or text)
- Create a task
- Add follow-up questions
- Include intro and thank you screen
Recruitment Options
For testing in Useberry, you can either bring your own testers or recruit from Useberry’s 240K user panel.
Useberry Pricing
With a free plan, you can collect up to 10 responses. The basic plan costs 28€ (33$) / month and allows 3 projects. Pro plan is 57€ (67$) / month, which contains an unlimited number of tests and responses.
4. SurveyMonkey
SurveyMonkey is a survey platform that helps companies measure and understand feedback to drive their growth and innovation. It focuses specifically on collecting feedback from users, customers, or employees through surveys. Its interface is suitable for more advanced users.
They don’t offer a tool called a Preference test specifically, but by creating a survey, you are able to collect feedback and other data in a way that can substitute preference study. This platform may be better for someone who wants to send out simple questionnaires because only the Pro plan lets you have access to all features and analysis reports.
However, there is also a tool similar to Preference Study called Product Concept Analysis – “Get your product concepts, feature optimizations, and ideas validated by a trusted audience – in less than an hour.”
Recruitment Options
You can recruit from their 175M+ user panel, or bring your own testers.
SurveyMonkey pricing
There is a free version of SurveyMonkey, but the features available are very limited. The personal plan then starts at 43€ (50$) / month, and the team plan (minimum 3 users) starts at 34€ (40$) / month. All plans are paid annually, which might be a disadvantage for occasional users.
5. Maze
Maze is a remote user testing platform that allows you to test anything from prototypes through copy to conducting a preference test. It gathers user feedback in one place and collects actionable insights across the teams.
Maze allows you to:
- Build your test as you want block-by-block
- Choose from pre-made templates
- Include welcome and final screen
Recruitment Options
With Maze you get access to their 70K+ user panel where you can recruit targeted testers, or, alternatively, just bring your own testers at no additional cost.
Maze pricing
Maze, like all the platforms above, has a free version, allowing up to 3 projects. The Pro plan starts at 75$ available for up to 10 projects.
Statistical significance of the data
Statistical significance is the likelihood that the best design was actively chosen and determined randomly by chance. The amount of significance will vary according to the sample size. The larger the sample size, the more relevant the results.
It will also depend on the degree of difference between the designs, as two identical designs will make distinguishing one version from another difficult.
To be statistically relevant, preference tests should be carried out with at least 20 users. The panel should not solely include people from your typical intended audience. It should also consider the specific circumstances and viewpoints relevant to the design at hand. If you’re testing a design to attract potential leads, you could also try to draw in non-customers who fit your standard user’s profile in some way.
Preference tests questions
It’s generally a good idea to provide participants with some background information when writing questions.
When comparing two different icons, for example, don’t just question, “Which icon do you prefer?” Include details like “Which icon do you prefer for incoming messages?” or “Which icon do you think is more reliable?” This level of specificity allows you to get more relevant results.
However, avoid being leading in your questions. People are often trying to be accommodating, so if they suspect you like one of the designs, it may influence them.
The questions you will ask will always be highly tailored to your situation, however, here are some example questions you can ask:
-
Which version of the homepage evokes more trust in you?
-
Which font type and size do you find more comfortable to read?
-
Choose an icon that in your opinion more closely represents “volunteering”.
-
Which color scheme do you prefer for this app?
-
Which version of the notification design do you find more attention-grabbing?
-
Which option would you choose to access this feature – a single button on the homepage or a dropdown menu from the navigation bar? (Provide design prototypes for the user)
-
Which menu style do you prefer for this e-commerce website?
-
Which version of the button copy do you find more compelling?
-
Which layout do you find more intuitive for navigating through the app’s features?
-
Which product listing format do you prefer – a grid or a list view?
Preference tests example
There are many situations in which testing to see what users prefer is beneficial. It can answer questions like:
-
Which logo design and font type is best for your product?
-
Which theme do users prefer for your app?
-
What video clip to use for your new marketing campaign?
-
Which sound files do your listeners like more for a podcast intro jingle?
-
Which icon clearly communicates the intended message?
-
What landing page layout is more visually appealing to your users?
To see how these preference studies might look, check out these demos⬇️
Preference test vs A/B testing: What is the difference?
Sometimes people mix up A/B testing and preference testing. It is understandable as these methods for user and market research have the same goal: to determine which option to choose, based on quantitative data rather than your own preferences.
However, the process of these two methods is quite different. In preference tests, participants are asked to express their preference between 2 or more designs. A/B testing involves randomly assigning users to different versions of a design to determine which performs better. Preference testing can also involve qualitative feedback, in the form of follow-up questions, while A/B testing is a more quantitative method.
To run a preference test is significantly easier and quicker.
Merging preference tests with other techniques in hybrid research:
Because preference testing is constitutionally subjective and can only provide surface-level perspectives, it is best used in conjunction with other methods such as interviews and/or surveys to collect deeper insights and data that can help define the reasons behind participant decisions. It can also be used in tandem with A/B tests and behavioral analytics to gather more quantitative results.
What is stopping you from trying preference testing?
When it comes to design research, preference tests are one the easiest ways to get quick and actionable feedback from your customers. It’s important to test your ideas with your prospective users and not to rely solely on your own assumptions when it comes to designing a stellar product.
UXtweak is here to help you to get started. So, don’t wait around – register for free or login to your UXtweak account and try your first preference test right away!
