x

Register Now to Beegin Your Journey!

Register Now For Free to Beegin Your Journey!

Register Now to Beegin Your Journey!Register for free
Preference Testing in UX + Examples & Questions
User Research

Published on April 21, 2023

Preference Testing in UX + Examples & Questions

Imagine this. Your company's logo has undergone a significant transformation. You spent a lot of time and money on it, and now you're ready to put it up on your website. But you are concerned about how your customers would react. Well, why don't you just ask them about their opinion?

That’s what preference testing is for. Don’t worry if you don’t know how to conduct one. We’ve put together this guide to help you out, together with questions you can use and example preference tests.  

Key Takeaways

➡️ A preference test involves presenting testers with 2 or more design options and asking them which one they prefer

➡️ It is a quick and easy way to gather user feedback

➡️ For best results don’t include too many options as it will make deciding harder (2-3 is best)

➡️ Test with a minimum of 20 respondents to ensure statistical significance

➡️ Include follow-up questions to learn more about participants’ decisions

➡️For more in-depth research combine it with other research methods, such as surveys, task-based usability testing or A/B testing

What is preference testing?

Preference testing is a research method for establishing a preference between a variety of options. Participants in the study are presented with 2 or more options and asked to choose which one they prefer. These tests are commonly used to determine which designs are the most aesthetically pleasing, understandable, or generate the most trust.

A preference test can be used to determine logo, icon, palette, design, mockup, content, copy, and text style preferences. This type of user research helps create a better understanding of what consumers want, which helps in designing the best possible product for them.

When doing logo design, for example, you can present consumers with two different logo styles. The comments from target users will not only tell you which designs they like, but also why they choose them. The feedback from consumers will offer you useful information about your design, allowing you to make modifications until you find the correct option.

We also recommend you check out how preference testing works at UXtweak:

When should you conduct the preference testing?

Preference testing is most effective in the early design phase and early stages of prototyping before you spend too much time and effort on the development. It allows you to decide where to focus your resources, which choice is the most practical, and how you may improve it. 

Advantages of preference testing

  • A quick and effective way to get feedback from your target audience
  • Simulates what users do when purchasing products or services – choosing from a variety of options
  • Simple and easy to understand, making them appropriate for even younger audiences
  • Ability to get both qualitative and quantitative user feedback

Start Preference Testing with UXtweak

The quickest and easiest way to gather user feedback and validate your decisions.

Try for free
Start Preference Testing with UXtweak

Limitations of preference testing

While preference testing is amazingly simple yet successful, it has its drawbacks:

  • When you ask your users to pick a favorite design, for example, you don’t get any information on the size of the preference. It’s possible that they like both designs equally well, but one slightly more. Or they choose the one they loathe slightly less because they had to, but they still find it repulsive on its own.
  • Follow-up questions can help you learn more about why your users chose one choice over another, but individuals aren’t always great at explaining why they prefer something.
  • You might not get a significant difference between the number of participants that choose one design vs the other (or multiple designs)
preference test

How to conduct a preference test?

1.) Set your objectives

Firstly, make sure you understand what kind of data you want to collect from your users and why. Are you developing a new product or redesigning one that already exists? Do you have two design variants or more in mind?

Whatever your study goal is, write it at the top of your research board so that your test participants can see it.

2.) Gather materials

Whether it’s logos, symbols, color palettes, or website designs, gather the items you wish to use for your preference testing.

In most preference tests two or three options are usually utilized, but you can use more if you wish. However, remember that the more designs you include in a task, the more difficult it will be for your participants to choose and then justify their choice.

Your preference test design options don’t have to have the same size or shape or be in away way similar. On the contrary the more similar your design concepts are, the harder it will be for the user to choose.

3.) Choose a tool

There are many great tools online that allow you to conduct preference tests easily. Take a look at our list of the best preference testing tools and you will find one that suits your needs.

preference testing tool

You can also use the Preference Testing tool from UXtweak that has all of the features you’ll need to conduct thorough research. It’s simple to set up and use, and creating a UX preference test takes no more than 5 minutes.

4.) Set up your study

For every tool the setup of a preference test will be slightly different, however at its core it always means gathering all your prepared material (like tasks, questions, and pictures or videos) and pouring it into the tool.

Here is how you can set up your preference test with UXtweak

  1. Name your study and choose a language.
  2. Create tasks, where the respondents are supposed to express their preference for one of your designs.
  3. Design a variety of follow-up questions to collect qualitative data from your respondents based on your research objectives and the type and amount of elements you’re comparing.
  4. Create your personalized messages to be included during the study, such as the welcome message, study instructions, thank you message or you can use the default message.
  5. Add your unique branding and you’re all set!
preference testing

5.) Gather respondents

The respondents should reflect your customers, who will eventually be the ones to interact with the studied designs. The best approach is to speak directly to those who use – or are expected to use – the product or service and are aware of the context in which the designs are presented.

Don’t worry, you won’t have any difficulties with recruitment if you use UXtweak. You can send a link to your study via email or social media, or add an Onsite Recruiting Widget to your website to collect respondents right away. Alternatively, just use our User Panel to acquire responders for your UX study!

6.) Analyze the results

A preference test will provide 2 types of data:

Quantitative feedback

The number (and percentage) of participants who selected each design is the quantitative data preference tests will provide. You will be able to see it on the results page and determine which option was preferred at a glance.

Qualitative feedback

Your qualitative data comes from follow-up questions and interviewing test participants about why they chose the design they did. You can divide their input into smaller groups to obtain a clearer idea of how many people had similar responses.

We’ve put together a helpful guide to make the analyzing process easier for you!

Try Preference Testing with UXtweak

Set up your study in minutes, get actionable insights and handy PDF reports of the results.

Try for free
Try Preference Testing with UXtweak

Statistical significance of the data

Statistical significance is the likelihood that the best design was actively chosen and determined randomly by chance. The amount of significance will vary according to the sample size. The larger the sample size, the more relevant the results.

It will also depend on the degree of difference between the designs, as two identical designs will make distinguishing one version from another difficult.

To be statistically relevant, preference tests should be carried out with at least 20 users. The panel should not solely include people from your typical intended audience. It should also consider the specific circumstances and viewpoints relevant to the design at hand. If you’re testing a design to attract potential leads, you could also try to draw in non-customers who fit your standard user’s profile in some way.

preference testing

Preference tests questions

It’s generally a good idea to provide participants with some background information when writing questions.

When comparing two different icons, for example, don’t just question, “Which icon do you prefer?” Include details like “Which icon do you prefer for incoming messages?” or “Which icon do you think is more reliable?” This level of specificity allows you to get more relevant results. 

However, avoid being leading in your questions. People are often trying to be accommodating, so if they suspect you like one of the designs, it may influence them.

The questions you will ask will always be highly tailored to your situation, however, here are some example questions you can ask:

  • Which version of the homepage evokes more trust in you?

  • Which font type and size do you find more comfortable to read?

  • Choose an icon that in your opinion more closely represents “volunteering”.

  • Which color scheme do you prefer for this app?

  • Which version of the notification design do you find more attention-grabbing?

  • Which option would you choose to access this feature – a single button on the homepage or a dropdown menu from the navigation bar? (Provide design prototypes for the user)

  • Which menu style do you prefer for this e-commerce website?

  • Which version of the button copy do you find more compelling?

  • Which layout do you find more intuitive for navigating through the app’s features?

  • Which product listing format do you prefer – a grid or a list view?

Preference tests example

There are many situations in which testing to see what users prefer is beneficial. It can answer questions like:

  • Which logo design and font type is best for your product?

  • Which theme do users prefer for your app?

  • What video clip to use for your new marketing campaign?

  • Which sound files do your listeners like more for a podcast intro jingle?

  • Which icon clearly communicates the intended message?

  • What landing page layout is more visually appealing to your users?

To see how these preference studies might look, check out these demos⬇️

Choosing the correct clip

Preference Test
Choosing the correct clip

What icon designs do my users like better?

Preference Test
What icon designs do my users like better?

Preference test vs A/B testing: What is the difference?

Sometimes people mix up A/B testing and preference testing. It is understandable as these methods for user and market research have the same goal: to determine which option to choose, based on quantitative data rather than your own preferences.

However, the process of these two methods is quite different. In preference tests, participants are asked to express their preference between 2 or more designs. A/B testing involves randomly assigning users to different versions of a design to determine which performs better. Preference testing can also involve qualitative feedback, in the form of follow-up questions, while A/B testing is a more quantitative method.

To run a preference test is significantly easier and quicker.

Merging preference tests with other techniques in hybrid research:

Because preference testing is constitutionally subjective and can only provide surface-level perspectives, it is best used in conjunction with other methods such as interviews and/or surveys to collect deeper insights and data that can help define the reasons behind participant decisions. It can also be used in tandem with A/B tests and behavioral analytics to gather more quantitative results.

What is stopping you from trying preference testing?

When it comes to design research, preference tests are one the easiest ways to get quick and actionable feedback from your customers. It’s important to test your ideas with your prospective users and not to rely solely on your own assumptions when it comes to designing a stellar product.

UXtweak is here to help you to get started. So, don’t wait around – register for free or log in to your UXtweak account and try your first preference test right away!

Start Preference Testing with UXtweak

The quickest and easiest way to gather user feedback and validate your decisions.

Try for free
Start Preference Testing with UXtweak

People also ask (FAQ)

What’s the difference between A/B and preference testing?

The main difference between A/B vs preference testing is their approach. A/B testing consists of comparing 2+ versions of a product/design to see which performs better in quantitative terms. The importance here is on performance. Meanwhile, preference testing is more focused on understanding the user preference and qualitative “why” or logic behind that preferences, giving  in-depth nuanced insights. Another key difference is that with the A/B tests each participant gets to see only one version. Compared to that with the preference test you provide participants with all alternatives and you ask them to pick one.

How to analyze the results of a preference test?

To analyze the results of a preference test you have to:

 

  1. gather insights using statistical analysis methods
  2. consider statistical significance
  3. discover main data trends and patterns 
  4. analyze the outcome 

For faster, more reliable analysis, utilize a preference testing tool that automatically conducts statistical analysis and provides powerful data visualizations based on research outcomes.

How to select the right participants for preference testing?

To select the right participants for a preference test, it’s vital to correctly define the target audience of your study. Consider factors such as users’ demographics, goals and target behavior patterns to ensure a representative sample of users. To recruit the participants, you can use various channels such as social media, onsite recruitment or order participants from specialized User Panels.

Tereza Stehlikova
April 21, 2023
All author's articles

Tereza Stehlikova is a Marketing Specialist at UXtweak where she shares her expertise by leveraging the company's communications and PR activities. With a degree in International relations, she became passionate about user experience design and quickly found herself at UXtweak, working closely with designers and researchers.

Tereza is also a big part of the UX Research Geeks podcast, responsible for collaborating with our host Tina and getting new inspiring UX people to come talk to us and share their experiences.

Learn why these premium brands trust UXtweak

UXtweak is buzzing with expert UX research, making thousands of products
more user friendly every day

michelin-logo logo-hp logo-bmc

You might be interested in