When and how to run a preference testing?
User Research

Published on January 5, 2022

When and how to run a preference testing?

Imagine this. Your company's logo has undergone a significant transformation. You spent a lot of time and money on it, and now you're ready to put it up on your website. But you are concerned about how your customers would react. Well, why don't you just ask them about their opinion?

That’s what preference testing is for. Don’t worry if you don’t know how to conduct one. We’ve put together this guide to help you out. Let’s get right into it! 

What is preference testing?

Preference testing is a research method for establishing a preference for a variety of options. Participants in the study are presented with 2-3 options and asked to choose which one they prefer. These tests are commonly used to determine which designs are the most aesthetically pleasing, understandable, or generate the most trust.

The Preference Test can be used to determine logo, icon, palette, design, mockup, content, copy, and text style preferences. This feedback contributes to the formation of a wider and larger image aimed at a deeper understanding of the optimum design for consumers. 

In a logo design, for example, you can present consumers with two different logo styles. The comments from users will not only tell you which designs they like but also why they choose them. The feedback from consumers will offer you useful information about your design, allowing you to make modifications until you find the correct option.

Advantages of preference testing

  • A quick and effective way to get feedback from users
  • Simulates what users do when purchasing products or services – choosing from a variety of options
  • Simple and easy to understand, making them appropriate for even younger audiences

preference test

When should you conduct the preference testing?

Preference testing is most effective early in the design and prototyping stages before you spend too much time and effort on the development. It allows you to decide where to focus your resources, which choice is the most practical, and how you may improve it. 

When redesigning a product or website, preference testing can be used to evaluate the old and new versions of the design, as well as to compare your design to that of your competitors.

Quantitative feedback

The number of participants who selected each design will be displayed on the results page. This is a simple design comparison from which the statistical significance of the outcome may be calculated.

This is defined by statistical significance as the probability that the best design is picked and does not outperform the other designs by chance.

Qualitative feedback

Your qualitative input comes from interviewing participants about why they chose the design they did. You can divide their input into smaller groups to obtain a clearer idea of how many people had similar responses.

How to conduct a preference test?

Set your objectives

Firstly, make sure you understand what kind of data you want to collect from your users and why. Are you developing a new product or redesigning one that already exists? Do you have two or more different designs in mind?

Whatever your study goal is, write it at the top of your research board so that your test participants can see it.

It’s generally a good idea to provide participants with some background information when writing instructions. When comparing two different icons, for example, don’t just question, “Which icon do you prefer?” Include details like “Which icon do you prefer for incoming messages?” or “Which icon do you think is more reliable?” This level of specificity allows you to analyze your theory more precisely. 

However, avoid being too direct in your questions. People are often trying to be accommodating, so if they suspect you like one of the designs, it may influence them. They’re also less critical, even when told to be truthful, so it’s sometimes better to avoid asking direct questions to find out why they like one style over another.

Gather materials

Whether it’s logos, symbols, color palettes, or website designs, gather the items you wish to use for your preference test. In a preference test, two or three options are usually utilized. Remember that the more designs you include in a task, the more difficult it will be for your participants to choose and then justify their choice.

Choose a tool 

The Preference Testing tool from UXtweak has all of the features you’ll need to conduct thorough research. It’s simple to set up and use, and creating a UX preference test takes no more than 5 minutes.

Furthermore, the user interface is extremely user-friendly, making it an excellent choice even for complete beginners. We provide you with a free plan for small projects and subscribing to one plan you het access to all of our UX research tools. Here is where you may sign up and create your first preference test.

Here is how you can set up your preference test with UXtweak

  1. Name your study and choose a language
  2. Create tasks, where the respondents are supposed to express their preference for one of your designs
  3. Design a variety of follow-up questions to collect qualitative data from your respondents based on your research objectives and the type and amount of elements you’re comparing.
  4. Create your personalized messages to be included during the study, such as the welcome message, study instructions, thank you a message or you can use the default message
  5. Add your unique branding and you’re all set!

preference testing

Gather respondents

The respondents should reflect your customers, who will eventually be the ones to interact with the studied designs. The best approach is to speak directly to those who use – or are expected to use – the product or service and are aware of the context in which the designs are presented.

Don’t worry, you won’t have any difficulties with recruitment if you use UXtweak. You can send a link to your study via email or social media, or add a Recruiting Widget to your website to collect respondents right away. Alternatively, just use our User Panel to acquire responders for your UX study!

Analyze the results

The most basic piece of information you’ll acquire from a preference test is how many times each design was chosen as the preferred option. This data is presented in both absolute numbers and percentages.

It’s time for our favorite part: the analysis, once you’ve gathered all of the qualitative and quantitative input. We’ve put together a helpful guide to make the process easier for you!

Limitations of preference testing

While preference testing is amazingly simple yet successful, it has its drawbacks.

  • When you ask your users to pick a favorite design, for example, you don’t get any information on the size of the preference. It’s possible that they like both designs equally well, but one slightly more. Or they choose the one they loathe slightly less because they had to, but they still find it repulsive on its own
  • Follow-up questions can help you learn more about why your users chose one choice over another, but individuals aren’t always adept at explaining why they prefer something
  • Another difficulty is that some individuals mix up A/B testing and preference testing. In preference tests, different options are evaluated based solely on respondents’ internal preferences, while in A/B testing they are evaluated based on respondents’ performance and behaviour.

What is stopping you from trying preference testing?

When it comes to design research, preference tests are one the easiest ways to get feedback from your customers. And it is even easier with UXtweak. So, don’t wait around – register or log in to your UXtweak account and try your first preference test right away!

Tereza Stehlíková
January 5, 2022
All author's articles

Marketing & Sales Assistant at UXtweak who likes to discover the charm of UX and travel around the world.

Try our Free plan

It's free for small projects.

Bee
User
testing &
surveys
Card
sorting
Session
recording
Tree
testing

You might be interested in