You are currently reading one of our 101s. This blog introduces you to the basics of Unmoderated Usability Testing. If you are looking for more complete resources, check out our Complete Guide to Usability Testing.
Unmoderated usability testing is definitely on the rise. The fact that it’s cheaper, faster, and more accessible than moderated or in-person research makes it popular. Nowadays, it is sometimes even the only option for UX teams to conduct user research. Let us take a look at what unmoderated usability testing is.
Table of contents
What is an Unmoderated Usability Study?
An unmoderated usability study is where respondents are not guided by a moderator. The test is prepared beforehand, and respondents complete it on their own. With an unmoderated usability study, all you have to do is to prepare the study and recruit people to complete it. The advantage of this study is that the number of participants is unlimited.
What is Unmoderated Usability Testing?
Unmoderated usability testing entails creating your unmoderated usability study – usually in an online tool – and distributing it to your participants, which can then complete the study in their own time, remotely. As participants complete the study on their own, no supervisor is needed. As a result, with unmoderated testing, you can collect much more information in a much shorter time at a lower cost.
Usability testing is the process of testing designs with users, used for collecting information about user experience – how difficult or easy the product is to use. Because the experience of the user is subjective and intangible, it relies on observing how users interact with the product as a way to learn about their experience.
According to the to NNgroup “User Experience encompasses all aspects of the end-users interaction with the company, its services, and its products.”
It is a great way to test and collect valuable feedback on websites or other products. It is frequently used throughout a product’s whole lifecycle and provides a lot of insights into a user’s behavior.
What is Website Testing?
Website Testing is a tool from the UX research platform UXtweak for conducting unmoderated usability tests that also integrates techniques of Session Recording. This gives you the ability to conduct user tests online and then replay user interactions like a video. Allowing you to pinpoint the exact place and time when users had issues completing their objectives.
What can you find out with Unmoderated Testing?
Unmoderated Testing can help you find answers to questions such as:
- Can people use the website/app for important use cases?
- How do users interact with the website/app? What are they thinking when using the website/app?
- How do users tackle their objectives?
- Is there something that keeps users from fulfilling their objectives?
- Do they take a completely different way than they should have or you expected? Or do they take the correct path but get lost and confused along the way?
Thanks to Unmoderated Testing, you will find out answers to these questions and discover any usability and user experience issues. Knowing how users interact and use your website, will allow you to optimize and improve it.
When should you use Unmoderated Testing?
The great advantage of Unmoderated Testing is that it is very flexible – meaning that can be used pretty much anytime during the development or design process. You can use Website Testing to test simple HTML prototypes as well as live websites.
All that you need to make it work is a working piece of HTML that’s hosted somewhere public. That way, Website Testing can link to your website, web app, or prototype via URL.
Usability testing using Unmoderated Testing is versatile and scalable. You can set up a simple Unmoderated Testing study with minimum effort in a few minutes, or you can run studies with tens of tasks that cover main use cases, or you can achieve that with just a single task. Tests can be conducted with only a few people, or you can collect results from hundreds of respondents. With Unmoderated Testing, you can create studies that suit your needs.
Remote Unmoderated Testing Best Practices
How to conduct remote Unmoderated Usability Testing
The process of conducting remote unmoderated usability testing can be boiled down to 7 important steps:
- Define study objectives
- Create a tasks list
- Select a usability testing tool
- Set up the study
- Run a pilot test
- Recruit the participants
- Analyze the results
Don’t forget to keep these remote unmoderated testing best practices in mind⬇️
1. Defining your study objectives
During the preparations for your Unmoderated Testing, you should consider asking yourself the following:
What are you going to test?
Obviously, you should start by defining the scope of your usability testing. There will be situations that call for testing all the main use cases of your website or app because you just launched a full redesign to find out what works and what doesn’t. At the same time, all you need might be more focused testing of a few scenarios that cover the new functionality.
Who should be the respondents?
There should not be any user testing without considering your users’ identity. If you already have a website or an app, you’re probably well aware of whom your target group is. You can use any of your existing personas, real customer data, or even data from surveys and other sources of feedback you have.
The more information about your users you have, the better. Apart from helping you with the obvious (to recruit respondents who represent your audience), it will also help you prepare the right use cases and formulate relatable tasks that represent the user’s natural way of thinking. So, for these reasons – keep your users in mind when you start planning usability testing.
2. How to write tasks for Unmoderated testing
At its core, the Unmoderated Testing study consists of several tasks that the respondents are asked to complete to successfully finish the study. By preparing your tasks carefully, you will increase the chance that your user study will provide good results. Well-defined tasks lead the respondents to try to complete them as naturally as they can.
When preparing keep in mind that you need to:
- Write tasks that cover your objectives
- Write realistic tasks
- Write actionable tasks
- Write tasks without giving away the answer
Learn how to write good usability testing questions in our guide. For those who are not sure about what tasks to assign to their testers and how to conduct a study, we recommend choosing a template for usability testing and following its lead.
3. Choose the right tool for your unmoderated testing
Choosing the right tool for unmoderated testing is crucial for obtaining reliable and accurate results. Here are some factors to consider:
- The type of tasks you want to include – Some tools are better suited for certain tasks than others.
- The user interface – The user interface of the tool should be intuitive and easy to use for both testers and participants.
- Reporting capabilities – The tools should have robust reporting capabilities, allowing for easy analysis and interpretation of the data collected.
Here is a table of the 5 best usability tools of 2023. If you want to dive deeper now, take a look at the top 17 usability testing tools.
Ease of Use
Quality of Support
✓ Free plan
Plus plan - $80/month Billed annually
✗ No free plan
Upon request, starting at $50K/year
✗ No free plan
Pro plan - $229/month Billed annually
✓ Free plan
Pro plan - $39/month Billed annually
✗ No free plan
Upon request, starting at $15K/year
4. Setting up the study
Setting up the study, once you choose the right usability testing tool, is a matter of inputting all your tasks into the tool.
Be careful to include all the relevant information. Tasks and questions are the core of the study but things such as a welcome message and instructions are important as well so that the participants know what to expect and how to proceed. You are not there in person to guide them after all.
As you have defined the target audience for the test, don’t forget to create a screener question. A screener will filter out the respondents that are not relevant to your study and only let in the ones that will provide valuable information.
5. Run a pilot test
Running a pilot test is a crucial step in unmoderated usability testing to ensure that the study is effective, before launching it to the full participant group. The pilot test involves recruiting a small group of participants to go through the study and provide feedback. This feedback can include anything from technical issues to typos or confusing instructions and unclear tasks.
Running a pilot test can help you identify and fix any issues in the study before it is launched to a larger group, ultimately leading to more accurate and reliable results.
6. Recruiting respondents for your test
Once your study setup and pilot test are completed and you want to launch the test, you need to recruit respondents. Recruiting respondents for an Unmoderated Testing study is easy. Simply share a link to the study with your respondents. You can use social media like Facebook, Twitter, and LinkedIn to get it to your target or send the link directly to your testers.
Alternatively, you can order respondents form a User Panel, based on your criteria. If you have a list of your own respondents from channels like this, you can also upload them to your Own Database within the research tool – a CRM-like tool to help you manage your participant database and send emails with invitations for user tests.
In UXtweak you can also convert your website visitors to test participants with Onsite Recruitment Widget.
Of course, there is more to recruiting respondents than sending out a link. It is important to prepare a recruitment strategy as a good one will make the difference between learning a lot of valuable information to improve your website to learning nothing at all – wasting a bunch of time and money. It is therefore very important to define who your target respondents are and include a screening question in the study to filter out respondents who do not fit your parameters.
Check out these tips for creating a perfect screener question.⬇️
Questions that should help you with recruiting respondents:
- Whom do I want to recruit?
Your respondent sample needs to be representative of the end-users so you can learn about actual users’ behavior. When using the Website Testing Tool by UXtweak this is very easy. You can make use of UXtweak Recruiting Widget which will allow you to recruit respondents directly from your website.
- How many people do I want to recruit?
According to NNGroup 5 respondents are the optimal number of respondents. With five respondents, you typically figure out most of the main problems in the tested scenarios. Although to get statistically significant results, we recommend at least 20 respondents. You can check out our guide, where we analyze several usability testing examples of world-famous brands and find out how many participants they involved.
- What information do I want to communicate to the respondents and how?
This will depend on who your respondents are and how you recruited them, but the golden rule is that the respondents should be always well-informed. A well-informed respondent has more value for you than someone who has no clue about what’s going on.
Information we recommend sharing with your respondents before testing:
- How long does it take – e.g. 20 minutes
- What is expected of them – e.g. completing tasks on the e-commerce website of a fashion company
- What is the point of the test – e.g. to make our website easier to use
- Why would users want to participate? – you should always keep in mind that respondents are not always willing to give you their time for free, especially if the test takes more than a few minutes. Therefore it is an excellent idea to offer respondents some incentive for participating in the study (could be monetary, a chance to win something if they participate, or a different benefit). Always make sure that they’re well informed about the reward and how they receive it.
7. Analyzing the results
After a successful study, you obtained a lot of data. But you still need to analyze it to make something useful out of it.
The overview gives you all the information you need to check the state of the study. The first thing in the summary is the dates when your study was first created when it was launched, along with the number of respondents who participated in it. Check the donut chart to see how many respondents completed the study and whether there were any quitters.
Time Taken is a number that tells you how much time the respondents spent doing the study. Checking whether the Time Taken matches the expected completion time is a good way to keep tabs on any unexpected issues so they can be addressed quickly.
The Success score is the percentage of tasks that the respondents completed by submitting the correct page as the answer.
The overview contains a breakdown of various attributes of your users such as:
- Top Locations
- Device Breakdown
- OS Breakdown
- Browser Breakdown
- Screen Resolution Breakdown
In the list of respondents, you’ll find information about all respondents who participated in your study. It’s here where you can replay the completion of tasks, filter your respondents, and select which ones should be included in the analysis. If you want to learn more, check out Interpreting Website Usability Testing in our Definitive Guide to Website Testing.
Pressing Play on the task will open up the session player, showing you exactly how the interaction between the tester and the website/app went down. You will see what the respondent was doing while they were trying to complete the task. You can control the session player like any video player. If you read our Session Recording guide, you know that even though it looks and feels like a video, it is a re-interpretation of how the respondent interacted with your website/app.
We also created a complete guide to Setting up a Website Testing study for you to better understand the whole process.
Conclusion: Optimise your website for the world today
Unmoderated usability testing is a simple way to get a lot of information about how your users behave, with relatively low effort and cost. The Website Testing tool gives you the option for recruiting the right respondents that fit your needs easily – by sending them a link or using the recruiter widget.
All results are analyzed after the end of the study, giving you detailed information about tasks, answers to questionnaires, respondents, etc. This helps you to understand the issues on your website and to fix them quickly, easily, and in a budget-friendly way. Try it yourself – register to UXtweak!