Welcome to the third and final blog based on the results of our UX budget survey. This series started with us looking into the problem of convincing stakeholders about the value of the UX research. For the sequel we created an overview of the general motivation behind doing UX research. Now for the final topic we wanted to analyze one of the most important questions which you need to answer when you are planning your research: Online or In-lab?
We asked our survey participants (all of them UX specialists) how they conduct their research studies.
They could choose multiple from the following four options:
- In a lab
- Guerilla (intercept) tests
- Online
- Other
We expected that the “Online” would be selected by a majority (mainly because of their cost-effectiveness), however we didn’t expect that it would steamroll the “Lab” option in such a convincing way. 86% of the participants selected the “Online” option, while only 22% selected the “In a lab” option.
Since the participants could choose multiple options, we also looked into the percentages when all the selected options were summed together. 58,3% of all selected options were Online, while only 15,6% were in a lab. Why was the difference so significant? Is the lab approach outdated or does it still have something to offer? We discuss that and more in this article.
The influence of Covid-19
We ran this survey less than a year after the global pandemic of Covid-19 finally started to regress and life started to slowly return to normal.
The world may have started its recovery from the worst of the pandemic, but its footprint changed the UX field for good. For a long time it was difficult – to outright impossible – to conduct traditional in-lab research to full extent due to the strict pandemic restrictions.
This caused even the biggest opponents of online tests to be forced to use this approach, because there was literally no other option available. Resistive to online research or not, many researchers were most likely forced to increase the amount of online tests they conducted and it seems like the habit stayed.
What are pros of online user testing?
Online tests have multiple significant advantages over traditional in-lab tests.
1. Budget-friendly
First of all they are more budget-friendly. As we discussed in our previous blog about, convincing stakeholders to part with money for your research can be a steep hill to climb.
The online tests allow you to save up both on the incentive (you don’t need to compensate your participants for their journey to your lab) as well as the lab equipment. A well designed and furnished UX lab can cost a pretty penny.
2. Access to larger pool of participants
The next significant advantage is the availability of the participants. With the recruiting for lab tests you cannot reasonably expect your participants to travel for more than an hour each way. You can try recruiting participants from greater distances, however you will need to compensate them well to convince them their trip would be worth it.
With the online tests you can recruit participants from the whole world with a significantly lower strain on your budget. Recruiting participants for your Paris based company is easier from Argentina if it is an online test than it would be recruiting participants from Lyon if it were for an in-lab test.
The geographical coverage is the first advantage that comes to mind, but even if you don’t care about a specific region, online tests allow you to recruit your niche target group from a much bigger pool of potential candidates.
3. Efficient recruiting
Another key point is recruiting efficiency. If you are using an online recruiting panels such as the User Panel from UXtweak, all you need to do is select your targeting requirements, create your study and place the order. No need to contact the participants and try to juggle the available slots in the lab to fit all the participants.
With panels like this you usually don’t have to care about handling the incentive to your participants either, since it is included as part of the service. All panels that are worth your time do this automatically. Even if you are using your own participants, distribution of the study and management of time slots are much easier if it is done online. Using your tool’s built-in participant management makes this even easier.
4. Remote friendly
Online research is remote friendly. The home office and off-site meetings in general are more and more popular. If your participants are not bound to your lab neither are you. You can set up, conduct and analyze your studies wherever you are. This also means that outsourcing your UX research is not limited by the location either. For online research, you can just hire the best team option for you, regardless of their location.
5. Easier collaboration
Sharing and cooperation are much easier with online research as well. If the setup, the study and the results are all online, you can cooperate with your other team members effortlessly. No need to manually log all the results to shared docs or sheets, which is not only time consuming but also rather clunky.
If you are using an all around UX research platforms like UXtweak, everything is stored together and analyzed automatically.
UX Labs have still something to offer
After reading all the pros of the online testing listed above, it might appear at first glance that labs are redundant and worse in every aspect. Nothing is further from the truth.
1. Body language and non-verbal cues
First of all, no matter how advanced the online platform you are using is, you as a researcher will never get the same experience as when you are sitting directly in front of the participant.
Of course there are tests in which this factor is not as important. But in cases when you are testing scenarios crucial for your product, the in-person tests allow you to read the body language and other non-verbal cues of the participants better and learn additional information about their experience with your tested product.
The conductor is also able to get a better look at how the participants are behaving non-verbally while answering questions. These non-verbal cues can be invisible or hard to spot even with a high quality recording from the online test.
2. Eye tracking
Up to this point, online user testing platforms have not been able to cater to all techniques and methods that are available in labs. One of these techniques is eye tracking – a very advanced technique that provides you with deep insights on where the participant is focusing their gaze during the testing.
The analysis of fixations, saccades and gaze plots will tell you the exact story of the participant’s visual attention patterns. The equipment required for precise eye tracking has not been replaced by a software alternative, which would provide the comparable level of accuracy.
Other than eye tracking, mobile testing in the lab can provide more information compared to the online version. The standard testing sled is used to record not only the screen of the mobile device, but also the hand of the participant. Therefore this recording provides you with information about the hesitations and hovers of the participant’s hand. This data can’t be collected outside of a lab with a special camera setup, hence why recordings from online tools capture the screen and voice, but no hand.
3. The presence of the moderator
The in-person presence of the study conductor allows them to better react to the possible setbacks that the participants may arrive at during the course of the study.
If technical problems are encountered, the study conductor can directly help resolve them much more easily. The effect of the presence of the conductor on the comfort of the participant can be debated. Some participants will prefer that the conductor is present, because it gives them the opportunity to ask for help. Also thinking aloud is much easier for the participant when the participants are explaining their thought process to a person, not to a screen.
However, it is important to note that the presence of the conductor can be detrimental in cases when the participant is socially anxious. Social pressure may lead to fewer interactions or less honest interactions when compared to the online setting, where it may be easier for this type of a participant to provide an honest opinion.
4. In-person experience
The honesty factor plays a key role in UX research.
It is likely that the participants will provide more honest answers while taking part in online research.
This may be caused by the fact that they are usually in a familiar space for the course of the study, therefore they feel safe and can be honest without a risk. The participants are also likely to feel more detached from the product, since the researcher (who is on the same side as the owner of the product) is not directly present.
What is important, however, is the fact that with the in-lab tests, the conductor is more likely to notice when the participant is not providing an honest answer. The conductor can always encourage the participant not to worry and to express their opinion freely. Encouragement like this is much stronger when given in-person compared to the online tests.
Closing thoughts
So why exactly are online tests so much more popular compared to the in lab research? We could point at any difference, but in the end it’s mainly because they are not only cheaper, but also cost-effective. It’s not only the price you pay for the participants, but all the reduced management and higher efficiency too result in lowering the overall cost.
Despite the fact that lab tests offer much deeper insights it seems to be not worth it for just any run-of-the-mill research, especially when we take the difference in the price into consideration. The online tests are simply good enough for a much lower price.
Move your UX research online with a robust all-in-one testing platform! Register for your UXtweak account and start testing today!
