Get ready for some great content coming to your inbox from the team at UserTesting!
In this podcast episode, UserTesting’s Research Manager, Bryan Kern, and Lead Mobile Researcher, Lisa Preville, chat about destination testing and the capabilities of the new UserTesting Mobile Recorder. They discuss:
The difference between “in-the-wild” and “destination” testing
“Device” versus “beyond-the-device” testing
How UserTesting’s Mobile Recorder can be used to test real-world usability
Listen to the entire podcast here to learn more about building destination testing into your research plan.
Bryan Kern: Hi everybody. This is Bryan Kern, a research manager at UserTesting, and today on the podcast, we’re going to talk about in-the-wild and destination testing with Lisa Preville. Hope you enjoy. How are you, Lisa?
Lisa Preville: I’m really good. Thanks for having me here to talk about destination testing. It’s really cool. So it’s something new that we’re doing with our new mobile recorder. I’ve been totally entrenched in this type of testing, so I’m happy to talk about it.
Bryan Kern: Awesome. For some people who might not know, we just released an awesome new mobile recorder that you can now test things, not in the lab and not in someone’s home, so we broke the surface with saying, “Hey, we’re going to get out of the lab and we’re going to go into people’s homes and we’re going to show you people in their own home settings using their computers.” We’re taking it now a step further which is awesome.
Lisa Preville: Yeah, it’s so cool.
Bryan Kern: What is it? What is this next step that we’re taking?
Lisa Preville: The next step is that traditionally we had done mobile testing with an IPEVO camera or a web camera at our testers’ homes, kind of on their desk, wherever they set up their testing station, so to speak. But now we have this app that all our panelists and all our testers can download and they can actually record on the screen what they’re doing on their mobile phones or on their mobile tablets too.
Bryan Kern: That’s so cool.
Lisa Preville: Yeah, because now you don’t have to be at your desk. You don’t have to be tethered to your computer, and you can literally be on the couch taking a test. You can be out on the bus taking a test. We’re talking a little bit about that today. You can be pretty much anywhere and you can test where your users are just naturally using your product, but also you can tell them where to go if it’s specific that you want to see how they’re using your product in context with their regular life. It’s so cool.
Bryan Kern: Let’s get a little context around this. There’s a big issue in mobile testing, across all of usability testing. How do you get it in its natural environment? Like a desktop test or a laptop test, natural environment is easy to do. You sit them at the desk and they take a test.
Lisa Preville: So it’s pretty natural.
Bryan Kern: Whereas mobile, there’s a whole bunch of issues, like how you do it. What is someone’s natural environment with a mobile device? Can you give an example of a natural environment that someone might be in with a mobile device?
Lisa Preville: A good example might be that I was at Target last week and I was looking for a table, like a side table to put in my living room because it’s pretty bare right now. I saw this girl walk by me with her cart, and she had this side table that was so cool. I was like, “Where is that side table? I need it immediately.” But I didn’t ask her, because I hate approaching people in stores, so I went and found it on my own and it was just literally like a blank space in the shelf. She had taken the final, the last stock on the floor, of this really, really cool table, and I was so mad. I was like, “Okay, I don’t like talking to store employees. I’m going to do this myself.”
I pulled out my phone. I found the table on the website and then looked it up to see if it was in stock in my store. The website told me that 2 were in stock, and I can only imagine at that point that those were the 2 that she had in her cart. I was like, “No!” I tried to find this really cool table. I’ve been looking for weeks. I didn’t want to pay for Ikea furniture. I just wanted to find something easily after work and this girl had it in her cart. I was like, “Okay, well, let me see if I can try and figure out if I can get another employee to help me.”
I got an employee to help me, which I hate doing, but I managed to do it. He looked it up on his side, and then I looked it up on back order online, so I was using my phone in this context of being at Target. It wouldn’t have happened, me tethered at my computer. If I was at home, I probably would have just done it online. I wouldn’t have done it on my phone, but that’s kind of the magic of mobile. You’re doing things on mobile that you can’t do at home.
Bryan Kern: Yeah, so shopping is a big thing there. People don’t necessarily shop online with their mobile device, but they use it when they’re in the stores.
Lisa Preville: Totally. Yeah.
Bryan Kern: I think that’s the kicker, like there’s a lot of space that you can test. That’s where I guess this destination testing comes into play. I know we’re kicking around like “in-the-wild,” “destination.” What is the difference?
Lisa Preville: Yeah, totally. We were kind of coming up with this methodology and we were throwing around this term that we heard from prospective clients called “in-the-wild” testing. That is such a flashy, really cool name to describe what we’re doing, and it catches people’s attention. It’s really neat. It kind of invokes a more anthropological, ethnographic approach to testing. But what I discovered is, I was kind of looking at a lot of sites that offer in-the-wild testing, and what I realized is that what they mean by “in-the-wild” is actually “outside the lab,” so moving your usability research from a lab setting to an in-the-wild setting outside of the lab, obviously.
When I was thinking about how to phrase or how to define this methodology, that wasn’t really appropriate, because what UserTesting has done, since the beginning of the company, has been remote usability research, in the home, outside of the lab, anyway. Suddenly I was faced with this horrible conclusion that this really, really cool term doesn’t actually mean what people think it means. A lot of people think in-the-wild testing is more of this location-based, destination testing, and that it’s out, not really at home, but more in the stores or out on the street and stuff like that.
Bryan Kern: Not surrounded by 4 white walls.
Lisa Preville: Right, not surrounded by 4 white walls, with somebody looking over your shoulder in a lab setting.
Bryan Kern: Yeah, with his clipboard and all that.
Lisa Preville: Yeah, so we had to come up with this new term, and the reason why we came up with “destination testing” is kind of to invoke the fact that you would be leaving your home to go do a test. Whether it’s going to Target or going to Whole Foods, going to a specific store, even using apps that require you to be in a location, with something like MapMyRun. You’re not really using MapMyRun sitting at your desk. You’re using it out on the trail, wherever you’re running, or anything like that. We decided to come up with this term “destination testing,” because we felt like it was pretty flashy. It might not be as flashy as “in-the-wild testing,” but it invokes this need to leave your home to do a test and specify that someone should be out and about doing something on their mobile phone.
Bryan Kern: Okay. Does it just have to be on the mobile device, or is this like… Let’s say I wanted to go buy a toaster and I wanted to see the usability of a toaster.
Lisa Preville: Yeah, totally.
Bryan Kern: Are we able to do that?
Lisa Preville: Absolutely.
Bryan Kern: As like a destination test in my kitchen?
Lisa Preville: For that type of test, where you’re kind of looking at the experience outside of a mobile experience. There’s 2 types of testing. One is device testing. That’s where you want to see how somebody interacts with something on their phone, on their tablet, the actual interaction with the website or the app on the device. Then what we also have is something that we like to call “beyond the device testing.” It utilizes our mobile app to help record these interactions and these experiences, but it’s not totally tied to a mobile experience, like shopping for a toaster or …
Bryan Kern: Shopping for a toaster would be destination …
Lisa Preville: Right.
Bryan Kern: I would go to the store, buy the toaster, and then…
Lisa Preville: Bringing it home maybe …
Bryan Kern: Whoever the person is, can they actually use the toaster?
Lisa Preville: Exactly, yeah. So if you’re actually at home, looking to see how the toaster works, does it toast appropriately? Is it going to burn everything you put in it? How well does it work? Yeah, exactly. We can definitely do that kind of testing in the home, just beyond the device.
Bryan Kern: The whole package almost.
Lisa Preville: The whole package, yeah.
Bryan Kern: Let’s go buy it. Okay, great, you have it now. Okay, let’s not stop there. Let’s actually use it.
Lisa Preville: Yeah, exactly. What they would call that in the retail space specifically is called omni-channel testing, where you’re looking at multiple different pathways a user can take from the beginning of their path to purchase all the way to when they’re at home with the product and understanding how they use the product, how they feel about it. After using the product, what kind of brand trust, what level of trust do they have with the brand, how they’re acclimating to using it, how it fits into their daily lives. We can really record that whole interaction using the mobile recorder and that’s really cool.
Bryan Kern: Awesome. Getting back to destination testing, we were talking about taxi cabs, right?
Lisa Preville: Oh yeah.
Bryan Kern: It just blew my mind when I heard, like, “Wait, wait, wait, wait, wait.” You can do this with a taxi cab, and not even think, like different apps and stuff for hailing a taxi. Of course, you want to see the usability behind that. It’s not just going out in New York City, raising your hand out there anymore. You’re hailing a taxi with your finger to the screen now.
Lisa Preville: Yeah, you’re just tapping to hail a cab now. I think it’s really cool that we can now capture these experiences. They are so many mobile-enabled companies, so maybe these companies don’t… Their whole product isn’t an app, but their main service is supported by an app. Companies like Uber, for example, like taxi cabs or car services, they’re enabled by their app and their do or die is based on the app being successful, working properly, and things like that. We can test the whole experience behind those apps and how they relate to the service that they’re offering.
Bryan Kern: So cool.
Lisa Preville: It’s really neat. We’ve never been able to do this much exploratory, new methodology before, and it’s something that I know the research team is all really excited about. It’s been so cool to just develop and work on this methodology and see how far we can push the limits, essentially.
Bryan Kern: Yeah, yeah, that’s the other part that’s really cool is that there are people who want to help you do this. I think that’s the other really, really interesting part, is that it’s just not we created this thing and now it’s, “Okay, have fun trying to do it.” There’s actually people that are really interested in investing a lot of time and energy to help people do destination testing. That’s what I’m really eager to see.
Lisa Preville: Yeah, I’m hoping that other … At UserTesting, we have our research team, but I’m hoping that other research teams at other companies will get excited about doing this type of study, getting this kind of feedback, and incorporating it into their research process, because I feel like there’s so much strict usability where it’s just interactions with the device or the app or the site, and you almost tend to lose context when you do that type of testing. You can get so funneled into which button works, which color’s receptive [00:10:00], which feature or which interaction is working the best, that you’re kind of missing the point in that you’re not really sure or you’re not seeing a whole lot of the time how someone fits that product into their normal life or how they’re actually using it.
Bryan Kern: I guess, going real psychological here, the gestalt of the holistic approach, that’s what this kind of encapsulates and captures as well.
Lisa Preville: Totally.
Bryan Kern: Not the nitty-gritty, but it can get the nitty-gritty, but it’s more of a holistic approach to how people just use it in their everyday life.
Lisa Preville: Yeah, absolutely. I feel like so many companies and products can benefit from looking at things holistically, like you were just saying. The whole picture, understanding the whole process of adopting to an app or a product or a service, and how it really impacts your users, more than just kind of, “Okay, how does a user interact with this particular feature?” You’re thinking, “This feature makes this user’s life easier, because it gets them to A and B faster.”
Bryan Kern: The “why.”
Lisa Preville: Yeah, the “why,” exactly.
Bryan Kern: But really seeing the “why,” not just hearing the “why.” Now, it’s like they’re in the environment, that this is why they are here.
Lisa Preville: Yeah, exactly. That’s what’s so great, because we get good feedback from testers and things like that, but to actually see them interacting with struggles at a store or in real life context is something that’s just like… It’s so much more impactful and meaningful for our clients or just anyone who’s interested in doing this type of testing. It just offers another layer of research, of benefit.
Bryan Kern: I guess, going on the last point you just said, why should my company, let’s say I have a company, why should my company be doing more in-the-wild, or more destination testing?
Lisa Preville: That’s a great question, and I feel like we’ve had a lot of those questions come fielded to us, like, “Okay, this is something new, but why should I start changing up my research plan, just because you’ve got this new capability?” The short answer is just that the data you get about your users and about your product is so much richer with destination testing. You’re getting a level of insight and feedback that just isn’t possible if you’re tethered to your computer or just doing this kind of more restrictive testing. That is useful in its own way for strict interaction testing and usability.
That’s great, but what you should be concerned with, especially mobile developers, is how is your app received? How do people use your app and accommodate for it and make time for it? That’s kind of the big question that I feel like a lot of people are asking, like, “How does my app fit in with someone’s life? How can I get them to stay engaged?” Understanding where that engagement may or may not fall off, that’s where destination testing is going to come in to help you understand that process with your users.
Bryan Kern: That’s an interesting point. There’s so much worry about, in the app culture.
Lisa Preville: Like submitting to the App Store, yeah.
Bryan Kern: Right, like that, “I have to get a 5 star.” It’s okay, but this is how you get the 5 star. This is why you are getting a 5 star.
Lisa Preville: Totally.
Bryan Kern: They’re in their environment. This is what they’re doing. That’s really cool.
Lisa Preville: Yeah, and I feel like a lot of the results will probably surprise you. You think you know how a user is using your app or your site. I don’t know if Target thinks that I would look at their site to check stock levels if I’m actually in store. I’m sure somebody at Target would have assumed I would have gone to an employee, but I didn’t. I actually did everything I could to avoid doing that. It’s so interesting to see what the discrepancies are and it’s just such a great point to learn more about your users and discover how they really act, as opposed to a one-time test, tethered to a camera, where they give their best feedback that they can, but it’s so different if they’re actually at a store and in a normal, real context to react to challenges or just using the app in general.
Bryan Kern: Cool. I guess, do you have any other thoughts, comments, or anything, I guess, you just want to get off your chest about how to use it, what people should be doing with it once they have it?
Lisa Preville: Yeah. A lot of what I’ve been doing is talking to researchers about how they can use the product, and I’m hoping to make it easier for people to claim this type of research and make it their own, but it is a learning process. It’s kind of a new thing that we’re doing and we’re constantly trying to improve the process and make it easier and make it more efficient and things like that, but it’s fun for me to just think about the different scenarios that you can test with this type of methodology. It really excites me to think like, “Oh wow, I can send testers to a store. I can send them to…” Not even things that are really tech-related, just like understanding how families organize their homes.
I’ve been reading this book recently called The Art of Tidying Up. It’s by a Japanese author and she talks about how, in order to clean up your space and live peacefully, you need to do one entire, huge purge of all the items in your life that you don’t care about, which is one, exciting and kind of cool to read about and just very inspirational, but that’s something that I was thinking I know so many people that kind of struggle with organizing their homes and organizing their family schedule.
The type of testing we can do can also help with product development. Maybe you want to create a product that will help somebody organize their home better or help them schedule their family time or help them manage their personal family life, but how can you develop a product unless you know where the blind spots are? With this kind of beyond the device testing and using the camera to record your experience at home, you can get a really fine-tuned view of how people live their lives, and then try to accommodate for those issues they face and how you can develop a product that’ll help them do what they need to do at home. It’s kind of a cool opportunity for companies to not only help their users, but develop a whole new product or a new service that can help with an issue maybe they never even knew about.
Bryan Kern: It just keeps hitting me that the limits of this are almost limitless.
Lisa Preville: Limitless.
Bryan Kern: Yeah. I’m not Bradley Cooper, but just like what you could actually test. You could bring it into cars and do destination testing, but moving destinations.
Lisa Preville: Oh, man.
Bryan Kern: Right?
Lisa Preville: That’s super cool.
Bryan Kern: Yeah.
Lisa Preville: We did this “MIX” event here at UserTesting where the company got together. It was kind of like a hackathon, and everyone was assigned to different teams and different projects. I worked on a project that was developing… the backing of the idea that we can start testing more car interfaces. Even testing products, it might not be in the store, it might not be in their home, but you can test the largest product that a person buys, like a technical product, is a car.
Bryan Kern: That’s not their house, right?
Lisa Preville: Exactly. It’s a huge purchase. People spend time. I recently bought a car like 5 months ago and it was the biggest purchase I’ve ever made in my life. It was so intimidating. Knowing how to navigate that process, there’s this whole subset of people that they need to do research, they need to understand how… I don’t want to say how the car works, because that sounds like, okay, cars are pretty simple and the driving mechanics are…
Bryan Kern: But like every little feature.
Lisa Preville: Yeah, the features and everything’s coming out with a new infotainment system. I know we were looking at some cars just in the UserTesting parking lot. There was stuff that I had no idea how to use at all. I felt like an idiot, because I was like, “This doesn’t make any sense to me.” I was like, “I’m glad this isn’t my car, because my car is just the basic.”
Bryan Kern: It’s been the little things, like I think Ethan showed them trying to unlock the car door. I’m just like, “That’s such a simple thing.” You can touch a button and it should unlock your car, or maybe it was like fingerprint or something like that. He was pushing around. He’s like, “I think this works,” and then he’d lightly touched it. He was like, “There you go.”
Lisa Preville: Yeah, I tried doing that myself. Basically it’s on the handle. It’s a button on the handle that, if you press it with your thumb or a finger, it locks the car for you. You don’t have to insert the key anywhere. But if you have too many fingers on the handle, so if you’re clutching it like you would open it, it doesn’t do anything. It doesn’t actually lock. You have to remove your hand and then tap the button lightly, and then that’s how it locks.
Bryan Kern: Just little things like that, they’re like… It would happen to me twice, and I would be like, “I’m just going to lock the car. I’m going to use the key fob, boop boop, and we’re out.”
Lisa Preville: Yeah, exactly.
Bryan Kern: It’s a cool concept, but is it usable?
Lisa Preville: Yeah, totally, and it’s like these little blind spots, I guess is what I’ll call them, you just have no idea that users deal with these tiny issues, but if you have enough tiny issues, then it’s hurting you.
Bryan Kern: Snowball, right? Snowball effect, right?
Lisa Preville: Yeah, totally.
Bryan Kern: 4 snowflakes doesn’t equal a snowball, but put a million together and you’ve got a snowball, and it’s probably blowing out of control. Cars, I had a crazy idea. I was saying space.
Lisa Preville: Space.
Bryan Kern: You can bring this to space. I’m crazy. That’s where I thought, kind of limitless.
Lisa Preville: It’s limitless, totally.
Bryan Kern: Let’s interstellar it.
Lisa Preville: Yeah.
Bryan Kern: The destination doesn’t have to be something that’s set in stone, right? It can be something that moves. I think that’s pretty cool. Cars is a great example with that. I guess, anything else before we wrap it up, Lisa?
Lisa Preville: No, I feel like we covered a lot today. It’s been fun talking about destination testing. It’s really cool stuff.
Bryan Kern: Yeah. I want to thank Lisa for taking time out of her day to come talk with us.
Lisa Preville: Thanks.
Bryan Kern: Thanks for listening!
Join 120,000 subscribers and get articles like this every month.
Get ready for some great content coming to your inbox from the team at UserTesting!