this is the Nielsen Norman group ux podcast [Music] I'm your host Therese fessenden on this monthly show you'll hear interviews with user experience thought leaders about fundamental ux topics common industry questions and discussions on how great ux can truly transform the world today I'm joined by Kate Moran senior ux specialist with nng Kate's research spans a breath of topics and she's a Pioneer in the industry for her research insights from discovering new browsing patterns with eye tracking research page parking tone of voice dimensions and many other groundbreaking Concepts she holds a master's degree in information science from UNC Chapel Hill and teaches Six certification classes on a number of different topics including the topic of today's podcast return on investment otherwise known as Roi on the show Kate shares her insights from her latest research on benchmarking and Roi what to think about when communicating the business value of ux what to avoid when making metrics-based decisions and how to ensure you're not letting the Tail Wag the Dog so to speak by letting data make your decisions for your team rather than having your team make decisions with the data with that I'm excited to welcome Kate Moran [Music] thank you for joining us how are you doing today what has been exciting what have you been working on recently oh I'm pretty good you know like like a lot of people been hunkered down but luckily work has been busy so it keeps me busy um yeah I will say also I happen to have a English bulldog puppy sleeping next to me and she is a big snorer so you'll have to let me know if if that gets too loud um we know is my favorite I will I will let you know don't worry um so yeah you know I I feel like like I mean you know you know how it is like with nng we've always got so many things that we're juggling at once so um but I've been really focused on reports like in general this year that's been kind of a focus been doing a lot of like research and a lot of report writing and so that's something I'm focused on right now yeah so you've recently published one of them if I remember correctly is that the ROI report is that right yeah it's called ux metrics and Roi um and essentially it is a big collection of case studies that describe how teams are using quantitative data in their day-to-day work so it's 44 case studies and the really fun thing about that report is we've been updating it since 2006. so we've got Legacy data going back to 2006 for how teams were improving their products and and quantifying that and collecting these ux metrics that's really cool and I know for some folks hearing Roi sounds like a foreign language so I would like to benefit for the benefit of others listening how would you define what Roi is exactly yeah so the the report is kind of about two different but related things and so the first kind of aspect of the report is about ux benchmarking so that is just collecting you know some sort of metric that represents the Improvement that you're making in the design hopefully an improvement usually an improvement but hopefully so things like you know [Music] like how efficient people are with the product or it could be something that you get from analytics like your conversions or it could be returning visitors just something that is a numerical representation of some aspect of the experience so that's kind of the first part of the report is we we break down how that process works collecting those metrics analyzing them um and we look at a lot of different case studies where teams were doing that so then the second sort of part of the report is Roi so return on investment so that's sort of taking those metrics and showing how those metrics connect to business goals as expressed as kpis or key performance indicators if that's uh not jargony enough for you that's perfect yeah key performance indicators is I think some folks might be a little bit more versed in those in that it tends to impact a lot of people's jobs like yeah certainly plays into a lot of people's you know compensation or not just compensation but like how well or not well the entire team is doing yeah well that's that's what kpis are they're from a business perspective it's ways that you can sort of check in and see if the the company or the organization is meeting its goals um and usually those are tied to revenue in some way so a lot of times we're talking about like profit or you know cost savings um and as you say it is often tied to how people are evaluated and like whether or not they're going to get bonuses or whether they're gonna get a new hire that they need things like that yeah it could be pretty stressful I imagine and and for some of the folks I imagine that you researched uh Roi must have been either something that went really well with the organization or stuff that stress people out so I could you give people any advice if they were you know trying to figure out how to quantify their impact not to lose their mind while doing it um well um in that report we we have some step-by-step instructions for doing that but essentially the thing that I really like about looking at ux metrics and collecting Roi is that it requires you to think really deeply about what is the value of ux and then sort of Express that through quantitative data um so you have to understand you know what we in most cases we're not doing ux work because it's nice right we might you know want to believe that that's why we're doing it but in most cases we're doing it because it makes sense because it makes business sense and in many cases makes Financial sense the problem is that a lot of teams don't do the work of actually showing that they don't actually do the calculations so um for example McKinsey about a year or so ago they did a big survey of different design teams and they found that over 50 percent of the design teams that they surveyed weren't doing that they weren't quantifying their impact and so as a result they had no way of showing how they were impacting the business so the first thing to do is to think you know what what is the purpose of this product that you're working on or like why does this why does this product exist what what value is it offering for people um you know for like let's imagine like um you know an intranet so why does an intranet exist to help employees find information that they need to make them efficient to help them get their work done and kind of support them so okay so that's the value of an intranet so like what are the things that we could measure that would tell us whether or not it's doing its job probably a lot of efficiency things are going to be important so things like time on task and also you know making sure people can find the information that they want and that it is the correct information you know things like that so there are ux metrics that we can collect that describe those things and help us assess how well is this design working now and then once you start doing this over time then you can look back and see okay how much how much progress have we made how much better have we gotten now kind of like looking back so that's kind of the first place to start um and then with the ROI calculations it's really about figuring out okay I have have these things these like aspects of the experience that I've measured that's on one side and on the other side I have uh Roi metrics or kpis things that the business values so to sort of oversimplify it a lot about making that conversion kind of almost I like to explain it to people like converting units um so taking the the Improvement in the design and showing how that is also you know something that contributes to cost savings you know that's probably what we would look for with something like an intranet we would want to see hey look we made people more efficient we helped them do their jobs more efficiently how much value is that really adding to the business got it that makes a lot of sense and and I'm glad that you're talking about it like converting units because I think people get very stressed about it and worry that if I don't justify this in just the perfect equation or if I don't get the value absolutely correct that you know this may not be taken seriously um but I guess what are your thoughts on that are there things people should keep in mind or mistakes that people make when they calculate yeah I actually or communicate Roi I guess right oh yeah well I actually wrote an article on that on those um mistakes that people often make when they're doing these calculations I think part of the problem is that we call it Roi to be honest like if we could Rebrand it and call it something different there'd probably be fewer confusion there would be less confusion but um I think the problem is that people hear Roi and they think okay this needs to be some super you know super accurate super specific Financial calculation and the problem is that it will just never be that it can't be that we're trying to predict you know what human beings are going to do um and you know like just the way that we do these calculations it assumes that everything else in the world is going to stay constant and we just don't know that that's the case like you could make huge improvements to an e-commerce site and calculate that you know it's going to translate to two million dollars in added Revenue every year but then a huge competitor comes out of nowhere you know like Amazon decides to get involved in your product space whatever it is and all of a sudden like you know that revenue is not there so it we have to understand that it's it's not a prediction it's really a strategic exercise like sometimes I tell people to think about it like a thought exercise it's really just trying to communicate what is the potential monetary value and sometimes it's monetary sometimes it's not like sometimes you can just Express something as like time saved and that can be powerful I think people need to remember we're not doing these Roi calculations because they're fun in most cases um we're usually doing them you know maybe to communicate to our stakeholders or our clients that we you know there's something we need to do there's a redesign that needs to happen there's you know some change we need to make and it's going to be valuable or we're trying to you know maybe after the fact we're trying to show those people this is how much value we delivered for you or if you're in a leadership position you might do these kinds of calculations to help you prioritize and say look I've got these six different ux projects I could work on we're going to pick this one because it's going to be potentially the most value so we're always doing Roi calculations for a reason and so you should do as much work in that calculation as you need to right does that make sense like you should do just enough to help accomplish your goal but don't do more than that got it okay I like that idea partially because sometimes the thought of doing massive Financial like predictions sounds terrifying but yeah you're right it's a thought exercise it's meant for us to strategize it's meant to be a tool for us and this leads me to another common theme I see in a lot of our conferences and classes that a lot of businesses kind of for lack of a better phrase fetishized numbers when they strategize they often obsess over these increased outcomes or increase performance at all costs and sometimes I think about that especially nowadays you know you you see a lot of patterns that are somewhat dark patterns or nefarious design choices for micro conversions or just to get that additional 0. 5 percent increase in in clicks so you know I guess another question I would have is you know do you have any advice for how teams can stay growth oriented and avoid unethical behavior that might be incentivized by looking at these numbers or you know should we prioritize some metrics or should we prioritize others what do you think of this very big complicated question um well I definitely know what you're talking about I've definitely seen that like in the teams that I've helped um and also just kind of the culture around metrics in ux can sometimes be toxic unfortunately um I think part of the problem is that when we start collecting these metrics we can like the power can kind of go to our heads a little bit we can get a little excited about all the data that we can find especially with tools like analytics where it's so easy to just pick out all this information about what people are doing with your product in real life I think the problem comes in when teams let the metrics make their decisions for them versus just looking at the metrics as as a tool um and I think they're I think there's a distinction here between using quantitative data to help inform design versus letting it lead design sometimes I think this can be like a cultural problem in organizations it certainly seems to be something that's true of people in leadership positions they tend to care a lot about the data care a lot about quantitative data and you know this is a sweeping generalization of course there's exceptions but they tend to be you know more impressed by the quantitative stuff that's part of why I recommend ux teams get involved with benchmarking and with calculating Roi so they can sort of speak the language of the business but but the problem really comes in when as I said teams are letting the data make the decisions and also I think when there's maybe too much emphasis on a single metric to the extent that we sort of lose sight of the bigger picture so I I have two examples for you first is a a non-ux example and then I'll give you a ux example so the non-ux example if you remember a couple years ago gosh I don't know was it five years ago getting so old five ten years ago something like that anyway so you might have heard about the Wells Fargo Scandal that happened do you remember that being on the news somewhat but if you could refresh my memory since I'm also getting old you know fantastic yeah so Wells Fargo which for anybody who's listening who's not an American this is an American bank and they had a huge Scandal a few years ago um where it came out that some of the account managers were creating accounts for people without their permission so they were creating these bank accounts for their customers without their knowledge and the problem was that these people were being charged fees for those accounts and also it's a violation of their their privacy and security so it was a huge Scandal um and I remember reading about it at the time that the problem ended up being that the company was so fixated on one number on new accounts opened I think it was like monthly new accounts was their metric that they used and they stressed it so much to the employees and they would fire people if they didn't meet those those targets so these the I know it was wrong of them but these account managers were saying all right I'm gonna have to either I get fired or I create these illegal accounts so some of them were creating the accounts so um that's just a really dramatic example I think of this this um concept this phrase that you know it gets uh thrown around a lot which is what gets measured gets managed so what we take the time to measure and to talk about and to use as a one of those kpis that does have consequences on the real life decisions that people will make um so that's a non-ux example but it's definitely true in the ux world as well so let's think about a digital example um there was a trend that came out now this was a couple years ago I feel like it was like two years ago where we started to see these things that we call manipulinx yes which are happening yes I don't recommend manipulating so I'm saying yes because I just remembered Kate's video yes recorded this fantastic video just that's all I can say just look up manipulinx and in group you'll well thank you we had fun with it for sure at least um so manipulinx are these are named for these little pop-ups that appear on usually on e-commerce sites sometimes on content sites um but they're these little pop-ups that are basically trying to bully users into signing up for their newsletter and part of the way that they do this is they'll have an opt-out link underneath that newsletter sign up that says something like um no thanks I don't like fun no thanks I don't like deals so the the idea there is they're sort of putting words into the mouth of the user and like making them say something negative about themselves to sort of punish them for not wanting to give their email address over to this company now obviously from a ux perspective where we care about long-term relationships with our customers that's not ideal it's definitely not so the question is how do these teams do this and how are there so many of them like now if you have anywhere if you hadn't noticed this before you're probably going to start to see them all over the place I still see them constantly I'm always finding new ones I think what's happening is I think these teams do see an increase in the number of people signing up for those newsletters when they use these kind of approaches and some of some of the ones I've seen are really bad they will you know not even make it clear that someone's signing up for a newsletter they'll say something like put your email in and you'll get 12 you know recipes you know for cooking over the weekend or whatever it is so they'll give you a little plug like that and they don't make it clear that yeah you're going to get those 12 recipes you're also then going to be subscribed to our all of our newsletters forever right so I think there's there's in some cases there's multiple things at play but I think teams are seeing that those approaches you know they get results they get you know 12 increase in newsletter signups and and if your boss is telling you you have to get newsletter signups at all costs then that might become your priority now the problem there obviously is you're sacrificing your long-term relationship with your customer for a very short term gain and you're sort of bullying them into something they may or may not actually want or need so I think there's a short-sightedness there I think there's a lack of understanding the context there and I think that's the danger with these metrics is they can't be the the be-all end-all um I couldn't have said it better myself um looking forward into another Ballad of lockdown with the pandemic slash the next fiscal year Well calendar year I guess you know lower risk tolerance is probably going to impact a lot of ux decision making I think everyone's kind of trying to prepare for worst case scenarios and it might get harder to innovate in that you know in this next coming year teams might air more on the side of safety or you may be keeping existing designs as opposed to trying to radically redesign or change things so you know if if you have any advice what would it be to help teams perform well whether it's to meet those metrics or change their metrics if if that's applicable or you know what do you recommend to teams as we look forward to the next year well um that's definitely something that I'm pretty anxious about and I'm sure most other people are too you know nobody knows what's what's coming in 2021 the vaccine news sounds really promising but and you know this is like an economic situation that nobody has seen before so um it is a definitely a concern that especially depending on the industry that people are whatever industry people are in some Industries are going to be hit harder by this than others um and I am concerned that some companies will start looking for places to cut their budgets they'll start looking for places where they can you know reduce overhead and get some savings and I'm concerned for those teams who haven't done this work yet those teams that haven't you know whether or not they're doing this in a quantitative way or not if they haven't been making the argument here's what ux delivers here's why we're not a nice to have thing here's why we're an essential thing I worry for those teams because I think that is where we might see um a little bit of of regression so um one of the things that I'm working on in addition to these reports is um I don't know how much I can say about this it's kind of it's kind of top secret but it is related to um our ux maturity work that we've done in the past so we love to study you know how teams how companies evolve and become more mature in terms of their approach to ux and their approach to their relationship with their customers um so that's one thing that that we're working on right now something upcoming related to that but so it's very top of mind for me and top secret um but it is true that unfortunately we we don't see that teams always get more mature over time that that companies always get more mature like that's what we like to think about we like to imagine that you know we're always working towards this end goal of someday every company will be super ux mature and they'll understand that it's a necessity and value um and valuable but um it doesn't always work that way sometimes there is regression we see teams and companies that you know were more mature and then for various reasons there's there's some backsliding and that is one thing that I'm really anxious about in the coming year is I I hope we won't see a lot of companies falling back in that way but I think a important tool that can help people to fight that is trying to prove your value trying to show the company that you know you're worth keeping around this is not something that all all teams have to worry about it's um really tends to be more of a tool that you need if you are a little bit lower in that ux maturity if you feel like there is an already an established understanding of the value of ux but just last week I was I was helping this woman who works for uh the United Kingdom's government she works on one of their digital products and she was like you know I'm really struggling with figuring out how to prove like the value of ux and I'm struggling to do this calculation so I ended up like talking to her for a while and sort of probing a little bit deeper and we ended up realizing she didn't need to do the calculation because you know their government already has a really high ux maturity like they already see the value so this is not something that you always have to do it's not something that everyone has to do or that you have to do forever it's just you know one of those things in your toolkit for making sure that you know people understand the value of ux yes that is that's some really nice advice as well and and yeah remember that you you're really doing this for your team and for the company you work for and if you have that value known then embrace it and keep chugging along keep doing the good the good work of ux so yeah definitely yeah well Kate if anyone wants to follow you can you point them to any social media or any related places to follow your work yeah um so you can always follow me on Twitter I'm at um my name Kate Moran with two underscores in the middle um sometimes it confuses people uh in retrospect I probably should have just picked one underscore but that's Kate underscore underscore Moran um of course you can always visit nangroup.
com to check out our reports and our articles and and all the videos that we mentioned today awesome thank you Kate hope you have a great rest of your day and I hope we get to talk to you again soon yeah thanks Terese you too bye bye foreign [Music] for any of the resources we talked about today check out the show notes for links and the report that Kate's been working so hard on you can also subscribe to our weekly email newsletter where we share free articles and videos and upcoming opportunities to get a ux certification at one of our now virtual ux conferences to learn more go to nngroup. com that's nngroup.
