Gojko Adzic
I think it's this 4 steps. And then for people to remember it easy. The initial letters, that kind of a lizard out of this lizard and it's kind of learn about how people are misusing a product that is zero and one behavior change or whatever you want to improve, then remove obstacles to user success and then double check that no unintended or second order effects happened. And there's lots and lots of ways you can discover how people are misusing your product. For example, for ...., I have, lots of monitoring around what people are doing with the site and especially when they get into an error condition. That's really interesting. For example, we have a dialog where people can upload different types of documents, and then the product imports it, and occasionally people upload crazy things. I mean, we have, you might see on a text to speech conversion screen, people keep uploading Android package files. Makes no sense at all. I can't explain it. My best explanation is that it's one of these hacking techniques where you leave a USB outside of an office and you hope that somebody is going to love it. But some people upload things that kind of make sense. We just don't support that. And we had a bunch of people uploading a subtitle files from videos, and this is like, okay, if I can kind of see the logic in it, it was a fluke, really rare edge case, but I can see the logic in it. You can actually upload the subtitle file. You want to do text to speech conversion. And you don't think there was enough of these people that, let's implement it. Let's support that. Let's see what happens. And there was kind of a direct result of us capturing these workflow errors. So you upload the not supported file, you get an error saying this file is not supported. But I also get some monitoring and it get some aggregate statistics what people are uploading so we can track what crazy things people are doing. And then what we kind of I built this kind of subtitle conversion to audio with timestamps and making sure that's aligned that turned out to be one of the most profitable things ever in the product because people from large e-learning departments, in enterprise organizations, they do these training videos all the time. They need to translate into different languages. You need to kind of you want to have a voiceover talk in a different language. And even with the text to speech engine, that kind of speeds up how quickly you generate the audio. You're still left with hours and hours and hours of editing the video to align everything nicely and make sure it's done. But if you upload the subtitle file, you get back a synchronized version. It saves you hours of time and not a lot of people needed that. But those of our users that needed it use it for quite a lot. And a couple of our biggest kind of spending customers, came to the product because we had this kind of subtitle to do audio conversion that nobody else had in time. And again, that was a direct result of discovering what crazy people were doing on the site. And I think that there's a balance between research with your users and what I think is common sense now in the product communities that you have to do a lot of research with your users. You can't represent users, but there's a balance of what you can do with research upfront, and at some point you need to stop and build the product. And whether you involve the right people in the research, whether the audience changes, whether the audience preference change over time becomes a really interesting thing. Because I think when I originally built the product, subtitle importing wasn't really kind of that nobody needed it. But then it turned out to kind of have a simpler text to speech function, a video function than people tried to do lots of these text speech things from weird formats. And somehow we came to this subtitling import. I just focused on the initial release, and we would have kind of completely mixed it because we were not researching that. We were researching a different audience. And I think of this optimization where we look at unexpected things that people are doing. These are products, how people are hacking the products, how people are misusing the product. It can complement this initial research and can point to our blind spots, you can point to changing preferences or changing audiences. Your product audience is one thing this year then you have a lot of growth. It's a different audience. Their preferences are different, their needs are different, technology moves very quickly. The research that you've done two years ago might no longer be valid. And by looking at what unexpected things happen with our software or usage of the software is a really interesting thing. There was a lovely video I found researching this topic a bit more from Rachel Neumann in Australia, and she was at Eventbrite, I think, looking at bridging the gap between product management and product support. And her idea was that you have this kind of product support function or customer success function where people follow them and report problems and complain about bugs and all this other stuff, and that usually, then goes to testers, testers do some testing, then it goes to developers fixing whatnot. And she said, kind of these people are speaking to hundreds of thousands of your users every day or every year or every month, depending on your product kind of volume. It's such a rich data set that needs to be mined for product ideas needs to be addressed, and lots of organizations have this big disconnect between product management and support, which I think is very weird in a sense, because you can get all these ideas to complement your research for free effectively. Not every customer call about the bug is a bug. Not every bug is important. Not every unimportant bug is that unimportant. Maybe this is a hidden gem that you need to know. Hey, I think bridging this gap is easy for me because I said, everything in this product. But larger organizations, I really think should look at this disconnect. And I remember well, maybe even 10 years ago or something like that, we were working with an insurance company as consultants here in England to help them rebuild their testing function. But lots of testers left very quickly, and they were left without internal testing knowledge, and they tried to kind of hire new people to take over, and they wanted to make sure that everything gets started quickly. My colleague helped them hire this kind of head of testing, and this lady turned out to be such a wonderful, cross-functional expert in everything that she became the head of product at the end. Because she learned so much about kind of the whole product, approaching it from a testing perspective. And she had this mentality that you need to evaluate everything, you need to figure out how to do things. And I think this from like most of your audience are going to be testers in this podcast. I think, if you look at something like that, then testers are selling themselves kind of short or cheap. Testers can provide a lot more value to the organization by figuring out these things rather than just reporting bugs or evaluating whether the product is ready for a release or providing kind of feedback on that quality, I think. I'm thinking more about this in a sense where when there's a problem with the product, it might be. I kind of as a consultant, I think about quadrants. And it might be that the product is doing something that the user expected or user didn't expect, and it might be that the product is doing something to the product management expected or didn't expect like did we want the product to do this? Yes or no? Do the users expect that to happen? Yes or no. You have this thing where if the product does something that you didn't expect it to do, as a tester or as a person developer, product owner. And the users didn't expect that, that's probably a bug. That's a straight problem. But then you have these things where the product does something that you wanted it to do, but users expected something else. Is that a bug? Is that ux issue? Is that unexplored potential for growth? Is that something that is very, very interesting feedback that says, we don't know something about that audience or we've missed something that. And then you have these kind of strokes of luck where the product does something the users expect, but it's not what you wanted, which is sometimes a security issue, sometimes you have people figure out how to exploit you. Sometimes it's, again, an interesting area of growth because it might be that you think it's a bug and it's actually not a bug that's how people expect it to work. And then the I think, there's a lot more here than just is it a bug or it's not a bug. And I never had kind of I think that your book called mismatch by Kat Holmes while researching this and it's a book about accessibility in software and this concept of a mismatch, that's kind of a much more neutral word. It's not a bug, it's not a problem, it's not an issue, it's a mismatch between what the product does and what the users want it to do or expected to do, or what the user's capabilities are? Now, that mismatch can come from lots of different sources. You can have a mismatch of somebodies cognitive abilities. You give them too much text, they can't read it quickly enough. Or it might be a mismatch of ux where the button is in the wrong place. So it might be a mismatch of thinking about this mismatch is really, really interesting because then you can figure out whether you want to address it, whether it's worth addressing, whether it's, something that we do differently, maybe, and things like that. And one thing I really love about this whole mismatch concept is they talk about one of the four principles of accessible design from Microsoft, where the author worked. And one of those principles is he solve for the one expand to many. When you solve a problem like that, you don't solve it just for that particular group, but you try to improve the product so the product generally improves for everybody. And I think that's kind of thinking back it, some of the best kind of bugs I found working on other products and some of the kind of most interesting experiences improving that that's actually. You find a weird thing? You find the weird bug, but then, you don't fix just that bug, you fix a whole class of different things. You improve it in a way that makes the product generally better. And I think that compounds growth. And that's really, really interesting. They have some statistics. And again the book is focused a lot on accessibility and people's capabilities and things like that. But I think there's lots of things we can take from that and also parallels we can take from there. And you talk about kind of three types of disabilities that cause three types of mismatches there. And one is permanent. Say, if you're looking at accessibility and disability and things like people, there's I think they said the 16,000 people in the U.S who have a permanent disability of one of their arms, their arms was amputated or, some kind of genetic disorder that they can't use their arm. And then you suddenly have somebody who cannot use two hands with your software. That's not something that people often test like, can you use your software with one hand? But in the book, they talk about how if you just look at that population and then it's kind of often not economically justifiable to fix it. It's one of these things where you say, well, it's not the use case we want to cover, but actually so for want to expand too many, then you look at two different to more types of mismatches and disabilities. Let's say, you have these kind of temporary. It's not just somebody who is completely without one arm, but some like a mother holding a baby in her arm and she does not have to free arm. So you have somebody who's carrying a bag from a shop with one arm. He can only use your software with one hand. And then you have these kind of situational where somebody's profession prevents them from using both arms like they have to hold something, and then you talk about how in total, there's like 30 million of these people in the U.S at the moment. By optimizing the software to be usable with one hand, you're not making it better for 16,000 people. You're making better for 30 million people. And that's a totally different economic proposition. And I think and looking at that, that's kind of I think drawing parallels with lizard optimization, if you find somebody was doing something crazy, like trying to upload subtitle files or trying to create like videos, and you make the software better for these people, you don't just make the software better for these people, you make it better for lots of other use cases. And that's why I think doing that is a wonderful way of thinking about product development experiments, figuring out how people are misusing the software. And then you can say, well, out of all these misuses, we caught because there's going to be lots of lots of noise. You need to figure out the signal in the noise. You can't, of course, fix everything. You can then kind of from a product management perspective design, well, this is the one worth pursuing. And let's then design some experiments around that. And I think that becomes a really interesting aspect of the whole process. We collect lots of noise, and then from the noise we need to kind of draw some signals. And then when you do that, I think one of the kind of the way my thinking really evolved around this is that when you have a mismatch like that, people often think about, oh, I don't want to build more features or we don't want to invest in kind of just this because it's adding more complexity to the software. But if you think about it as well, it's not adding complexity to the software, it's about removing obstacles that the product is putting in front of users unintentionally. But my product was getting forcing people to create a blank PowerPoint to upload it, then to download an MPEG-4 file to extract an audio. Those are obstacles in front of a user's workflow. They're not additional features that should be something completely different. And then removing obstacles to user's success is really, I think, the kind of key there because then we can look at what the user success looks like, how do we make them more successful. And this is where kind of growth really comes from because when you get people to be more successful, they will stay longer with the product. They will bring their friends, they recommend it. And then the last step is, is double checking whether we've caused any additional unintended impact. And this is really, really important because the whole premise is that people are doing something we don't understand or we didn't expect. That means that our research, our internal knowledge, our data, our mental models about our users are wrong. And then, we try to do something else, but they still might be wrong. They might be some other kind of type of crazy and things like that. And one really good example of that is I was trying to optimize gaming success rates, especially for people in Europe. Companies in Europe need a bit more bureaucracy than in the U.S, in general, and they need to have their tax numbers on the invoices, they need to have a proper invoice and things like that. I'm using a American payment processor. They don't really understand Europe. That's kind of and they have this field where you can put in a tax id, but they don't validate it correctly, lots of people were dropping off because they couldn't input the number correctly, and no trivial number of people were kind of selecting Russia as their country just because the payment processor doesn't validate Russian tax numbers. And then I would have to go and say, well, this is a French company. You selected Russia country for tax. This is insane. And I thought, well, what about just removing the number there. And I think it later in the step on a web page, I control and say go through the process. You can put in a tax number later that will generate the invoice. And that way I was able to configure the payment processor not to prevent these people from succeeding. But then actually the results were that fewer people were paying because they were expecting the tax fields. And if they meant a lot was to send me an email saying, where do I put the tax field when I removed it? It's something, again, where I thought this was a good idea. Actually, it wasn't. And we double checking that it was good then and measuring and proving what the effects are, and if there are second order unintended effects is also really interesting because then it helps us kind of again, improve the loop and get some insights about what we want to do.