Intro / sponsor read
Yeah, now the key thing to notice there is automation would be unable to detect that problem, or at least automation of the kind that most people implement using the popular Selenium, Playwright, Cypress kinds of tools, or indeed anything that is driving the browser by interacting with stuff at the browser level like that. Now, that's not to say that these problems are necessarily common, or that the developer was particularly swift in this case. That's a whole different deal. But it is the kind of problem that automation would not alert us to. But maybe a good way to do this is to look into, to start again with the secret process. I'll do it this way, I guess. There's a secret life to work in any kind of work, really. A lot of what we do because people are hiring us not to think about it, their job, they don't want to think of it in their job, so they hire us to do it, which is great. Yeah, that's fine, it's a good thing. But there's some processes that we like to talk about, and then there are other processes that we don't like to about so much. We prefer to keep it secret. Partly because we don't want to necessarily expose people to the sausage being made, but also because sometimes there are certain things that we might not be aware of, or certain things we might be a little bit embarrassed about if they were to know about it. Here are some reasons why there might be secret processes, and we're focusing in particular on the world of automation. One thing that we hear from a lot of testers when we quiz them to kind of interrogate them on what they're doing, is that there are lots of things that they actually do that they've not noticed before that they're not taking account of. For instance, I was speaking with a tester yesterday and asking her about what she does. And she says, well, I get a set of checks that have been developed by other testers and I turn those into code. Okay. Is that really all that happens? And she said, yeah, yeah. Well, okay, so what else happens? What else is happening in there? Let's break that down a little bit. And she says, well, I start the product up and I interact with it and I make sure that I can do the things that the other testers were able to do. I said, oh, okay. There's actually some interaction that you're going through yourself. What happens when you find a bug? I asked, and she said, well, if the check crashes or if it yields a bug, well then I investigate it and I try to figure out whether it's a bug in the check code or a bug in the product code or something about that. Her epiphany at that point was, oh, wow, if we really break it down, there's lots of stuff that I've never noticed before. Lots of stuff in the process of developing and applying tools that people don't talk about because they don't think about it. It just goes below the level of their consciousness and they're just doing work. That's a big factor. Sometimes, there's something going on that they don t really know how to describe very well, and it's partly because lots of what we do is involved, is embedded in tacit processes, stuff that even we have not made explicit to ourselves. We don't notice it and then when we do notice it, well, then we don't have words for it, we don' have terms for it. Part of what James and I do a lot in our work is we try to make visible and legible and comprehensible things that are sort of below the surface of what we think about and talk about much of the time. We work on that. We try to name heuristics that people apply, for instance. Another reason that people are kind of secret about some of the work they do is that when we describe it to people, they don't get it. They don't have the lived experience that would be required for them to understand what's going on and why it's important and why makes sense. Some things are secret because we don't record them. And we don't want to record them. That feels icky to lots of people. Some processes are forgotten. Now, one of the things we say about note taking and testing is, listen, it's perfectly okay not to keep records and not to take notes as you go. Perfectly okay to do that. As long as you're cool with forgetting lots of stuff. I mean, that's my experience and your experience too, that it's stuff that we don' diligently record. We forget about it. And lots of the time that's perfectly okay. And sometimes it's important. Sometimes people say, well, managers actually don't care what I do. They just want results and hear about that. Or even worse, they say they just want numbers. I don't think that's true. Actually, they don't want numbers one way or the other. What they want is a feeling. They want a belief. They want an understanding that was John Bach that James's brother once said to me. What do they want to know, they want to know of me as a tester, they want know, am I on top of it? Well, that's true to some degree, but I think what managers really want to know is from the manager's perspective, am I on top of it? Managers want to be aware of the status of their product. Now, there's some people who have a sort of cynical belief that managers don't actually want that. And to a certain degree, that is always true. Nobody wants to hear about problems in the product. Nobody wants the hear about that. It's just that pragmatic, responsible people would probably prefer to know about them so they could deal with them rather than being oblivious to them in such a way that they could cause harm. This is one thing that afflicts me all the time. And that is I'm shy about the code that I write because among other things, I cut my teeth in a company where the programmers were assembly language level programmers. And I feel like if I showed them my code, first of all, it wouldn't be up to their exacting standards, but also it would look simplistic and like a newbie stuff and that sort of thing. And in a way that's just kind of silly because these guys were a lot smarter than that. They've recognized that, hey, code is what you need to get the job done. When you do that, don't worry too much about it. If it's not frightfully important that it be super, super precise or super clean or super elegant, then it's a not a big deal. But as humans, we kind of get shy about that. A lot of the time, people are embarrassed about the way they test as being redundant. And there's actually something to that sometimes. I'm sure you've heard stories of people who are testing at the GUI level, things that are not only easier to test at the API level, but have been tested already at the API level but because the GUIs automation is dazzling, they sort of get away with that. It looks good. The opinion of the person who's not looking too closely is, well, if it looks good, it must be good. Sometimes the process involves recognizing that there's maintenance stuff. Sometimes people talk, decline from talking about certain aspects of their process because it's really hard to do and they can't tell when it's gonna end. They just dampen that conversation. Here's something that I actually often have a hard time talking about. I don't know sometimes how to talk about things in a general way, such that other people won't come along and say, well, that doesn't apply to us. Well, let's actually true but if it doesn't apply, oh, well, it does apply usefully in this sort of circumstance. So that's something that all of us need to get over a little bit and recognize, hey, if we found a useful way of approaching a problem and using tools to do it in a certain situation, we can talk about how useful it is in that kind of situation and not worry about the circumstances where it's not. Sometimes the code that we inherit is opaque. Sometimes, we're secretive about our process. We like to tell people, oh, well, I read the requirements documents and I read specifications and I read the API documentation. Well, no, actually we don't do that sometimes. A lot of the time you just sort of dive in and try to get to work. And that, again, that can be a little bit embarrassing. Lots and lots of text on this slide. But basically, we're trying to help people become aware of their processes in our work. And there's certain obstacles to that, with respect to knowledge and awareness of the work, the nature of the work itself, right, this sort of higher level strategizing part of it, the relationship to the work that people have. How much agency they feel they have over it, how much they have to be accountable for it, the actual data about it, the data that needs to get collected, and then our own feelings and the feelings of our clients about that. And so sometimes those processes are secret. The outcome of that is that a process may not be described in a legible way. Legibility is an important kind of idea in our work. Legibility, it's almost literally readability, the ability to observe something and to make sense of it because it's been presented in a way that's easy to see and easy to wrap your mind around, easy to understand. One of the things that's kind of seductive about automated GUI-level output checking is that it is quite legible on a certain level. You can see stuff going by on the screen. You can see stuff that looks like user interaction. You can see browser behaviors flying by. And that looks impressive. It looks like a superhuman doing things at superhuman speeds. But there's lots of stuff that might be going on in that product that the dazzling nature, the legible nature of that stuff doesn't illustrate so well. The processes that underneath that are the processes that lead to it or the processes of what we do with the information after the checks run, that is not so legible, that's not so visible. For instance, when I was asking my student the other day about, well, what actually goes on in your work? She didn't talk at all about the processes that involve investigation of what happens after a check has run red or after a cheque has crashed. She made a distinction between her work and what the other people in her crew do. She is the automation tester and they are the manual testers. Now this is a distinction that bugs me no end. And the reason that it bugs me is because it seems to me that neither one of those terms, manual nor automated, do very useful work in describing what we actually do, especially the manual bit. By manual, people often mean, well, let me give you a little visualization of this. By manual people often means, and where did that slide go? Oh, there it is! And that is slide number 50, so it seems. So people talk about manual testing. Now, what do they actually mean by that? For a long time, it seemed to me simply that when people talked about manual testing, they didn't say, I do manual testing, what they meant was I don't write code to operate the browser. But it bugged me one day because a tester was saying, I'm worried that I'm gonna get replaced because I'm a manual tester and my company is going whole hog into automated testing. And first of all, testing can't be automated. All those things that I mentioned earlier on about investigating, critical thinking, risk analysis, interacting with the product, and that sort of stuff. Interaction with the project on a sort of surface-y level, that can be automated, but all of the thinking stuff and all of investigative stuff and all the analytical stuff that goes into testing. That can't be automated. Those are distinctly human processes. So it bugged me when she said, I'm a manual tester. I said, no, you're not a manual tester. You interact with the product. You get experience with it. You're an experiential tester. And that sort of started turning on a bunch of light bulbs for me. How can we express what testers actually do so that we recognize how kind of bankrupt the term manual testing really is. What do people actually mean when they say manual testing? Sometimes what they mean is they mean interactive testing. They're interacting with the product directly and they gotta be there. There are some processes that go on within testing work that can indeed be done in an unattended, non-interactive kind of way, right? You give a process for a machine to perform and it performs that process and you come back later and you look at the output and you look at, the logs and you look at the state of the application of the data afterwards and you make some decisions about what to do next. Whereupon you're back into interactive work again. Another thing you could say though, is that you interact experientially with the product. That is to say, your encounter with the product is practically the same as that of a particular user that you had in mind. If somebody were watching you working, they'd have a really hard time figuring out the difference between you, the tester, and some contemplated user, except for one thing, maybe, and that is when the contemplated users encounters a problem, the user is going to say, Damn it! And try to work around it. Whereas, a tester is going to say, whoa, that's interesting, and it's gonna start investigating. But when we talk about experiential testing, we're talking about testing where we are interacting with the product in a way that reproduces or that anticipates the way that somebody affected by it is going be interacting with it. Sometimes we say user, I wanna open the fence on that idea and say that an ops person is a user of a product. The person who is the client of somebody who is directly using the product is also a user. We want to have a very expansive notion of that. Now, contrast experiential testing with instrumented testing, where something gets in between the person interacting with the product and the product itself. And automated checking is like that, right? There's a medium. Something in between that is going to alter, or change, or distort, or accelerate, or intensify, or extend, or enable, or disable, or limit, or hyperspeed the interaction with the product. A lot of problems in automated checking are the consequence of that the tool is running too fast for the product to respond. The distortion that we're seeing in instrumented testing can be a feature, it can also be a bug. This is not a value-laden statement that the experience is distorted. It's just changed in some way from the naturalistic experience of the person using the product. We could be testing, when we say manual testing, what we could be talking about is testing where the choices have not already been made for us, where we're exercising agency over what's going on and our procedures might be open rather than constrained by something. The essence of exploratory work is making choices. And we do that in automation work all the time. We make decisions about how to write and maintain our scripts. Another way we put it is there's no script that tells you how to write a script, and there's no script the that you how to investigate a problem that you found, that your check has revealed. That's an issue. And then finally, sometimes this is on the list, and sometimes it's not. But let's put it on the lists for today that some interactions with the product that we call manual testing are sort of transactional. We don't learn anything from them in particular. We're just walking through a routine that somebody set down beforehand, and the test doesn't affect the tester. But to us, if it's going to be a test, it has to affect the testers some way. At very least, the tester needs to learn something from the experience of performing the test. People who are heavily engaged in developing and maintaining and executing automated checks are doing this kind of stuff all the time. So-called test automation work involves an enormous amount of what people would call manual testing. And the trouble with manual to us is that it hides what's really going on. It could manual, after all, it means using your hands. But the hands are merely an input mechanism. They're not the essence of what's going on. So we just encourage people to drop the idea of manual or automated testing and talk about testing instead, and then talk about how we would apply tools in testing.