DevOps In Test with Rosalind Radcliffe

[00:00:01] Get ready to discover some of the most actionable DevOps techniques and tooling, including performance and reliability for some of the world's smartest engineers. Hey, I'm Joe Colantonio, host of the DevOps Toolchain Podcast and my goal is to help you create DevOps toolchain awesomeness.

[00:00:19] Joe Colantonio Hey, it's Joe, and welcome to another episode of the Test Guild DevOps Toolchain. Today, we will be taking a blast from the past as I share with you a never-heard-before session on this show from the 2017 Automation Guild with Rosalind Radcliff. Even though this was a while ago, all information is still relevant today and that's why I want to share this gem with you. So in today's episode titled DevOps and Test, our speaker, Rosalind Radcliff dives deep into the world of interface DevOps testing and its crucial role in the API economy, cloud, and digital transformation we all see ourselves in. So join us as we uncover the secrets to successfully testing in a DevOps environment and learn how to optimize your DevOps testing practices. If you don't know, Rosalind Radcliff is the IBM Distinguished Engineer responsible for driving the DevOps for Enterprise System Strategy at IBM. That includes working across IBM to provide support to include all platforms in a DevOps culture and remove any barriers to the transformations from a technology standpoint. She also works with clients in the DevOps transformations and she is one of the go-to folks I listen to for anything DevOps-related. Also, since this was taken from a previous Automation Guild, if you want to speak at the upcoming Automation Guild 2024 online conference dedicated to all things automation testing, including DevOps, performance and security, the call for speakers is now open. So if you'd like to submit a session idea, I'd love to hear from you. Just go Guildspeakers.com and submit your idea now.

[00:01:57] Hey, if your app is slow, it could be worse than an hour. It could be frustrating. And in my experience, frustrated users don't last long. But since slow performance is sudden, it's hard for standard error monitoring tools to catch. That's why BugSnag, one of the best production visibility solutions in the industry has a way to automatically watch these issues. Real user monitoring. It detects and reports real user performance data in real time, so you can quickly identify lags. Plus, get the context of where the lags are and how to fix them. Don't rely on frustrated user feedback. Find out for yourself. Go to BugSnag.com and click on the free trial button. No credit card required. Support the show and check them out.

[00:02:44] Rosalind Radcliff Hello. And I'd like to personally welcome you to this keynote session of DevOps and Test. My name is Rosalind Radcliff. I'm an IBM distinguished engineer responsible for DevOps for Enterprise Systems. I spend my time talking about DevOps with clients, helping them understand the cultural transformation, understand the changes that they need to make in order to become a DevOps environment. I also work with our internal organizations to help them and their transformation. And my job is to help us build a set of products and capabilities that will help you in your job. So I'm really happy to be here to give this session. So let's get started.

[00:03:27] Rosalind Radcliff The world is changing. Technology is changing the way we all work, the way we interact with the systems and the way we want to do business. Collaboration is significantly different than when I started IBM 30 years ago. Mobile. Did you ever think about a mobile device? Not back then. So the world is changing, but more importantly, it's changing even more dramatically. If we look at what's happened in just the last few years with the growth and cloud, with the Internet of Things, with the growth in digital transformation, think about the Uber's of the world. The fact that you can just build an application and become a business. And I used to think that, well, that was only for certain parts of the industry, but not really. It's really for anyone. Any part of the environment can be transformed with a simple app. I can build a bank with just an application using someone else's services. And so all of these technologies are transforming the way businesses have to work, the way businesses have to think about doing their job. And on this, you see security and security is the key piece of this. We need to make sure that we're doing this transformation, providing this capability, all while providing it in a secure environment, making sure that nobody can get to that client data or nobody can get the information they shouldn't be able to see while still having access to all of the real-time analytics, you need to be able to make the right decisions. I get the message from my phone that it's 10 minutes for me to get to my next meeting. Wait a minute. It's a 10-minute drive? Why do you know that? Because my calendar has the location of my next meeting. So the data that's available and accessible, anything you make available is now being used to change and transform the information that's provided to be able to do personal marketing, to be able to provide the right information at the right time. So this is critical. How do we deal with this transformation? How do we deal with all of these changes to make sure that our companies and what we're doing stay the most advanced and stay on the top?

[00:06:13] Rosalind Radcliff Well, if we think about this transformation. APIs are what's providing the ability for this digital supply chain. APIs are transforming the way businesses are working. There are sets of APIs available, and by using those APIs, I can bring them together to provide a new business. So not only do we have to focus on the providing of these APIs, but it's also providing a set of APIs that can be composed together to provide additional business value. And if we think about traditional enterprises, we have a set of resources and systems that have been available and we've developed over the years. How do we expose this function, this value to ourselves and to outsiders? In many ways, this transformation is all about I.T. becoming a service provider to that business. And by facilitating this transformation and by allowing ourselves to be able to work at the speed business requires, requires a true transformation to the way we work. That's where DevOps comes in.

[00:07:36] Rosalind Radcliff What is DevOps? DevOps at its core is the thought process around Agile and Lean I.T. brought together to drive a cultural transformation. At the core of DevOps, it's all about your culture. It's about how people work together to provide business value. It's about the time that they can work together to provide what you need to provide to your clients. It's that underlying culture. And through that the change in processes. Technology is there to support it. And so we all need to have the right tools and capabilities to support this transformation and to support the automation required. But at the heart of it, it's all about culture and process. It's all about how we work together, how we break down those silos to deliver business value faster, how we can actually improve quality and reduce risk. At the same time, while delivering faster. And it's about reducing the time to getting the function into the end user's hands to get feedback. You need that early feedback to understand if what you're building is the right thing. As I said, I've been in the industry for 30 years working for IBM, and over that time I've worked in many different jobs and many different responsibilities. And over that time, when I started in the company, we used to build things in 18 months. We'd have a list of requirements, we'd build the product and we'd ship it, and then we'd find out whether or not it had satisfied the customer requirements or was nowhere near what it was required. I've worked with companies throughout the years who had the same processes, the same old waterfall processes. Everything was manual if you weren't lucky. Sometimes things were automated, but you had this overarching set of different teams that you passed function on to the next level. By the time we shipped the product, did we really know what the real requirements were? No. So it's important to get this continuous customer feedback. It's important to do this transformation to this new way of thinking, to this new way of working together to provide the functions that we need at the speed required. And to think about doing things in much smaller chunks.

[00:10:25] Let's do the smallest function possible minimum viable product. I hate the name, but I like the idea. The smallest thing I possibly can produce that provides real value and can show business benefit. That's what we need to focus on. This transformation to these small functions, delivering value, getting customer feedback, and then repeat to provide more value and build up the capabilities over time. If we look at DevOps, it's really about this cultural transformation and changing the processes to support that. And then in support of that, you have to select the right tools and the right capability to allow your organization to do this transformation and to be able to do what they need to do as part of this. If we look at DevOps and we see the diagram here, it's focused on culture at the middle and then the different practices that are around that cultural transformation. As part of this, we're focusing on actually the code generation, building the code, building the services necessary and testing of those features to make sure they are fully tested from the standpoint of testing, unit testing, function testing, system testing, all of the different phases or kinds of testing that you need to test all this through automation so that you can deliver through this pipeline as fast as possible to be able to provide that value. Here in the diagram, I'm using the Bluemix Garage Method to describe how you can take these concepts and practices and apply them within your organization. The Bluemix Garage Method specifies a set of practices and capabilities to allow you to help transform and understand the changes and practices and processes necessary, including things such as test-driven development. Making sure you're doing the right kind of testing at the right time and automating it in the correct way. If we think about real environments, they are complex and there is not just one application in order to provide business capability. It's built based on a set of different components that all have to work together. When we think about an environment like this, we call it hybrid applications or hybrid platforms. You have a set of teams who are working, building a set of functions that have to come together to provide the end value to the client. As we're going through this cycle, you have different teams working at different speeds that all have to coordinate to some extent. So you need to figure out the right ways for these teams to collaborate in the right ways for these teams or these different applications to interact.

[00:13:35] Rosalind Radcliff In a DevOps organization, you may easily have separate teams building these separate sets of functions. And if we think about building them as APIs, hopefully, you're building them in such a way that there is independence between them. You're building a set of services that you can call independently so the services can be available in the system and upgraded as they need to be. And then when you use that function, it is available. But being able to test in such an environment requires additional capability, thought process, efforts, processes, and procedures. Take your choice of term. You need to make sure you've set up the environment to handle virtualizing the services that you don't, aren't directly needing and aren't directly under test while you are testing the portion of code that you need to test. When do we think about these hybrid applications and we think about environments today?

[00:14:45] Rosalind Radcliff One of the key reasons that customers are looking at DevOps and looking at DevOps as a way of for this cultural transformation is the fact that in many ways they're pretty far behind in tests, especially with those systems of record. Many times I see where the percentage of test automation is very low. The amount of unit testing is very low because these applications have been built as large monoliths. It's harder to do unit testing, but in order to do the transformation, in order to do the right thing for the business, you need to start somewhere. Start with some amount of unit testing. Start with automating the tests that you have. Understand what tests you have and how you can bring them together to do better testing through test automation. If we think about many organizations and the way they do tests in a traditional waterfall environment, you're building all your code and then you're just turning it over to test. So it really hasn't been tested at all. And now this test team is working and finding things that logically should have been found right away. And by the time the test team does test, the developers have moved on to something totally different and they have to go remember what they wrote by building in the test automation and by building the proper delivery pipeline. You have all of the information you need right away about the capabilities that you're building. Now, I can remember a very long time ago when I started development. The products I started with, we had a regression bucket so I could actually run the regression bucket when I went home at night. Back then, 30 years ago, we actually could go home at night and not work because we couldn't access the system. Times have changed. So you don't have all night to run the regression bucket. But we could. So the next morning when I came in, I knew if I had broken anything in the environment, having that automated testing, having it available and having it part of the system and just part of the build helped get the feedback I needed as fast as possible.

[00:17:15] Rosalind Radcliff Now, one of the key things that DevOps also brings us is a set of standardization of tools and practices in order to drive this enterprise focus. If we can get rid of the silos, if we can get rid of the islands, and if we can have people collaborating and working together, it doesn't actually mean everybody does everything exactly the same. But it means there are a set of standards and a set of normal practices that allow the teams to work together, to be able to collaborate and to allow them to move between functions as necessary. Simple things like standard testing practices, ensuring you have the test automation available, the automation frameworks available to do the testing that you want to do. Having the automated delivery pipeline where everyone can use it so that they can get the function. Built as fast as possible with the capabilities that they need.

[00:18:26] And if we think about DevOps, we think about application development, we think about this transformation to the digital economy. There are sets of very significant architectural decisions that are being made that influence the way development happens. They influence the way development works and influence the way people can work. The first thing is the application type. What type of application are you building? Is it cloud-native? Is it cloud-ready? Is it already existent? And a monolithic app so that you already have a set of functions and a set of capabilities that you're dealing with? Are you working in a hybrid environment? Most people will be. Do you have a mobile front end? Many people do. There are maybe some applications that don't, but many do. Can you move to microservices or have you already moved to microservices? Are you doing microservices for parts of the application or parts of the capability while you're making other choices for other parts? All of these influence the way. You use DevOps in the way you do your testing, but none of these change the fact that you can't or cannot do automated testing or do DevOps. When we think about the transformation, monolithic apps or the one that I think most of us probably wouldn't choose today, however. If we think about some functions, the speed required to deliver that function may mean that it's more of a monolith. You may not want to spend the time calling between different microservices. If what I'm trying to do is process a credit card, I don't want to have any additional hops, any additional extra work done. I just want to get that authorization for that credit card as instantaneously as possible. And so I want to make sure I'm making the right decisions from the way the application is built, the way I'm delivering that function so that I can provide the value. Those are all nonfunctional requirements that need to be tested as well. So as part of this DevOps transformation, we need to make sure those nonfunctional requirements are still as obvious. They are still tracked and they are managed and they are tested into the solution. The sooner than I can get the information about needing to fix a performance problem, the better off you are. The sooner that I can find that this transaction is not going to be sub-second, the better off we are. We also need to think about the tool chains that you're going to use.

[00:21:50] Rosalind Radcliff In today's environment. We have Mobile front ends. We have Internet of Things devices. We have middle mid-tier portions of the application. We have systems of record. We have data we need to manage. We have data that has to stay within countries. We need to be able to manage this end-to-end environment and provide the security and auditability for the auditors coming in and reviewing to make sure you're following the practices that are required. You need to follow the separation of duties requirements when you're dealing with certain industries. All of that can be handled via the right end-to-end tool chains, bringing the right pieces of product capability together to allow you to have requirements tracking from the very beginning through the delivery of the function to provide that source control so that you easily know this is the source that maps to the thing that I'm running in production. Ensuring that you have that complete deployment automation so that there is no manual intervention, not only for the application but for the parts of the application. When I think about doing tests in DevOps by having a fully automated deployment for not just the application but by for all the configuration changes, all the infrastructure changes, I can then easily deploy the environment and run all my automated tests and then throw it away. Moving to this dynamic creation of environments so that they can be tested and then removed allows the transformation in how we use our system resources as well.

[00:23:48] And then there are lots of choices about where the platform is going to be delivered. Is it on-prem? Is it off-prem? Is it in a hybrid cloud? How am I going to do that? And all of those decisions affect the way I'm going to do my testing and I'm going to do my automation. I need to be able to support the different environments. It may be that my primary interest is running it on-prem, but I need to have the ability to burst into the cloud at times. If that's true, then my testing environment needs to be set up to support that same model to allow me to test that first feature, to allow me to include all of the different scenarios that are possible in my design selection. So now if we think about test and we think about DevOps, there are a set of key functions, solutions, and capabilities that we think about. The first one is the automated deployment for additional test environments. It's the ability to dynamically provision an environment as part of your pipeline so that you can ensure that you are testing the function as soon as possible in your delivery pipeline. You're delivering the value and making sure that you have all of your configuration, all of your changes, everything. Infrastructure is code as part of your deployment, as part of your test, so that you can ensure by the time you deploy it to production, everything is. Correctly done. Another key change that comes with the DevOps transformation is the shift left testing is moving the testing as far left as possible. Now, honestly, this is just a best practice, but DevOps has brought to us the thought processes that were moving everything sooner in the lifecycle. We're bringing the teams together so I can do that test-driven development. I can write my test at the same time I write my function and I can make sure that I'm delivering the value with that test during the same sprint.

[00:26:11] Interface Testing is another key way that through the API economy and the digital transformation, we can make automated testing easier. By testing at the interface level, we can make sure that we have the testing we need of those APIs and can allow other organizations or other parts of our own organization to use those services. Interface testing is easier through an API than actual user testing, and so automating that is a key part of this process. It also gives us a way to virtualize off services easier to allow the testing to test only the area that I'm immediately interested in when I'm doing my earlier stages of testing. When I think about this transformation, I have to actually say that a long time ago or numbers of years when we were doing SOA or services-oriented architecture, we thought about these service interfaces and thought about the requirements for virtualization. This way, I could have this virtual service for that service that I was going to call that was going to be provided by another partner or another vendor and not have to pay for all of those calls early on, making sure that I wasn't dependent on their test environment, making sure that I could truly work and build the function without necessarily having to involve other partners. Making sure that you have the testing in place to allow you to do the refactoring of the applications. Today, we do have many monolithic applications, many monolithic environments that people want to split up. By having appropriate automated testing in place, it makes it much easier to refactor those applications, makes it easier for you to actually make those changes and ensure that your business continues as normal while you are adding value by making this function available to others. And one key area of testing and DevOps is ensuring that you have the right management, monitoring tools and operations feedback along the whole cycle, using the same monitoring tools that you're going to use in production so you can get the application performance understanding as early in the cycle as possible. As I mentioned earlier, if you have a transaction that has to run in nanoseconds and you're working on a part of it. And you can get performance data that tells you that that transaction piece that you're working on is not sub second. Then you know you have a problem early. Finding performance problems earlier in the cycle is very critical. Being able to monitor the environment with the same tools also helps ensure you don't run into some of the more interesting challenges that I have in my past.

[00:29:36] Rosalind Radcliff I actually have a really fun example of making sure you're using the right true production monitoring tools in your development environment. It was quite a while ago when I was working in SOA and working with a client to build a set of services and provide the right monitoring for those services in the environment. As we worked through the system, we had developed the services, deployed them in test environments, validated their function, validated what they were going to do, but they couldn't get the production monitoring tool in a test environment because, well, they couldn't for some miscellaneous set of reasons. And so on deployment weekend here we went and deployed the services or tried to deploy the services into the production server, but this was in the early days of Web services and trying to deploy the service into the production environment wouldn't happen. It failed because they were both using the JAX handler. And so there was a problem between the monitoring tool and the application. Now the customer knew well enough to know the monitoring tool was working just fine, worked well with all their other applications, and was actually working with a few other services-based applications. So this was a development problem and so they sent it back to the drawing board. But the problem was this application had to be deployed this weekend. So it's a significant set of work for a number of people to get it all fixed, stop interfering in the handler in order to actually deploy the application into production. If the production monitoring tool had been running in the development environment, that would have been found on day 2. So it's a simple example, an unusual example, but a simple example of a place where it's critical that monitoring tools be in your development environment. There are better examples, though, when you look at actually monitoring the performance of an application. If you use two different set of monitoring tools, I can almost guarantee they're going to generate different data. So is your end-to-end transaction performance has to meet a certain set of criteria. You need to be using the same tooling to monitor it everywhere so that you can ensure that you're actually satisfying the requirements. I've seen in a couple of cases where different tooling was used and when it got into production, it was failing, the audit was failing the actual SLA because it didn't satisfy the monitoring tools, recording of the process transaction time. So testing is critical. Testing with the right tools, testing with the right capabilities, and testing an environment that truly does look like the production environment with the same monitoring tools and using those monitoring tools to get the data so that you have the information. Now, I know you may not want to run production monitoring tools on everyone's laptop, but there are plenty of stages of test between a laptop and the production servers to allow you to run the monitoring tools, get the data and have the information that you need early in the process. And this last tile Modern development Tools and practices. Test-driven development is one that has been in practice in a number of organizations and helps drive significant value. It's part of a DevOps cultural transformation. And yet there are many organizations who still don't focus on test in that way. Truly having test and development working together so that they're building the function and testing it at the same time to make sure that you're delivering the value, having your developers build unit tests for their test cases, for their test function, to make sure that for every piece of code, they're writing, they have appropriate unit tests being added into the system. And then if we think about this with applications over time, it's important to manage these tests, manage the test cases, manage the test environment so that you're not just forever adding, but you're making sure that the tests that you have are the critical tests. So using tooling to make sure that you understand whether or not you're running the same test, that test the same function, or you're running a set of tests that actually test a variety of scenarios. And the other piece of this is doing non-happy path testing. It's one of the critical functions that I've discovered as part of this DevOps transformation. When we're thinking about automated testing and we're thinking about doing the tests, more often than not, we're focused on are we successful. Does the function work the way it's supposed to work? And we forget all those error conditions and all those scenarios that we aren't necessarily planning for making sure that we're doing all the negative testing, security testing, penetration testing early in the lifecycle as early as possible so that you get all of that information, so that you have all of what you need in providing your solution before you ever get to production is critical. I want to thank you for your time today. Hopefully, this discussion of test and DevOps has been helpful and you've got some insights into how you can better prepare and you can better do your own transformation. I really appreciate your time and I look forward to your questions.

[00:35:35] Joe Colantonio Thank you, Rosalind, for your DevOps awesomeness.

[00:35:36] For links to everything of value we covered in this DevOps Toolchain Show. Head in over to Testguild.com/p127. Remember latency is the silent killer of your app. Don't rely on frustrated user feedback. You can know exactly what's happening and how to fix it. With BugSnag from SmartBear. See it for yourself. Go to BugSnag.com and click the free trial button. No credit card required. That's it for this episode of the DevOps Toolchain Show. I'm Joe. My mission is to help you succeed in creating end-to-end full-stack DevOps Toolchain Awesomeness. As always, test everything and keep the good. Cheers.

[00:36:20] Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps, and automation professionals in the world, we'd love to have you join the FAM at Testguild.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real-world value that can improve the skills or solve a problem for the Guild community. I love to hear from you head on over to testguild.info And let's make it happen.