How To Optimize your Automation CI/CD Pipelines (and Save Money) with Gaurav Mittal

[00:00:35] Hey, want to know how to optimize your automation CI/CD pipelines and DevOps and save money doing it? If so, you're in luck, because today, we'll speak about Gaurav Mittal. If you don't know, Gaurav is a cyber security data science and IT expert with over two decades of experience, leading high performing teams in cloud computing, machine learning, and data security. He's also a thought leader in AI and automation and has published numerous articles on optimizing ML deployment, security, email communications, and automating workflows, and a bunch more. He's also recognized with multiple industry awards and has an AWS certified cloud practitioner and a lean Six Sigma certification professional. You don't want to miss this episode. Check it out.

[00:01:16] Joe Colantonio Hey Gaurav, welcome back to The Guild.

[00:01:20] Gaurav Mittal Yeah, hi Joe. Thank you for having me again. It's wonderful to catch you again and it's a pleasure to be on your show. Thank you.

[00:01:29] Joe Colantonio Absolutely. So you're like a jack of all trades. Why is this topic top of mind now? Like how to make DevOps more cost effective?

[00:01:39] Gaurav Mittal Whether you are working as an automation engineer, you are a security professional, you are AI/ML, DevOps is everywhere. Because ultimately, you have to build the pipelines. Everyone loves to click one button and make sure whatever they are coding, it's gone. Okay, they don't want any hassle of deploying it again and again. And like IT professionals today, they are working on like branches. So 10 people, they have their own branch, they want to push the code to the main branch. They want to get it deployed as soon as it's done so that all the checks, all the automation tests, they are being triggered automatically. DevOps is everywhere. That's the reason I also learned it. I implemented it in my organization and I brought it up this topic. Hopefully, I'm happy to share my knowledge on this.

[00:02:26] Joe Colantonio Love it. And actually, you wrote a really cool article on curtailing software automation to licensing costs by curbing licensing needs, which I have a link for it down below. But let's dive in a little bit about that. What's up with licensing needs? What's the roadblock there?

[00:02:43] Gaurav Mittal Yeah, so most of the product based companies, they love to procode licensed tools. For example, like when it comes to automation, okay, like there are different engineers are working in different kind of domains. Some engineers, they are deployed in Salesforce, others are in web. Some are in data. Each of these domains, they're using different language. Ultimately, you will end up in hiring automation engineers who are expert in this language. Instead of that, the best way what I have seen the trend, what organizations they are rooting towards is procured a licensed automation tool. The problem is you have procured an license tool, you are paying for it, but license are limited. Most of the automation engineers, they are sitting at the offshore. Now add onshore to how many folks you will provide this license. The best outcome is implement a CI/CD pipeline, which will sit on top of this license tool. The advantage is let automation engineers, they write the test cases, but anyone you are giving the flexibility to trigger the license automation tools through CI/CD tool. That's the main benefit of it, like how you can curtail the uses of license tool. Just make like a CI/CD tool, CI/CD pipeline implemented, which is going to trigger automation tool. And hence, like product order, or like managers, or anyone, they can trigger the automation test tool. No need to have license.

[00:04:15] Joe Colantonio When I look at the article, I think you mentioned something about using GitHub actions to help with this. How does GitHub actions help?

[00:04:21] Gaurav Mittal Yeah, that's the second thing which I want to touch. Actually, there are several CI/CD tools in the market. Okay, like for example, we have Jenkins, we have Garset, we are Copado, for different domains we have different GitHub actions, GitLab. When like I was supposed to like implement a pipeline, I have done a research and ultimately narrow it down to Jenkins and GitHub actions. So Jenkins, like everyone knows it. They love it because it has a very nice UI and it's quite easy to use. You have to just write Jenkins file and like you can create the jobs. If there are some existing jobs, you just copy it. Managing is quite easy. The problem is Jenkins is still paid because you need to pay the server costs, which is not much, but you have to bear for it. And second drawback is like there is a quarterly unit to rotate the security keys which is again a pain point. But the benefit with the GitHub Actions is, it is open source. And secondly, if the developers, they are using GitHub, they are very much familiar with the Github Actions. There is no learning involved, there is no license, nothing. You implement GitHub Actions, which will trigger your GitHub repo and your pipeline is implemented. Other main advantages, like in most of the companies, this GitHub is being maintained by their security team. In contrast to Jenkins, Jenkins is calling your automation test tool. You are making an API call. There is some sort of permissions issue may arise. With GitHub Actions, it is all internal. Your security team, they are handling GitHub actions. They are handling GitHub repositories. No need to worry about any permission issue. It is quite straightforward. GitHub Actions is very winner. It's easy to implement. Developers, they already know it. QA, they were writing Jenkins file. Here, they have to write a YML script and you can invoke your licensed automation tool.

[00:06:30] Joe Colantonio All right. I'm not sure if I'm following all along. Say I have a paid vendor tool that has licensing. I have 10 licenses and I have a team of 30. Is it because it's running in CI, it's an ephemeral environment and only uses that licensing while it's up or like how does it actually save or help optimize license usage?

[00:06:50] Gaurav Mittal Yeah, so like you said like suppose there is a 30 team members .

[00:06:54] Joe Colantonio Or maybe I'm not following along.

[00:06:55] Gaurav Mittal Yeah, so like we have 30 team members and only 10 licenses are available, okay? You have given those 10 licenses of automation tool to automation engineers. Now, tomorrow when the rest 20 they want to execute the automation test, how they will execute? They have to ask those 10 guys, hey can you please trigger this particular set of tests? Instead of that, if you are building a CI pipeline which is capable of invoking these automation tests, you don't need to ask anyone. You have the flexibility. This problem is very helpful when you have offshore and onshore because most of the license that goes to offshore guys, they are working there and because of the billing reasons, they are being, the automation scripts, they are been developed there. But at onshore, like in our time, if you want to execute those tests, you don' want any dependency for that, like just to make a CI pipeline which is invoking this automation test and anyone can execute them, no need of license.

[00:08:03] Joe Colantonio Alright, cool. You keep mentioning offshore onshore teams. Are there any other ways you can help cut costs besides this approach?

[00:08:09] Gaurav Mittal Yeah, like even within one team only, like the license, they are quite expensive. Suppose your team starts only at onshore and there are 2 QA resources. Instead of asking every time like, okay, can you execute them? Or even if they are scheduling, but you want to run, like there was a production incident and for the sanity check, you want run automation test. So anyone should be capable to trigger it at any time. It's not only about onshore and offshore. The point here is about the flexibility. Who can trigger the automation test? And for that, the CI pipeline is actually very helpful.

[00:08:46] Joe Colantonio What are some other benefits? It sounds like once again, you get rid of the manual oversight of the UI test executions because the CI pipeline takes care of it for you when all you have in a monitor, am I understanding that?

[00:08:58] Gaurav Mittal Yes, that is another advantage what happens like you are running your UI web application automation test and when the tests are running, sometimes what happens there are some tools which capture your screen. What I'm trying to say you cannot do anything else. The QA has to just sit like this and have to watch the execution but when you execute them through like a CI tool, CI tool generally like you need to provide a runner. Runner is like a virtual machine, and CI tool will execute the test on this virtual machine. So we are giving the flexibility to the QA or anyone else who has triggered the automation test to do his or her own work. And automation tests, they are being triggered on another virtual machine. What happens like in today's world, we are giving AWS EC2 instance as a virtual machines. When the tests are executing, based on the type of EC2 instance you have given, automation test will be executed there. And the QA, they can do their own work. This is another advantage.

[00:10:07] Joe Colantonio Are there any tools where this wouldn't work? I would think sometimes companies get kind of sneaky with their licensing, like you can't run it in CI/CD. I don't know. Are there are any kind of limitations with this approach?

[00:10:19] Gaurav Mittal Limitations with this approach, with CI, I have always seen advantages. The limitation is you need to be thorough somewhere. You need to have knowledge. Another is what kind of automation test you want to execute. How to configure your automation? You need sometimes customize your automation framework accordingly to how you're going to set it up in CI/CD tool. Okay from your automation tool you can choose okay 10 tests you can execute but if you want to execute from this CI/CD tool you need to club them in some category. For that, you need to customize your automation framework. There is one limitation basically learning is involved that is something is there. And again, you are integrating you are integrating the CI/CD tool with the automation tests so that integration like as I mentioned like we are calling API. This CI/CD tool is going to invoke your automation framework. Maybe there are some firewall issues. GHA is going to invoke the GitHub repository only. So it's quite easy. There is no firewall constraints are there.

[00:11:29] Joe Colantonio Why wouldn't everyone do this? Is it hard to set up? Is it just people that don't know, hey, I could probably save doing it this way?

[00:11:38] Gaurav Mittal Yeah, so that's the point, like that's the advantage of CI, and that's what I want to propagate. Like license is something which every company, they are struggling in today's world. Everyone is running after the budget, cost cutting. But this is a very nice way. And there are free open source CI/CD tools available, like the GHA. CI/CD tools, they are never free because they require a runner machine. And that runner actually procures some cost, but I have found a solution for that also. For example, like automation engineers, they are using a license tool, which is quite heavy. Sometimes company like they provide a virtual machine like Windows 365 to the QA engineers that you install your automation tool on this machine. Okay, so that you can save the memory of your local machine for the regular, like manual testing or whatever you are doing. From GHA, you can configure Windows 365 machine as the runner. So what we are trying to do, we are doubling the usage of this Windows 365 Machine. We have installed the tool, automation tool on this Windows 365, as well as we are also using it as a runner. So ultimately GHA is totally free of cost. GHA is invoking the machine which is already available there in the organization. You are not using any AWS EC2 instance or launching any other virtual machine at any runtime. You are using which is all ready being provided to you.

[00:13:18] Joe Colantonio And once again, I believe this is all in the article that I'll have a link for down below. This also helps with AWS instances, I assume as well?

[00:13:26] Gaurav Mittal Yeah, it will help if you have already procured any AWS EC2 instance for 24 by 7, but there is some bandwidth, you are not using it, then why not use it? The whole idea here is the CI/CD tool requires a runner. Instead of launching a runner at runtime, asking AWS to provide a new EC2 Instance, first check within the organization if there is already existing virtual machine which you can use. It will save lot of money for the company. That is the main reason. I have put the calculation also like what I was able to save.

[00:14:02] Joe Colantonio What percentage? You don't have to give the amount, but what was the percentage of that you saved?

[00:14:07] Gaurav Mittal Percentage depends on the test, but like comparing to EC2 instance versus like using your virtual machine, existing machine, it's zero. Like with the existing machine it's already there. You are not paying anything. With EC2 Instance, like it depends on number of tests. Like you are executing 500 tests and like EC2 Instance was launched for four hours. For four hours based on the type of EC2 instance, you are paying suppose $50. Now it is zero for one execution. Monthly executions by 10 teams, you can calculate like it's a huge saving. It's an innovation actually. We are not invoking outside virtual machines. We are trying to use what is already being provided to us.

[00:14:52] Joe Colantonio Very cool. Once again, I guess I scanned through the article. I think you mentioned other things you can incorporate in your CI pipelines that people probably don't take advantage of. I think he mentioned SonarQube and CodeQL. Can you talk a little bit more about that?

[00:15:07] Gaurav Mittal Yeah, so I have actually built the architecture for data projects. And when the developers, they are pushing the code, what I found, they were directly like, once they were like, they're a team members from their local branch, they have pushed it to main and from main, they will directly pushing it to the production. This is not a good DevOps practice. There are several stages involved. The first stage is like static code analysis. So what happens like when you are writing the code, maybe the code is not in a good shape. There could be a data leakage or there are some security vulnerabilities, which the hackers, they can hack the code and they can find it. For example, like I have seen, it's very common when the developers, they are writing code, they want to print what is a token, JWT token value, or what is the password, or what the username, password. They are printing those secret features, which they should not be print out or that should not we go. Outside, I am saying because again, like when you are writing a web application, and there are some external tools which are being used for logging. Ultimately, you are passing this information to third party tools, which you should be worried of. How to catch this? For this, we have static code analysis. There are several tools available. SonarQube is there, CodeQL is there. CodeQL, is a very heavy tool, but it's so very nice tool for catching the security issues. The first stage should be static code analysis. It is making sure like, it is basically known as code smell. It is checking like your code is in good shape. It can be projected to the outside world with no issues. Once you are done this, the second step is like run the automation test. Because you are going to deploy a new feature. When you are deploying it, you have to make sure the existing functionality is not getting impacted, that is still working fine absolutely. And once your automation tests are being passed, then you should deploy it to production. This should be the ideal flow from QA to production.

[00:17:18] Joe Colantonio And I would assume with that flow, baking security into the DevOps pipelines without having to rely on, say, really time consuming security tests, you could save money kind of backhandingly because you're not relying on.

[00:17:32] Gaurav Mittal Yeah, even today like this code editor tools, which developers they are using, they are very smart. They are like using AI but what I have seen is still there are some security issues which they miss because these code editor tools, they guide you how to write code better. They cannot see if there are any injection errors. These tools, which are security tools, which I have spoken that you can implement in static code analysis part, these tools, they are expert in checking if there are any injection errors, there are no null pointer, man in the middle attack could be caused. They catch these kinds of things. Till the time an issue is not happening, everyone is happy, but sometime like crowd strike issue happen, everything come on a halt. It's better to secure your code before you push it to production.

[00:18:22] Joe Colantonio Absolutely. I guess, you work with a lot of different tools and a lot companies I know they have a lot of licenses for a lot of different tools they don't necessarily use or need or they have duplicates of everything. Did you have like a framework or process you recommend to people can go there to say, do I really need this tool? Is it worth having rather than having a nice to have? I guess that would save money in the long run if they were doing this type of evaluation all the time.

[00:18:49] Gaurav Mittal Yeah, so this is one thing even which we try to do but we were not successful because when we talk about different domains, there are tools which have expertise for this particular domain. For example, like GitHub Actions is very good. It works well for web data, but for Salesforce, there are some other tools like Gearset, Copado, which like developers they want to use for deploying the code. It's very hard to say like one tool or one framework is best, which can work for all different kinds of domains. But depending on the domain, like if you are in a data, like you are working on Python, it's better to use SQL Fluff or Rust as a static code analysis and uses unit testing. Unit testing is one area which I have seen like developers, they don't write it but it's actually very useful. These are some common practices, I would say, which everyone should use. Common framework is actually quite tough to say. Common framework or common tools. Because different domains have their different needs.

[00:19:59] Joe Colantonio All right. So you mentioned AI briefly. What's the role of AI? Can you use AI now? I know a lot of tools like Kubernetes. The solutions that built on top of that use AI to limit resources, or be able to read when it should be up and down. Do you see those type of AI solutions helping with bringing costs down?

[00:20:17] Gaurav Mittal Yes, I can cite you a couple of examples. AI is useful if we talk in terms of soft savings, not hard savings. So I give you a number of examples, like in my projects, any data project, they are very heavily relied on SQL. And what I did, I created one agent, SQL Optimizer. And it has given me wonderful results. For some SQLs, existing SQL was taking, Like suppose 25 seconds. the GenAI-generated SQL was taking seven seconds. It's like more than 50% of performance improvement. On top of it, it will also guide you like why GenAI is making that change. So for your learning, if you want to enhance your career growth, like this is a good usage of GenAI. The problem is it comes with its pros and cons. There are a couple of problems. One, GenAI has a limitation on token like for example it cannot work more than 500 600 lines of SQL. And secondly, it's AI okay AI has a particular confidence level of 80% or 90% but not 100%. Once existing SQL was giving you 100 rows the new SQL is giving you 90 rows. Okay, it completely changed the like instead of improvement it changed the joins or anything. So you need to be careful. What I'm trying to say, AI is good to learn and check the performance, but you cannot blindly use it to deploy it into the production. Another example I talked about are these tools, security scan tools, like CodeQL and SonarQube are there. With GenAI, also, you have written your code. You can ask, can you provide me any security scans on this? And it can give you the output. Like if there is some hacking, hackers can crack the codes and then how you can write it in a better way. You can modify your code. These are some couple of examples. AI comes into the picture. And one big example is whether it's automation or it's a development, people are using it to analyze the production logs, like where the error is occurring. It helps you to pinpoint the location from which code line the issue might be occurring. You're basically trying to save some time for you. It's all false and the soft savings.

[00:22:46] Joe Colantonio Yeah, is that leveraging OpenTelemetry with the AI to be able to let you know that it failed in production and what line of code it came from?

[00:22:55] Gaurav Mittal Production logs, like whatever tools you guys, like teams are using, for example, like in Kibana or anywhere, or like in your local, and you want to pinpoint like, okay, from which function this error could be pointing of? AI can help you in text extraction, like, Okay, this error, could be because of this. There are thousands of files which are getting generated on daily basis, it's difficult to manually fast. But with AI, you can do it a little faster and it can give you some hint. Like, okay, maybe this part is failing.

[00:23:31] Joe Colantonio Are there any open source AI type of tooling you can use in your CI/CD pipelines or is this these all probably paid vendor based solutions?

[00:23:40] Gaurav Mittal No, no, no. There are several open source AI I have used. I have not used in my DevOps pipeline. But all the work what I have done in AI, the models I've used, they all are open source. For example, I have spoken about text extraction. So NLP is there, Natural Language Processing. And it's open source, we can write a model around it. We can figure it out like, OK. We can do the text analysis, we can do sentiment analysis. Then for image classification, like, okay, what kind of image it is, we can use deep learning ML models like Keras, TensorFlow, these are open libraries. We can use them. But I have not used them yet in the DevOps pipeline.

[00:24:26] Joe Colantonio All right, Gaurav, before we go, is there one piece of actual advice you can give to someone to help them with their DevOps automation testing efforts? And what's the best way to find or contact you?

[00:24:34] Gaurav Mittal Yeah, so my solution will be like right now, like penetration testing is there, which is a good area. And in the DevOps, we should implement it by using this security tools. As a QA, we all love to find defects. And this is one nice area, niche area, I would say I have found like where I'm shining and giving back to the my company, like, okay, yeah, these are the issues. We should take a look over it. For your learning, for your visibility, try to use security scans in the pipeline. This will be very helpful for you as well as for the company. And yes, you can reach out to me at my email address, gauravmittal1995@gmail.com.

[00:25:14] Thanks again for your automation awesomeness. The links of everything we value we covered in this episode. Head in over to testguild.com/a541. And if the show has helped you in any way, why not rate it and review it in iTunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe, my mission is to help you succeed with creating end-to-end, full-stack automation awesomeness. As always, test everything and keep the good. Cheers.

[00:25:48] Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation, testing, and DevOps. If you haven't yet, join our vibrant community at TestGuild.com where you become part of our elite circle driving innovation, software testing, and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real world challenges, we're excited to collaborate. Visit TestGuild.info to explore how we can create transformative experiences together. Let's push the boundaries of what we can achieve.

[00:26:31] Oh, the Test Guild Automation Testing podcast. With lutes and lyres, the bards began their song. A tune of knowledge, a melody of code. Through the air it spread, like wildfire through the land. Guiding testers, showing them the secrets to behold.