Automation – First DevOps Focus with Kedar Kulkarni

[00:00:00] Joe Colantonio Get ready to discover some of the most actionable DevOps techniques and tooling, including performance and reliability for some of the world's smartest engineers. Hey, I'm Joe Colantonio, host of the DevOps Toolchain Podcast and my goal is to help you create DevOps toolchain awesomeness.

[00:00:19] Joe Colantonio Hey, want to know more about automation first DevOps and more? Well, you're in luck because today joining us, we have Kedar, who is a DevOps and cloud infrastructure expert, and has deep expertise in CI/CD, Kubernetes, and automation for us DevOps. He's written extensively for opensource.com and Red Hat and also co-authored a popular ebook on IT automation and created the ATCasC framework that's now part of Red Hat's automation stack. He's also known for his unique approach to infrastructure test automation, which has shaped how teams think about testing infrastructure like code. We'll be diving into GitOps, open source tooling, and what it really takes to build DevOps frameworks that matter. You don't want to miss it. Check it out.

[00:01:02] Joe Colantonio Hey, before we get into this episode, I want to quickly talk about the silent killer of most DevOps efforts. That is poor user experience. If your app is slow, it's worse than your typical bug. It's frustrating. And in my experience, and many others I talked to on this podcast, frustrated users don't last long, but since slow performance is a sudden, it's hard for standard error monitoring tools to catch. That's why I really dig SmartBear is Insight Hub. It's an all in one observability solution that offers front end performance monitoring and distributed tracing. Your developers can easily detect, fix, and prevent performance bottlenecks before it affects your users. Sounds cool, right? Don't rely anymore on frustrated user feedback, but, I always say try it for yourself. Go to smartbear.com or use our special link down below and try it for free. No credit card required.

[00:01:58] Joe Colantonio Hey Kedar, welcome to The Guild.

[00:02:02] Kedar Kulkarni I'm really happy to be here. Thanks Joe.

[00:02:05] Joe Colantonio Awesome to have you really excited. I guess before we dive in, I'm always curious to know, how did you get into DevOps?

[00:02:11] Kedar Kulkarni So it's an interesting story actually. And so basically when I was in the school, I was studying for becoming a computer engineer and everybody was running behind making software, writing apps, web apps, everything. But my final year project in my undergrad was in OpenStack. OpenStack is one of the opensource cloud projects that was started by NASA jointly with Rackspace, I think back in 2020. And from 2010, it has grown quite a bit. It's used in large telcos and what not nowadays for all the 5G or empowerment. And I got into that for my project and I learned, Oh, there is a whole lot more between the software that people write and how it actually runs. And that's what got me into system's space and that eventually drove me into DevOps through my career.

[00:03:03] Joe Colantonio Nice, what I also like is you're an active contributor to open source. This is the first time I actually heard about, hopefully I'm saying it right ATCasC, just curious to know what is ATCasC and how did you get involved in it?

[00:03:15] Kedar Kulkarni ATCasC is a name that I came up with. It was inspired from jcasc, which is Jenkins configuration as code. ATCasC is Ansible tower configuration as a code. Currently, it is actually known by different names. I think it's called, so it was called Ansible Tower when I started, then it started being called as Ansible controller or P, officially. And so the way we got started with that is, when you use Ansible Tower, or controller, now that it's known as controller, you need to create job templates and projects and credentials and a bunch of other things in the UI. And as an administrator, you're constantly creating these things for your customers to consume those to make that automation easily accessible. But for you to click around in the U.I., it is very cumbersome if you're managing a large set of automation jobs. I was like, why not use APIs? Why not use Ansible to automate Ansible itself? Ansible team had some APIs they exposed as Ansible modules, which I used to create Ansible playbooks that could automate creation of all the resources in the Ansible UI in configuration as codeway. And if your configuration is code, then you have all the benefits of version control and GitOps and everything. That's how the project started. And currently it is hosted in Ansible Galaxy, which is the open source repository of all the Ansible related collections, roles, specific artifacts that you could download, which is very similar to, let's say, Docker Hub for Docker images. And this collection that we created ATCasC got renamed a few times throughout the years, but as of today, it has collectively more than 450k downloads. And not only it has downloads, but it became part of Red Hat's product portfolio and it is now offered through console or cloud.redhat.com to paid customers as well. That's the journey of ATCasC going from a small pet project that I started from my personal pain in the systems to getting it out to the customers, all of the customers.

[00:05:22] Joe Colantonio Maybe I'm too much of a capitalist, so it's being offered to pay people now. So do you get anything from that? Or like, it's all completely.

[00:05:29] Kedar Kulkarni It's completely free and open source and the paid version that Red Hat derived from this open source version they probably are making money as part of the offerings but I have been an engineer at Red Hat right during the time I invented this so basically anything that I invented is Red Hat's property and we basically abide by that.

[00:05:52] Joe Colantonio Okay. All right. All right. Fair enough. So I mean, that's a huge adoption rate. So I know a lot of times people create things and it just it's on GitHub has one star, not many downloads. What does it take to build maybe an open source tool that DevOps team's really going to rely on? Obviously, this is probably a need you saw, but like, are there any other ingredients you see to make a successful open source project?

[00:06:13] Kedar Kulkarni I think successful open source projects are those projects where you actually have felt that pain yourself, like whatever problem you're solving. And that problem has to be a niche area like Ansible is one of the top tools in the industry for DevOps automation. It has more than 30% market share, I think, which is way more than anything else like Chef, Puppet, TerraForm, etc. So Ansible has a large community and anything that started as an open source project, if you like take it to the community and share it, evangelize it, that can actually help you get more eyes on your tool, whatever you're developing on. That's what I actually did. I picked an issue area. I was suffering some pain. I saw the problem, I solved it, and I took that to Ansible community and said that, Hey, this is what I'm building. And sure enough there were other people who had started to feel that pain too and there were two other contributors who also came forward and they said they started their own version of that some time ago and so all three of us came together we decided to open source each of our repositories and then combine all three them into one single repository. And then host it on Ansible Galaxy and so we went from there. That's how a project like this could come about to be I think People need to understand when they're talking about open source, it's not about you as an individual. It's about the community, and you have to walk with the community to succeed.

[00:07:42] Joe Colantonio Now, that's a great point, but I do see people that start these projects and they're the only ones really contributing or responsible for it. Obviously, with this many downloads, I assume you have a bigger community actually contributing to where you're not just, you could still contribute. I guess the question and what do you do to get other people to contribute? Cause that's also a barrier. I see a lot of teams and a lot of open source projects to deal with.

[00:08:03] Kedar Kulkarni I have had my fair share of projects which have no stars or contributors. It's very common. Here to just keep going through it. There is no magic bullet, unfortunately, to get the contributors. I think the only thing that you would really do is evangelizing your project and see more and more opportunities to talk about it. More you talk about it, more likely that somebody in the field is going to find that, oh, this is important. And that somebody could be important enough that they could push to get more contributors. Again, this comes down to like that community building aspect and evangelizing to make your project successful. And if you look at my GitHub, I have like 170 repositories. Not all of them are really important. A lot of them are like just toy projects. But then I also have to pick which one is actually important and take that one forward to the community. That way you can actually build credibility that you're bringing some nice ideas and not bringing every small that you would like dreaming of.

[00:09:03] Joe Colantonio All right, so which one are you currently working on that's important that you think should get some more love?

[00:09:07] Kedar Kulkarni I think currently, my open source contribution, active contribution is limited due to my employment. But the project that I mentioned, the ATCasC, which has been renamed and gotten into other repositories is most active one, I think. And it is benefiting people, hundreds of saved hours and maybe millions of dollars and savings because think about it this way, if you create some infrastructure in the UI of Ansible and if you mess up something, if somebody else has admin privileges, it's more likely that you're gonna have a team who has admin privilege and somebody mess up something, and they don't remember what they messed up and you're blocking thousands of deployments. You're blocking some important bug fix to go out to the customers. Who knows what, you are blocking out some critical workloads that need to be running, that need be scaled up and they cannot be because your automation is broken. And this kind of repository that helps you automate the automation, it's like meta level automation really, it helps you save those hours and money and frustration and that's the whole reason why I feel like that repository is the most important one in my career to highlight from DevOps automation standpoint.

[00:10:18] Joe Colantonio Nice. Besides this, I know you're also passionate in general about infrastructure as code. I mean, I probably should have let off with this for people that don't know what is infrastructure as code?

[00:10:28] Kedar Kulkarni That's a really great question. So infrastructure as code, when I was in my early career and I started to realizing that infrastructure is ever-expanding and if I keep clicking on the UI or using CLIs to bring up new VMs, new resources, new networking configurations, whatnot, I'm gonna grow out of myself quickly. I'm going to forget everything. 3 days later, if you ask me what was the IP address I set or whatever else for something, I'm gonna forget that. And that's the time when I got introduced more to Ansible and Terraform and all the other tools that help you write your infrastructure, configure your infrastructure. And as code, meaning that this is all the code that you're writing in terms of Terraform or Ansible, whatever you're using, or any other tools out there, when you write this code lives in your version control ideally. And that code helps you sleep peacefully every night knowing that if something burns down tomorrow in a data center where you have deployed something, you will be able to bring that back up within some reasonable amount of time without having to pull your hair. And a funny story regarding infrastructure as code is like one of the articles that I authored early on for that enables this admin blog is infrastructure as a code pros and cons. When I was in one of them new teams where our team lead did not know very well. He asked me, Hey, KK, I think what do you think about infrastructure as code? We should start using that. And I was like, totally. I am a 100%t for that. And he was like yeah, we need to convince our management about it. I said sure. Tell me how I can help. And then he went on the internet. He just searched for the terms, IAC pros and cons. And then, he found some articles. He found a great information in that article. And he sent that article to me and said, we should use these. These are the key points to use with our management to convince them that we should go infrastructure as code route. And two minutes later, he pings me again. Damn it, KK, that's your article. He found that article organically on Google and it still to date ranks in one of the top results on Google or AI summaries. I feel very fortunate to be contributing to such kind of early inception of IAC.

[00:12:46] Joe Colantonio Very cool. So it was early. Any new pros and cons you see? I know this is infrastructure code isn't a newer term. It's been around for a bit now. Now with your experience after you wrote there, are there any new pros and con that you would add to it or update?

[00:13:00] Kedar Kulkarni Actually, no, I think maybe one thing that I really feel is the I think I did not use the term back then in my article, but if I could go back and update the article, I would add bit rot is real, meaning if you write infrastructure as code, you exercise it once you deploy your infrastructure, three months later, six months later, you want to try and read on the same thing. Something may change in the ecosystem and can break something somewhere. And that is the maintenance and upkeep, which is one of the cons that I had noted in my article originally as well, but I did not use the term Bit rot, which I learned recently from one of my colleagues in the open source community.

[00:13:42] Joe Colantonio Should every company that does DevOps be treating infrastructure as code, or is that are there some cases where you're probably like, probably not a good fit for you?

[00:13:52] Kedar Kulkarni I have talked about this in my ebook like one of the myths is that sometimes people think may not be automatable, something is not possible to be automated or may not be worth automating because you're doing it only once and you're not gonna do it again this is one time thing. Both of those in my opinion are myths because most of the things if they are not fully automatable you can at least partially automate to some extent to reduce the amount of manual work or toil that you have to put in to maintain that thing or build that thing, right? And then, when somebody says it's not worth automating and it's one time thing, I have seen in my career that statement coming to me from various people time to time that hey, we just need to do this once, so just do it manually. And guess what? They come back again and they say, oh, we need to it slightly differently, the same thing again, but slightly differently. Can we do that again? I'm like, see we should have automated this thing.

[00:14:47] Joe Colantonio Do you recommend then developers build the automation in mind? Because sometimes they'll build something like I worked with the company that said we're going to CI/CD, but they didn't make a build that was deployable. You had to do manual steps. And I was like, this doesn't make any sense because I didn't think about it. So any tips on how to make your code more automatable? I don't know if that makes sense.

[00:15:07] Kedar Kulkarni I think when you start thinking about deploying the code, as I said earlier in my introduction, between the software being written and deployed and being used by somebody, there's a lot of things that need to happen. There's systems, there is automation. And for any company who is trying to work on any new project, they need to put that mindset first like how are we going to deploy that? And I think that really starts from the point when let's say you have a team of Developers, they are working on some projects. Each of them need to set up this infrastructure on their local system to some extent to actually develop that project. And then from there, it needs to be promoted to different environments, which are not your laptop, but some other shared infrastructure. And so at every step of the way, if you are doing things manually, you're going to just end up wasting a lot of time and money and the resources, and it's not good for anybody's mental peace, any company's financial health, or progress and et cetera. I really think that when you're developing, you should start with, let's say maybe use a Docker container or some kind of container to create a common shared knowledge of what all needs to be added for this software to actually run on your system. That Docker container, that image, that file, container file can actually serve as a reference point for let's say your DevOps persons to know exactly what it all takes to run this software, and they can take the automation effort from there forward to automate for the large scale environments. Maybe that's one thing that I recommend like try to create really good code that can help you spin up the environment quickly development environment quickly because that is the starting point for actually getting that app to the production.

[00:16:49] Joe Colantonio Nice. We have the starting point. Is there any other steps to create an infrastructure test pipeline? They have a DevOps pipeline, but say they want to include infrastructure test as well into the pipeline. Are there any steps along the way that they would do?

[00:17:01] Kedar Kulkarni Right, so one of the things I did for my ATCasC automation was it was automating the automated platform. AnsibleTower is a platform that lets consumers, like other team members who are not DevOps engineers, to quickly go and provision a resource or deploy something, which is the CI/CD platform, let's say, for the sake of this discussion. But if I'm modifying the code that is in ATCasC to modify an existing job template or project or credential. Now that is going to affect the actual production jobs that are supposed to be running. That is that gap that nobody was thinking back then when I started that project. I created a pipeline where I take the code changes made to the infrastructure code much like software code and I had a test pipeline for infrastructure code. I would actually deploy an ephemeral ansible tower instance that would basically have all the code that's running in the production plus your changes and be able to test that there. And I created another project which is also open source called Genealogist, which had ability to figure out from which job template, which playbook needs to be changed or from which play book, which job templates will be affected. And there's a whole list of things in between that are cascading failures if you don't pay attention to. And so Genealogists was able to figure it out. Like if you make change in test.ymlplaybook, then all of the things that need to be tested for that, it was able to derive that just by looking at the code. And this was way before all the AI stuff, right? So this tool was able to figure out what needs to be tested in infrastructure, deploy an ephemeral infrastructure, run the jobs for you and tell you that if you merge this, is this gonna go good for you or not? And that is basically an SDLC kind of like software development lifecycle, but for infrastructure code.

[00:18:55] Joe Colantonio Where are you learning all this? You're just messing around and saying, can I try to do this? Like, this seems like a lot of stuff you've done before. Like you said, AI is out now. There's a lot things that make things quicker and easier for developers. Like, where are you getting these ideas to try this stuff?

[00:19:09] Kedar Kulkarni I think the ideas I get are because of my diverse background in terms of what I've learned in my school. I was learning for computer engineering, doing a lot of software. Then I got into OpenStack. Then OpenStack got me to Red Hat. And in between, I took data science courses at my masters. So at Red Hat, I joined as a developer. Then I did testing. Then I read DevOps and systems administration. And at VM, where I was doing some development and automation, then there was some performance and skill benchmarking I did for Kubernetes. I kept doing different sorts of things for some reason in my career. And that maybe gave me that more well-rounded approach for thinking. That's why I was able to make that connection. Like, hey, I'm writing code, be it infrastructure code or software code. It is code. I can use the same principles I use for SDLC for software development lifecycle. I write software. I have some unit tests, I deploy some ephemeral environment, test it, make sure everything is working, review the PR and merge it. Why can't I do the same thing for infrastructure? It's code in the end and that's how it really came to me. There's nothing more to it.

[00:20:19] Joe Colantonio All right, very cool. You did mention Kubernetes. I know a lot of people deal with performance issues. Any insights around that? I think I have in my notes, GitOps in Kubernetes, maybe some views around that area.

[00:20:31] Kedar Kulkarni When I was doing the Kubernetes performance and scale benchmarking at Red Hat, it was for the Kubernetes networking plugins. And we would be running large scale workloads, which has thousands of nodes sometimes, thousands of pods running, hundreds of thousands of different kinds of objects just to overwhelm the Kubernetes controllers. And then one of the things we often found was we just need to make sure that we understand what workload we are running and what kind of resource utilization you're going to actually need for that. If you get that sizing correctly, you're more likely than not to be safe with all the performance issues. And that's where everything boils down to. So throw more resources is not actually the solution, but throwing less resources and running production with less resources can also bite you. You need to try and find that balance and that balance can only come if you are running your, let's say if you have a software that you're deploying in production, you need to run whatever kind of operations you're running against that software at large scale. You need to design some tests for that and see what kind of limits it is hitting and then make an informed decision for the underlying resources. Then only you can actually succeed, getting the highest performance without seeing any bugs or issues.

[00:21:53] Joe Colantonio So for someone's listening, what's the role that usually does the Kubernetes performance testing and the infrastructure as code? Are they separate roles or is this something that you would recommend any DevOps engineer or software tester would know something about?

[00:22:07] Kedar Kulkarni Anything that is not software engineering like writing code for making software is in my opinion honestly is a giant blur of roles because it is called DevOps, systems administrators, SREs, test automation engineers, software engineering in test or whatnot and really even CI/CD or release engineers. Everybody's actually doing everybody else's job to some extent. And so if you start exploring there are different career paths you can take in there and really I think to answer your question really well, there is no right answer to see which is the correct path for you. You have to just explore and arrive at a point where you feel most comfortable. In my career, I went from developer to QA to system administrator to DevOps to developer again and to DevOps again. And all of this journey, I realized that all these years that DevOps is the place where I feel most comfortable because every day is a new challenge. Every day, I'm supporting a different app maybe or a different kind of project that needs different resources in the cloud or on-prem or wherever. And all of this It keeps me feeling afresh about my job versus if I'm writing just software for one project. Software projects tend to be like very long, like it can take several years before you release the first version of your software. So you're coming in day in day out and writing the code to make one feature work for months at a time. That's great, and some people love it but for me it was not the path I really wanted to stay on and that's how I arrived in this area of the career.

[00:23:49] Joe Colantonio Excellent. Speaking of career, I know a lot of people worried about AI. You mentioned you did a lot of the stuff before AI, any views on AI, especially as it applies to DevOps. Is it a career killer, a career enhancer, hyped, under hyped, properly hyped?

[00:24:05] Kedar Kulkarni For me, it's just too soon to say. I don't have much inputs on that. I just want to sit around and see how things are playing out.

[00:24:15] Joe Colantonio Okay. Fair enough. What's the latest and greatest thing you're messing around with, then? If someone's like, you seem to always be doing something different, is there anything new or any new technology that you're excited about that may not be AI related?

[00:24:26] Kedar Kulkarni I'm always intrigued by all of the GitOps that goes behind Kubernetes, like the Argo CD and Flux CD. I know it's not new technology per se, but there's a lot of opportunities to explore it and use it to your personal gains to make your day job more easy to deal with a lot Kubernetes namespaces or projects. And I think that's one of the things that I want to spend maybe a little bit more time on in my pastime to see how these technologies are evolving with the time.

[00:24:57] Joe Colantonio All right, so speaking a little bit more about ATCasC, I think I saw a video on YouTube where you presented at a conference, and I guess it takes a little deeper dive. Is there anything else that you want to talk about when it pertains to ATCasC?

[00:25:09] Kedar Kulkarni So I think if a user wants to learn about how to use ATCasC in the context of Ansible, I have made a presentation in DevConf US 2020. It's up on YouTube. And one of the cool things I noticed after I presented ATCasC was there is similar thing that Puppet does in their offerings called CD4PE, I think continuous delivery for Puppets enterprise. If you hear that concept and if you hear my presentation, you'll probably be able to draw very striking parallels between what I presented and what they're currently offering. I want to argue that I think industry as a whole has caught up to the concept that, hey, we can actually test our infrastructure code before we deploy that infrastructure code into the production infrastructure, which is a great achievement for us as an industry, I feel like.

[00:26:03] Joe Colantonio All right, Ansible versus Puppet. Why Ansible?

[00:26:06] Kedar Kulkarni I think it was just easier for me to learn, really. I started learning Ansible when I was in my master's. I was spinning up Hadoop clusters because my master was in data science, and I was using somebody else's AWS account resources because I did not have money to have my own AWS. Sit on my professor's account. I did want to keep my Hadoops clusters running 24 by 7 because I was maybe using it 2 hours every day. So I was like, why should I not automate this? And I came across Ansible as one of the tools that I could use, it was very easy to pick up and maybe that's the first bug that bit me for Ansible and I kept going at it. I tried looking at Puppet and Chef, it just felt a little bit overwhelming to me at that time and then Ansible just became my second nature.

[00:26:53] Joe Colantonio All right Kedar, before we go, is it one piece of actionable advice you can give to someone to help them with their DevOps efforts and what's the best way to find or contact you?

[00:27:02] Kedar Kulkarni One piece of advice would be automate and automate, really. You really can not automate yourself out of the job. You just get more productive if you're automating your job, and you can actually make impressive things in your career and maybe get that promotion that you wanted or dream role you wanted just by automating and learning how to automate your work, and making yourself free for higher things in your day-to-day job. that's how you get more visibility, more work, more meaningful work, and maybe your dream job or promotion that you're looking for. That's my advice. You're not going to automate yourself out of the job. That's something that I'm really firm about. And then to find me, you can always contact me on LinkedIn. I think my LinkedIn is open to public and it's just hit me up there. I'm mostly active. And maybe Joe, if you want, I could share my email address then people can contact me that way as well.

[00:28:00] All right, before we wrap it up, remember, frustrated users quit apps. Don't rely on bad app store reviews. Use SmartBear's Insight Hub to catch, fix, and prevent performance bottlenecks and crashes from affecting your users. Go to SmartBear.com or use the link down below, and try for free for 14 days, no credit card required.

[00:28:21] And for links of everything in value we've covered in this DevOps ToolChain show, head on over to testguild.com/p182. So that's it for this episode of the DevOps ToolChain Show. I'm Joe, my mission is to help you succeed in creating end-to-end full-stack DevOps ToolChain awesomeness. As always, test everything and keep the good. Cheers.

[00:28:44] Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation, testing, and DevOps. If you haven't yet, join our vibrant community at TestGuild.com where you become part of our elite circle driving innovation, software testing, and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real world challenges, we're excited to collaborate. Visit TestGuild.info to explore how we can create transformative experiences together. Let's push the boundaries of what we can achieve.

[00:29:27] Oh, the Test Guild Automation Testing podcast. With lutes and lyres, the bards began their song. A tune of knowledge, a melody of code. Through the air it spread, like wildfire through the land. Guiding testers, showing them the secrets to behold.