About This Episode:
Automation of your functional tests is important, but many testers stop there. In today's continuous testing world you need to automate many aspects of your CI/CD pipeline as well. In this episode, Mohamed Labouardy, author of the book Pipeline as Code, will share tips on how to succeed with the automation of your continuous delivery efforts. Discover tips for using Jenkins, Kubernetes, Terraform, and much, much more. Listen up!
The Test Guild Automation Podcast is sponsored by the fantastic folks at Sauce Labs. Try it for free today!
About Mohamed Labouardy
Mohamed Labouardy is the CTO and co-founder of Crew.work, and a DevSecOps evangelist. He is the founder of Komiser.io, author of “Hands-on Serverless Applications with Golang” and “Pipeline as Code”, open-source contributor, and regular conference speaker.
Connect with Mohamed Labouardy
Want a FREE copy of Mohamed's book?
- Check out the book –>[here] GET 40% off using the code podguild19
- ***Want a FREE pdf copy of the book Pipeline as Code? Leave a comment below and Mohamed will choose the best 5.***
Joe [00:01:16] Hi Mohammed! Welcome to the Guild.
Mohammed [00:01:23] Hi Joe! Thanks for having me.
Joe [00:01:24] Awesome to have you. So Mohamed before we get into it is there anything I missed in your bio that you want the Guild to know more about?
Mohammed [00:01:29] No, I think you have covered everything.
Joe [00:01:31] Sweet. So I really like this book. You actually start off the book. If you've never practiced continuous integration and continuous delivery principles as well as DevOps practitioners, you can learn from this book. So it really starts almost from ground zero and ramps people up all the way to having a full-blown pipeline with all these cloud integration providers. So I really love that approach. So I thought we'd start the podcast off with this approach and talk about, you know, what is CI/CD? I say this term all the time on the podcast. Not sure I've ever really defined there what actually it is.
Mohammed [00:02:01] So basically CI/CD, it starts with the CI, which basically stands for continuous integration. So basically it's the idea of having a shared centralized repository and which are the changes and features of your applications will go through a pipeline before integrating them into this repository. So a classic CI pipeline will be triggered each time you push something to this repository. So for instance, if you push and you commit to this repository which will run the unit test, the quality test, the security, etc., and it will build the artifacts or compile the source code of your application and then maybe it will push this result to a remote repository. So, for instance, if you are using Docker, it might build in the docker image and store it on a private registry. If you are working with Java, for example, it might build JAR and store it on a Maven repository, etc.. So this is a simple example of continuous integration. As for continuous deployments, it's basically an extension of continuous integration. So all the changes that will go to continuous integration will be deployed automatically to our station or our preproduction environment. And finally, for continuous delivery, it's basically similar to continuous deployment. However, the deployment is not automatic. It requires a human intervention or a business validation. So, for instance, you will need to wait for acceptance tests or a business validation from the business department, etc. before deploying the release to production. So this is the definition of CI/CD pipelines.
Joe [00:03:39] Nice, so whenever you talk about CI/CD pipelines, what is a pipeline then? I guess you mentioned different phases. Is it phases that happen during the software development lifecycle as you're merging code into main or master or whatever?
Mohammed [00:03:52] So basically our pipeline is a set of stages. These stages can be executed sequentially on a parallel. So basically a stage is a set of commands or steps that you will run locally on your machine just with the CI/CD and with the tools that are emergent on the market today. You will run these steps on our remote server. So, for instance, you can use something like Jenkins or a SaaS platform like (unintelligible) to run your steps. So basically a stage is just a set of steps that you will run locally, for instance, to compile the source code or to run unit test, etc.. So you just put these steps on something, what we call pipeline or a Jenkins file to automate the deployment end-to-end integration of the application so it can be integrated every time you push something to the remote repository in which your source code is stored.
Joe [00:04:46] So why would teams want this? I just assume everyone is doing this, but for teams that maybe aren't, why would someone want to actually use CI/CD or create a pipeline in this type of infrastructure?
Mohammed [00:04:58] Okay, so there is a lot of reason why people need to embrace the CI/CD approach. One example of that is it will help you detect anomalies at an earlier stage, which will help you reduce the risk of releasing something which contains a lot of bugs that might impact your business. And also it will also minimize the technical depth because you will have functional test, unit test, etc. which will help you improve your code quality and reliability and also it will give you a better visibility into your project status and health. So we'll have a better idea about the product roadmap and the features that you are building. And also it will help you have some kind of preproduction environment in which you can do all your tests and also produce bugs and do some AB testing before releasing new features to the user. So this is an example of why people need to embrace CI/CD approaches.
Joe [00:05:57] You cover all this in your very first chapter. So there are a few more things I want to pull out of Chapter one before we go to the next chapter. You also talk about cloud-native apps, so I'm curious to know why you decided to talk about cloud-native apps. Are they different from normal apps? Is this a trend you see more people going towards now as well that they need to be ready for?
Mohammed [00:06:14] Yeah, I think the cloud-native application is something that has started to be used incessantly in the past two years, especially with the adoption of cloud. So you have a lot of people migrating from on premise to cloud, especially with tools like coverage test service, etc. So with this new architecture inspires them of building applications. It changes also the way people doing CI/CD, because when you are playing with the cloud, so you have a lot of services that you need to integrate with. So people also need to adapt their process and their pipelines to be able to build their cloud-native applications. So this is the idea why I'm covering a lot of chapters regarding cloud-native applications because there is a lot of people today who are using Kubernetes service applications like lambda functions, cloud function, etc. So it's something which is very trendy today. And there is not a lot of resources that we can find on the Internet about this topic. So that's why I have dedicated multiple chapters on how you can build a CI/CD pipeline from scratch for these kinds of applications.
Joe [00:07:29] Great. And along those ways, along those lines, you also talked about Cervalis. That's a new kind of trend as well. What kind of things do you have to handle differently with CI/CD when you're dealing with Cervalis?
[00:07:38] I think the first challenge is the complexity of pipelines, because, for instance, when you are dealing with microservices, most of the time you are dealing with one pipeline because you have a pipeline per service. However, you are dealing with a service architecture, you deal with what we call nanoservices. So you split your microservices into multiple functions. So now, instead of integrating one package, you need to integrate multiple functions. So there is a lot of synchronization to be done and a lot of code that will be shared on these different pipelines. So you need to do a lot of factoring. And also if you have an API that has multiple functions behind it, you will end up with a pipeline with a lot of stages, with a lot of branches. And so it will be difficult to maintain, also difficult to iterate on it in the long run. So these are some examples of challenges that people might face when they start building a CI/CD pipeline for service applications.
Joe [00:08:44] So at what point in the (unintelligible) lifecycle, should people start thinking about CI/CD? Should be from the beginning? So I'm just thinking of a company I worked for a greenfield application but they didn't make the application deployable automatically. So you had to manually do it. So it didn't make any sense from CI/CD to be able to actually automate the built. So at what point should I be thinking about CI/CD as a developing code?
Joe [00:13:46] Great. So, you know, you talked about roadblocks. I like how you said it should be helping the team, not making them frustrated. So how do you see most companies implementing it then? Is there someone in charge of CI/CD or is it like once again, the developer is responsible for everything? Does it depend?
Mohammed [00:14:00] It depends. But I think most of the time you have some kind of DevOps team which are handling their CI/CD. In my current company and also in my previous position, what I have tried to do is to maintain some kind of shared responsibility between developers and DevOps, because that's one of the purposes of DevOps for me is a setup practices and also it's a mindset shift of the entire organization. So if you want to have some kind of successful DevOps culture, I think everyone should be involved with this journey. And I think one of the things that you can do is to set up some kind of shared responsibility when it comes to setting up CI/CD pipelines, especially today with tools like Docker, infrastructure as code, etc.. So I think everyone can be able to get started with anything related to sysops and anything related to the cloud infrastructure, etc.. So I don't think developers now have any reason to say that it's not my responsibility to build the application or to deploy it, etc.. So I think now with just a sample Docker file you can build your image, you can test it locally or on a cluster like Kubernetes or swirl. So but to do so, you need some kind of knowledge sharing because the beginning developers don't have a lot of knowledge about what is CI/CD, how DevOps works, and all the CI services that exist today. And what I have tried to do in my previous position is start some kind of workshop on which I try to explain a particular topic, like, for instance, Pipeline as Code or how you can write your Docker file or Jenkins file. So once I started doing that, people started to get interested to learn more. And from that, I started to create some kind of templates and these templates developers started to use every time there is a new project or a new feature that you are working on, and then you just need to clone these templates and they can create their job on Jenkins or whatever the CI server without the need, without asking DevOps team for further help to create a CI/CD pipeline. It was very nice in the beginning and also it has a lot of benefits because, in the long run, developers started challenging the DevOps team on the solution. It wasn't just people that are asking for their help, but they were also people that give us good ideas about how we can improve the CI/CD pipeline. They have suggested new tools that we can integrate to improve the quality or to make the process faster, etc. And I think by doing that and by having this kind of workshop within the organization can help everyone and the team either on the development side or the op side to work together to have this kind of shared responsibility when it comes to maintaining CI/CD pipelines in general.
Joe [00:16:55] I love the idea of knowledge sharing. And I guess this book almost you can give your developers this because as I mentioned, it starts from beginning all the way to more advanced topics. And so far, we touched on some of the more of beginning chapters, what is CI/CD, pipeline as code with Jenkins, defining a Jenkins architecture I guess you go a little more into that, but there are a few things that I want to go over that I haven't heard of before. But maybe just because I'm also a newbie kind of. And that is Packer. You have a whole chapter on Packer. I'm just curious to know what is Packer and how does it help people?
Joe [00:19:46] So just to me, it almost sounds like Maven with the POM file. We have a POM file that has all the dependencies and automatically catch up and running. But for images for machines. Nice.
Mohammed [00:19:55] Yeah.
Joe [00:19:55] Cool. So another one I've been hearing a lot about is Terraform. I mean, a mastermind with someone that was having a hard time with their customers, getting them to build the application. And it was, “Oh, I just use Terraform.” They spent like two weeks creating some crazy Terraform script that does the whole build form. So I guess my question is, what is Terraform and how does it help people with CI/CD?
Mohammed [00:20:15] So basically Terraform is a tool that allows you to describe your infrastructure in template files. So it's an application of what we call infrastructure as a code approach. So basically, instead of deploying your servers manually by going to the console or by using the CLI or the API, I think in the beginning it will be easier because you will start with the few servers. But imagine you have a complex architecture in which you will have a VPC with subnets, with the servers, with load balancer, with databases, etc. And if you need to deploy this infrastructure manually, it will take a lot of time and you will have a lot of errors because it's really tough to deploy a complex infrastructure without missing something. And it's also hard to track all the changes that you have done on your infrastructure without documenting the changes that you have done. And that's where the infrastructure as code approach came in. So basically, all the stuff that you do manually, you just need to write it down on template files. And these template files would be read by a tool like Terraform or others and this tool, Terraform, for instance, will just pass these template files, and then it will just convert these resources that we have defined on your template files to API calls. So the cloud provider. So that's where you will have all the infrastructure versions on template files and you can treat them the same way you treat your source code so you can, for instance, push them to your GitHub repository. You can step out (??) of your process and you can track also the changes. So you can create requests, do some measures, have something with an advance in the review process, etc. also. And you can also do some testing on your infrastructure. So basically, this is the idea behind the infrastructure as code. And Terraform is just a tool that allows you to achieve that. In this book. I have used Terraform to deploy the Jenkins cluster. So instead of deploying the services, and then the server at the load balancer or the WPC architecture manually, I have automated all this stuff on the template files. That way users can just need to clone the repository, run a command and they will have the same infrastructure and the same environments that I have used on this book locally or on their cloud provider.
Joe [00:22:37] Nice. This isn't in the book, but you did speak at this year's Automation Guild. And one of the questions I came up with was what part of the pipeline is a tester's responsibility? I know you talked about shared responsibility earlier, but are there any stages in the pipeline you think are really tester-centric maybe?
Mohammed [00:22:52] When you talk about the tester's responsibility, you are talking about developers or you are talking about the ops…?
Joe [00:22:58] It's usually like a software tester. He's kind of like in a sprint team with developers. He is kind of like straddling both worlds almost.
Mohammed [00:23:05] I think it depends on the structure of your pipeline. So if you have some kind of acceptant test within the pipeline, I think that might be the responsibility of the quality assurance team or the search (??) to verify the outputs and also notify the developers or the product team about the regression or about the bugs that have been spotted. So for me, it depends on the structure of the pipeline, on the stages that are defined on the pipeline.
Joe [00:23:35] So another question that came up and you actually kind of alluded to it is that people should start small because you don't get developers frustrated with actual CI/CD and have them eventually abandon it. And one of the people that I mentioned at the conference was they were doing something with security scans and that it was giving a lot of warnings, but everyone was ignoring them. So it's just like almost like automation test and they feel people start ignoring them because they're not…they take up a lot of time effort. So I guess you have any suggestions in your book or in your real-world where you can get people really to pay more attention to the signals or the alerts that are coming out or CI/CD system?
Mohammed [00:24:09] So I think one of the things that you can do is have some kind of notifications. So it depends on the collaborative platform that you are using. I think most people are using Slack nowadays. So you can implement some kind of Slack notification to notify you about the important stages of the pipeline. And this is one of the topics also that I have covered in the book. And regarding security, I have covered two ways to implement security or to inject security onto your CI/CD pipeline. The first one is running security checks on your dependencies. So when you are working on an application, you will have or will be using a lot of external dependencies. So one way to check for this security vulnerability is to use some kind of open-source tools or premium or SaaS platform to check your dependencies and you can configure the pipeline to be failed or to be successful based on the warnings on the number of errors or the severity of the vulnerabilities that have been found. And the second way to inject security is, for instance, if you are using Docker, is to scan your docker images. So you can scan your docker images to see if they contain any kind of vulnerability that has been received by the public. And the same way you can also configure your pipeline through an error if there is some kind of critical security vulnerability that has been found. So to come back to your question, I think the best way to make people involved on CI/CD and all the stuffs that are going through the pipeline is to have some kind of notifications. And I think people also need to pay attention, because when you start implementing notifications, maybe you will end up spamming people. So we need to find some kind of the sweet spot between bringing the value, bringing the benefits of notifications and also notifying the right people. So that's where you can have every time or all the time developers on the products and the quality and security engineers involved with your CI/CD pipeline. Otherwise, they will just avoid it on you will end up with something which is not useful, it doesn't add or bring a lot of value to the company.
Joe [00:26:30] Awesome. So I thought we should wrap it up. I forgot the answer. I asked you this at the conference. Is your book ready? Because I have the MIP version. I guess I have six chapters. Are they going to be other chapters? Or this is the final?
Mohammed [00:26:41] Now, there are another eight chapters.
Joe [00:26:45] Okay.
Mohammed [00:26:45] Already finished it. They are under review. So I think we will have three more chapters early next week.
Joe [00:26:51] Nice. So what a little rest of the chapters that I may be missing that you think people might be interested to know more about?
Mohammed [00:26:57] Yeah, so the first part of the book was just the foundation of CI/CD and all the practices. Also how you can set up your Jenkins cluster because we'd be using it intensively in the next chapters. The second and third parts of the book is a hands-on experience, so basically will be building CI/CD pipeline for different cloud-native applications. So we'll cover how you can build CI/CD pipeline for our microservices application running on Swarm, on Kubernetes, how you can build a CI/CD for a Cervalis application running AWS, how you can write automated tests. We'll be also covering Jenkins X. We'll be also covering some advanced topics like monitoring log-in, how you can do the migration of Jenkins, and how you can also set up authorization in our back system. So yeah, so basically it will take people from the basics until the advanced topics about Jenkins. And as I said earlier, the book is covering Jenkins but the same concepts can be applied to other CI servers or other technologies.
Joe [00:28:03] So I'm calling this a must-read. So everyone listening to this, definitely the checkout show notes to get the link. I actually have a special discount code for people to be able to get any book on Manning, and especially if you want to get your hands on this book. I think it's essential and it's a great primer and also gets you from beginning to more advanced. I love it. So Mohammed before we go, is there's one piece of actionable advice you would give to someone to help them with their pipeline automation efforts? And what's the best way to find or contact you?
Mohammed [00:28:28] So I think the best way to start with CI/CD, I think, is just to grab your copy from the book, clone the repository on GitHub. You will find all the resources, all the source code that I'm using on the book so you can try everything either locally or on the cloud and just follow with the book. And I think by the end of the book, you would be mastering the CI/CD practices. You would be ready to build your first CI/CD pipeline from scratch.
Joe [00:28:55] Awesome. And Mohammed the best way to find or contact you?
Mohammed [00:28:58] I think the best way would be Twitter. So you can drop me a message any time. I have also a blog which I post regularly new posts about DevOps or DevSecOps and you know either drop me a message on my blog or on Twitter.
Rate and Review TestGuild
Thanks again for listening to the show. If it has helped you in any way, shape or form, please share it using the social media buttons you see on the page. Additionally, reviews for the podcast on iTunes are extremely helpful and greatly appreciated! They do matter in the rankings of the show and I read each and every one of them.