A Day In The Life With Dev Op with Michael Martinez

[00:00:01] Get ready to discover some of the most actionable DevOps techniques and tooling, including performance and reliability for some of the world's smartest engineers. Hey, I'm Joe Colantonio, host of the DevOps Toolchain Podcast and my goal is to help you create DevOps toolchain awesomeness.

[00:00:18] Joe Colantonio Hey, it's Joe, and welcome to another episode of the Test Guild DevOps Toolchain. Today, we'll be talking to Michael Martinez all about a day in the life with DevOps. If you don't know, Michael is a DevOps manager and the DevOps team responsible for stop light and SmartBear's platform products. He leans on previous experience as an SDET and a full stack developer. He knows the full pipeline, and he brings focus on automation to all aspects of the development and release cycle. You don't want to miss this episode. Check it out.

[00:00:47] Hey, before we get into it, do you know that you can elevate your entire DevOps testing process with tools designed for you to do more, better, and faster? Manage, automate, and execute all your tests all in one place. A simple way to guarantee app quality. Check out SmartBear's Test Hub and say goodbye to your DevOps blindspots by using it for better test coverage. Scale your automation, boost release confidence, and more. Head on over to Testguild.me/testhub or use the link down below.

[00:01:17] Joe Colantonio Hey Michael, welcome to the Guild.

[00:01:22] Michael Martinez Hey, thanks for having me.

[00:01:24] Joe Colantonio Awesome to have you. So you have quite a background. I'm just curious just to kick it off first, how do you even got into DevOps?

[00:01:31] Michael Martinez Yeah. So I started my career as a full stack engineer at a really big company, where that pretty much the deployment process was packing everything into a jar file and sending it off to India and crossing your fingers, I guess. Kind of moved into, like, more of the startup space after that, moved into, like as a role as an SDET where it was a ten person company and kind of wore a lot of hats there, really saw like an expansive, really saw my role as a more big, expansive role, not just writing test but really like understanding where testing fit in the pipeline and how that fits in the deployment process through a series of, I guess, right sizing and things like that. I kind of ended up being the last developer left. I became responsible for the whole thing.

[00:02:23] Joe Colantonio Wow!

[00:02:24] Michael Martinez Like the whole deployment process from start to finish, like building on all the ephemeral environments. And that was like I was really responsible for all the automation there. Kind of similar thing happened at the next company and kind of here at Stoplight. Like I had to like the opportunity to kind of grow and have that more formally recognized. And then leading the DevOps team here at Stoplight for the last two and a half years. And we're still alive and kicking. So pretty great.

[00:02:53] Joe Colantonio Nice. What's it like to lead after first being thrown into it? Probably with no experience. And now you're leading one. What was the transition like to there?

[00:03:01] Michael Martinez Yeah, it was pretty interesting. I mean I feel like I've kind of always taken on a lot of like responsibility. But it was interesting to kind of like have that role more formalized and have like be more like have like more formal meetings with stakeholders across the company. Whereas, before it was like everything was a fire, everyone needed everything. And it's like, okay, now we have more formalized processes for triaging things for collaborating and form and really being able to build out those long term plans and projects with stakeholders across the organization.

[00:03:36] Joe Colantonio Now, because once again, it sounds like you were kind of thrust into DevOps to own it. Is there one thing you wish you knew before that happened, other than you knew this was going to happen?

[00:03:46] Michael Martinez Yeah. I mean, I think it's like the main thing is stay calm, right?

[00:03:51] Joe Colantonio Yeah. Right.

[00:03:52] Michael Martinez It's like everyone thinks everything's a fire and sometimes things are fire. But I feel like I've always had it one thing that I think I've always been good at is like having empathy with all our stakeholders so you know understanding where they're coming from, but also like not letting that kind of trigger your immediate reflexes is like, hey, all right, let's take a step back. Let's take a breath and let's see what we need from each other here.

[00:04:21] Joe Colantonio Love it. I want to give this almost like a starter kit for people to get up to speed with DevOps. I want to go over some mean principles and get your views on it. Because like a lot of people that listen to this have your background as an SDET and then got into DevOps, or maybe they think about switching careers or switching roles. What are some of the main principles of DevOps? Like I said, collaboration would be one of them? Automation to continuous improvement, customer centric action. Maybe we could talk about each one of these?

[00:04:48] Michael Martinez Sure. Yeah. I mean, I think one thing that's really important or one of the really like foundational principles for sure, is collaboration, right? As a kind of touched on a little earlier in DevOps, like you're working with stakeholders across the organization of both internal and external as well. We have responsibilities to our customers who are like the developers, of course, but we have SLAs like especially for uptime with our customers outside, infrastructure issues, tickets come through. We have to like really collaborate with and we need to help, developers meet their goals as well. Having really good tight connections like and really good even especially personal relationships with folks across the organization to make it easy to communicate and bring up our concerns like it's really important to the success of any DevOps organization. I feel like we don't operate in a silo. You can't just throw things over the fence where we're that's like one of the main like, yeah, we need to work together to accomplish these goals as a whole.

[00:05:54] Joe Colantonio Nice, I guess you had another wrench thrown your way. You got into DevOps and then you were acquired by SmartBear?

[00:06:00] Michael Martinez We were acquired by SmartBear, yeah.

[00:06:02] Joe Colantonio How did that affect collaboration? And did you have your own processes? And now you're like, oh my gosh, no, I need a collaborator. Gets us large organizations without these silos. Now how does that work?

[00:06:11] Michael Martinez Yeah, it's a lot different, right? Going back that small like small company where these relationships we built, it's like you have to relearn or you have to learn like the new organization structure, the new when you're the startup and like, you're the only DevOps team, you're very empowered. We have the keys to the kingdom. And now here we're like one piece of the bigger whole. So really negotiating that and like figuring out like ownership across there. But I think like one thing that we really have done well in that acquisition is bring the energy that we've had, like at it's a startup right at Stoplight to inject that into the larger organization as well, kind of like leading by example and like really digging into their processes as well. I'm like helping bring an automation mindset to their stuff too. And really breaking down those barriers. Yeah. Working more to break down those like silos that sometimes exist, especially in larger organizations. It's like, how do I talk to you about this? Like, well, we'll see.

[00:07:15] Joe Colantonio Absolutely. I love that how you said automation mindset. What does automation play in the principles of DevOps? Most people think of just functional animation. They miss out on other automation. What is an automation mindset?

[00:07:27] Michael Martinez Yeah, I think it's like especially with DevOps and especially traditional ops. There's a lot of like manual operations and requests that come through. I think a lot of it is like thinking through processes and requests that come through and like being critical about what comes and like thinking about how we can better improve the process and how we can better improve. Like from a technical standpoint and but also from like a higher organizational standpoint too, right? For example, we have sometimes there are manual requests, manual database requests that come through. And if it becomes a pattern, after the second or third one, we kind of bring it up from to the developers and like the product folks, see if we can productize that. But that takes its own timeline. Then internally, we work on creating tools to make that more seamless like create automate JIRA tickets that come through that can like we can use and have our automation on the other side where we can process those requests like in a consistent and repeatable manner because those are dangerous and we don't want those. We don't want we want to make them as safe as possible.

[00:08:39] Joe Colantonio How do you get buy in? Develop probably has a large backlog definitions of done, then you get done. And now you're like okay can you automate this database script is something that you get a lot of push back or is it more just like you said, collaboration trying to educate.

[00:08:54] Michael Martinez Data helps.

[00:08:56] Joe Colantonio Right. Right. Okay. Right.

[00:09:01] Michael Martinez Data helps. So very. Yeah. Exactly. Usually, it's like collecting the number of tickets and like the time that we've spent like on this manual task also showing that we've done work on our end to improve it that we're not just kind of like throwing things over the fence ourselves as well. But also, yeah, putting together the data and showing like what it's actually costing our team, but also the organization at large. Coming from a place of empathy. Understanding that they have their own backlog, as you mentioned. And we have our own backlog and working together to have things in the interim like short term fixes that we can have medium term and the long-term goal that we're working towards together to kind of bring that all together.

[00:09:46] Joe Colantonio Absolutely. Just in my experience, I find a lot of times where people have a process in place, they hold tight to it. Maybe because I come from a regulated environment, but I know one of the key principles of DevOps is continuous improvement. I assume you had a certain system when you were a startup, and then you're part of this larger team now. How do you work in continuous improvement to learn like what works and what doesn't work specific to your specific organization what's going on there?

[00:10:13] Michael Martinez I think the first part was like understanding where like they're coming from and where we're coming from, right? No process is perfect. And everyone has a reason for whatever process they have. Being flexible is a really big important part. As you said, people hold things like near and dear to their heart, but really nothing is sacred and taking a critical eye towards everything. Our own processes are included as well. Especially like being part of this larger organization at start. At Smart bear, there were more or there have been more I guess metrics like from like the product management perspective, I guess that they manage like from the smart site, like there's a whole organization that monitors like project metrics, I guess, whereas like in a startup it's like, hey, did we get it done? I guess, no. And we had our own pointing system and things like that. They kind of working together with like the stakeholders there to understand what they needed from us. And like what they needed from us and like what boxes needed to be checked then we weren't outliers in the whole organization, but also making it a process that we felt worked for ourselves as well and that we can kind of bring to other folks too, especially, for example. We use story points on our team to like take risk and other things, risk and like effort into consideration where it's previously the organization was tracking like hours and like just time to action. I kind of bringing that kind of mindset and explaining to other stakeholders why we use this and like how we feel that better most accurately shows like the actual work that was done or the projected, the work that can be done. Coming up.

[00:12:07] Joe Colantonio Awesome. I guess in that last principle is our customer centric action. Is this also have to do with internal because it seems like you're very internal as well with customers or developers or product owners and things like that?

[00:12:20] Michael Martinez Yeah, exactly. I feel like, especially for ourselves, like our team has a very unique-I think DevOps in general has a very unique place within the organization where we have two kinds of customers, the internal customers and the external customers as well. Whereas, like other parts of the org are really external focused. We are as well. Our team is responsible for uptime and incident response and monitoring and all the infrastructure that goes along with that. We have to keep those SLAs and like those objectives. But also we're responsible for the pipeline. We're responsible for infrastructure internally as well. It's like making sure everything is running for our developers, making sure that the developer experience is seamless, making sure that code can go out. We're deploying, I think like at the peak of our development, we were deploying like almost ten times a day, making sure that the pipeline is clean, like having everything going out and changes like smoothly. We need to work internally with the devs as well as like making sure that like our commitments are kept to our, our customers as well. I'd say because of the company.

[00:13:28] Joe Colantonio How do you get to that velocity of having not only releasing quickly, but having confidence in releasing it that quickly?

[00:13:35] Michael Martinez Yeah, I think it's having making sure that the process is repeatable and that every step of the process is or especially every environment is as close to production as possible. When I originally came in here as an SDET, for example, our testing or so we deployed using Kubernetes and Helm. But we were using CircleCI is like kind of like zombie Docker container kind of situation to create a weird ephemeral environment. And we were having a lot of issues like with tests regressions making it to production because like those things didn't match. We revamped that and we use the same helm charts like just different value styles and very similar environments. We made sure that bring everything into infrastructure as code. We can see just even just looking at the code that everything is as similar as to production as possible, bringing as much automated testing as we can and making rollbacks like as simple and as possible. If we find a regression we're confident that we can roll that back easy. And socializing that to like the developers like are comfortable and confident in deploying to production which all it does is take a merge letting them know like you open a PR it's all those tests are green. You're good to merge. And then it's good out in the wild, and that's okay.

[00:15:06] Joe Colantonio Awesome. So how about uptime? What does DevOps and SREs begin to use a SREs or are you responsible for the pipeline uptime? Sounds like you're definitely responsible for provisioning and maintaining infrastructure. How do you handle uptime?

[00:15:22] Michael Martinez We're responsible for yeah. We're like also SRE like for the SRE role as well. It's like we're responsible for uptime. We have an on call rotation that we've actually gotten developer buy in into. It is our job to keep things up. But like we have like gotten buy in from the devs as well to have that kind of shared responsibility. We're on the escalation rotation but also triaging is kind of a shared responsibility across like a lot of the organization. And I think that's something that's really great about the culture that we've built here at Stoplight, it's not again, going back to collaboration. It's understanding and building that kind of shared ownership where we own the infrastructure. But we're also here to help each other.

[00:16:16] Joe Colantonio Do you have any advice for triaging to make life easier or any steps or procedures you follow?

[00:16:21] Michael Martinez For triaging, I mean, having a runbook obviously like documenting. Obviously. And like making sure we have different levels of alerts. We have alerts that could be kind of more yellow lights. Like our lower priority alerts. But we also have a separate set of definitely everything's on fire. Like alerts so being. And socializing and documenting that I think and having that it is easy space has really helped us be able to like have like especially devs who are not as confident with the infrastructure, kind of take on that triaging role and knowing that we're going to be here to escalate as well like that we're on our own rotation and there's a safety net. That's good.

[00:17:10] Joe Colantonio Awesome. Do you have any favorite DevOps tools that you think everyone should be using, or that you find really helpful for any of the DevOps pipeline issues?

[00:17:19] Michael Martinez I mean, one of our projects was moving into Datadog. I feel like Datadog has been really great for monitoring and for monitoring, obviously, it's one of those tools where they make it really easy to spend a lot of money. I feel like that's part of like the management job that you don't really hear. There wasn't I guess cleared to at the be not clear it's like before it's, oh yeah. We just provision a lot of things and it's like no, the bill comes in, you're like, why did you spend this money? It's like, oh snap. It's been great to have like a centralized location for all our monitoring and logging and being able to cross reference that. I feel that's been really great for us. I mean, using some of the more modern CI/CD tools my first couple jobs we use Jenkins. Like here we use CircleCI and GitHub actions. I really like those are more flexible. You can have more directed acyclic graph kind of workflows, which is nice. And I mean and there's a lot of integrations with the other tools that we use already like that are native and they have their own runners that can run in your infrastructure so you don't have to worry about firewall stuff as much. That's all good. And those are pretty good tools that we've used here. We use Cloudflare as our WAF which has been pretty nice.

[00:18:42] Joe Colantonio Nice. All right. So this probably isn't controversial anymore. But there was a movement to start calling DevOps DevSecOps. How much responsibility do you have leading a team with security? Do you have like a separate security team, or is that something you really think you need to encourage people to have DevSecOps as part of the whole?

[00:19:02] Michael Martinez Yeah, there's like all the Devs, DevFinOps. All sorts. All the ops.

[00:19:08] Joe Colantonio All right.

[00:19:09] Michael Martinez And I think, it really depended on the size of the organization. At Stoplight when we were just a startup, my team was responsible. We took a lot of the responsibility for getting us up to ready for a SOC2 audit and getting our SOC2 certification. We did a lot of that. We stood up the infrastructure like we were answering a lot of the security questions that came through. But the second we got acquired by SmartBear, there's a security team. There's a whole organization dedicated to that. On one hand, we're responsible for the implementation of like whatever security things come up. Again, anything else, the responsibility is always just shared in a different way. If they have the kind of scan the agents that run the. They have that they run typically on their infrastructure, they open a ticket with us, we created and instead of us, they're coming from us like that. Hey, we need to run the certain thing, which we had our own, kind of like security agents that were running, they have their own flavor that comes to us and we do the implementation side, but it's still a conversation back and forth. We still do a little bit. But yeah I think it really I'm not kind of depends on the way your organization is structured. Like how you want to share those responsibilities.

[00:20:31] Joe Colantonio Gotcha. Do you have security built in to DevOps pipeline with some checks in code, maybe running kicks off or anything like that. High level kind of sanity checks?

[00:20:44] Michael Martinez We have like automation around like making sure dependencies are okay. We have code coverage security like code analysis and that front as well. We also have I guess like now that it's been handed off, I'm not sure. But we had like contracts with some vendors who would do like regular pen testing and things like that. And I'm sure the security team has their own stuff right there as well.

[00:21:10] Joe Colantonio For sure. Nice. So every time I look at DevOps online, I see images. I always see the DevOps lifecycle discover, plan, build, test, deploy, the holding operate, observe continuous feedback. As a beginner, I've never done DevOps. I've been an automation engineer all my life. How much do you spend on each of these steps? Is it really this discrete or is it just like in the day in the life, how much do you spend on each of those little segments and that little figure eight.

[00:21:35] Michael Martinez Yeah, I mean, it really depends on the day, right? But for example, we work like we work in an agile framework like development cycle. So like for example, every two weeks we have our spring rituals where we do like a lot of the planning where we do grooming, planning, things of that sort. We do a retro. That's kind of like the feedback part of it. There's dedicated time for that. Ideally, deploying and testing is kind of like automated. But we're always like building, we're also always building improvements to that as well. Building improvements or automation in that test integration as well. And observability, building again with going back to the automation mindset making sure that our observability tools are up to scratch. And like that we have processes in place there and like for operations that there's flexibility like high availability, that our infrastructure is already designed to be flexible with those things. It really depends on the day and the week and what's going on. But like ideally we do spend a lot of time on the building so that the other parts can kind of flow naturally and flow back to that. And we can iterate on that again.

[00:23:02] Joe Colantonio This is probably a dumb question. Back in the day, my companies like, oh, we're going to do DevOps. But they didn't focus on deployment. We had to do manual steps before test kicked off and on the environments, so wasn't built to be deployable. It was kind of stupid. Is that still an issue? Did people automatically build it to be deployable? Do you ever have to say, you have to build an API or a hook or something to be able to get this to run seamlessly?

[00:23:27] Michael Martinez Yeah, I've actually kind of been surprised sometimes by how common it is that people don't just automatically deploy. That's not as much automation. There's still like a lot of manual release processes out in the world. And I think like for us, when we see that we try to address that and make it empower the devs. Empower create a process to empower developers to kind of deploy their code and also educate because I think one of the main reasons, when we came to SmartBear, one of the things that we saw was some teams weren't deploying or they didn't automate their pipelines, the releases because they did not, or they didn't think that was compliant with SOC2 regulations like the SOC2 certification. We had done that in a SOC2 compliant way. And kind of like educating them and bringing that as well and educating our SmartBear counterparts. And like bringing them up to scratch with those letting them know that it can be a fully automated process. It's been good. I think that's the goal. And like that's what we strive. And like I personally, here and wherever I go in the future. That's kind of like the first thing, you see that needs sometimes it needs to be fixed for sure.

[00:24:49] Joe Colantonio Absolutely. All right. So speaking of future, let's put on your little psychic hat here. I'm asking everyone this obviously, AI has been a big deal because it's almost impossible to know, but because you're part of a larger company now that probably has a lot of different customers, a lot of verticals, probably see things a lot of people don't. Where do you see DevOps heading the next five years? And how much of AI is in play or how much is it going to change things, do you think?

[00:25:13] Michael Martinez I don't know, I personally see, AI is more of a tool, right? I know some people get afraid and I was like, oh, it's a replacement. Even now we're using kind of like we are using AI in our normal job. We have licenses for GitHub Copilot. And that helps make a lot of like the simple things, you have like a Terraform template or something and it's like, okay, here these are the basic parts instead of like having to go look at the documentation. I see AI helping us build better tools to empower our developers going forward. I don't think it's necessarily the end all be all. I think it's anything else. It's also especially it's a lot of trust but verify, right.

[00:26:02] Joe Colantonio Yeah. Right. Right.

[00:26:04] Michael Martinez With that I know Amazon has tools to kind of help with cost and efficiency and things like that. We'll see a lot of AI-empowered kind of development going forward. But I think it'll kind of speed us up. But eat my words in five years but not replace us.

[00:26:26] Joe Colantonio Absolutely. All right, Michael, before we go, is there one piece of actionable advice you can give to someone to help them with their DevOps efforts, and what's the best way to find or contact you?

[00:26:36] Michael Martinez Yeah, I think, I mean, to help with the DevOps effort. I think it's just collaboration, I think is one of the things and empathy, have empathy with your stakeholders. Don't be afraid to reach out to people and work together, I guess. It's not a silo, it's a group effort. And I think that's a good guiding principle that I've used that I think is really helped a lot. I have a LinkedIn I think that I sent that I can be reached there. And yeah, I think that would probably be the best way to reach me.

[00:27:12] And for links of everything of value, we covered in this DevOps toolchain show. Head on over to TestGuild.com/p155. That's it for this episode of the DevOps Toolchain show, I'm Joe. My mission is to help you succeed in creating end-to-end full-stack DevOps toolchain awesomeness. As always, test everything and keep the good. Cheers

[00:27:34] Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps, and automation professionals in the world, we'd love to have you join the FAM at Testguild.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real-world value that can improve the skills or solve a problem for the Guild community. I love to hear from you head on over to testguild.info And let's make it happen.