Scaling Smart: DevOps Techniques for Cloud-Native Success with Jacek Marmuszewski

[00:00:00] Get ready to discover some of the most actionable DevOps techniques and tooling, including performance and reliability for some of the world's smartest engineers. Hey, I'm Joe Colantonio, host of the DevOps Toolchain Podcast and my goal is to help you create DevOps toolchain awesomeness.

[00:00:18] Hey, if you want to learn more about how to make your apps more scalable and how to create secure cloud infrastructure, you're in for a treat because today we have an expert with us, Jacek who has over 10 years of experience building and managing cloud infrastructure. He's worked for companies like Saber and Oracle on mission critical systems, so he really knows this stuff. He's also has his share in startups. He's done a lot of different things. He's been an early joiner, many of them, and he's helped promote DevOps culture and advocated cloud native architecture from a very long time. Also, he recently co-founded Let's Go DevOps, which is a company that helps others design, build, and maintain cloud native applications and infrastructure. He's a big fan of cloud transformation and helps others to leverage it to its full potential by choosing the right components for the job. If you want to know what some of those right components are, you're in for a treat. Don't miss it. Check it out.

[00:01:07] Hey, before we get into this episode, I want to quickly talk about the silent killer of most DevOps efforts. That is poor user experience. If your app is slow, it's worse than your typical bug. It's frustrating. And in my experience, and many others I talked to on this podcast, frustrated users don't last long, but since slow performance is a sudden, it's hard for standard error monitoring tools to catch. That's why I really dig SmartBear’s Insight Hub. It's an all in one observability solution that offers front end performance monitoring and distributed tracing. Your developers can easily detect, fix, and prevent performance bottlenecks before it affects your users. Sounds cool, right? Don't rely anymore on frustrated user feedback, but, I always say try it for yourself. Go to smartbear.com or use our special link down below and try it for free. No credit card required.

[00:02:03] Joe Colantonio Hey, Jacek. Welcome to the show.

[00:02:07] Jacek Marmuszewski Thank you, Joe, for the warm welcome. I'm really glad to be on the show. So yeah, let's get started.

[00:02:12] Joe Colantonio Yeah, let's do it. I guess the very first question that comes to my mind is why DevOps? How did you get into DevOps? Why are you so gung ho about cloud native stuff?

[00:02:20] Jacek Marmuszewski So well, the story starts with me finishing the college. I was thinking I will be a programmer like everyone else. And back in the days that Java was a top thing that you could actually get hired. And for my college internship, I was actually hired as a Java developer. I quickly learned that in the old, especially in the old days of Java, amount of scaffold that you need to do in order to build application was so big that I decided that Java is not the thing for me. I switched to C++, which was a great opportunity to learn some cool stuff, the hard way.

[00:02:56] Joe Colantonio Well, you moved from Java to C++ as something better?

[00:02:59] Jacek Marmuszewski Something faster.

[00:03:00] Joe Colantonio Oh, faster. Okay.

[00:03:02] Jacek Marmuszewski Like I was not so keen about the speed of Java in the very first editions. I went for C++, spent a year or two trying to learn the stuff, learn that this is not my thing. But at the point I realized that I was in the really cool place where we had a really cool system and hardware components. I started getting into those meetings and starting doing the stuff with operating systems with mainframe that Saber had at this time and decided this is way cooler from my perspective than writing code. The code can be written through Jira ticket to somebody else. Well, I can actually work with the low level stuff. I started as a system admin slash system engineer. And when the DevOps culture clicked, I was one of the first to join in, cause like I had a background in actually programming for a year or two. had a little bit of this background. I was taught how to program. And for me, the transition from being admin to being a DevOps was a really smooth transition, especially when I joined companies that showed me how you can use the cloud for building scalable applications. And the knowledge from the programming and system engineering was actually something that just clicked in and this is how I became a DevOps infrastructure platform engineer. I'm not sure how to call it, to be honest.

[00:04:22] Joe Colantonio Nice. Cool. So I guess when it comes to infrastructure, then scaling, why is it so difficult? A lot of people think when you go to cloud, everything just scales infinitely. You have all the resources. What is the biggest, maybe misconceptions or things that usually chip people up into trying to make things more scalable?

[00:04:39] Jacek Marmuszewski I think there are two things. The very first misconception is that cloud only claims to be infinitely scalable. There are limits. Most of the people will not reach them, but we actually broke the cloud in a couple of scenarios when we reached those limits. But I think this is like the hard part. Like you probably will not have this problem when it comes to scalability. The biggest issue is with application design itself. We often get a new startup thingy and we have the system that was built, designed pretty rapidly. But then we try to launch it somewhere and scale it. And first of all, we learned that you cannot have a second server because everything, including user sessions are kept in memory. Like you cannot set second server unless you are doing some kind of sticky sessions and some weird stuff. Then you come to the conclusion that, okay, the application is not booting correctly when it comes to multiple servers or the shutdown actually loses some kind of data. And when it's comes to scaling up and down, especially automatically, you need to make sure that your application behaves correctly. And I think it's when it come to scalability, it's not about the infra, it is more about the initial design and the concepts that you are doing. If you're following 12-factor application scheme, then probably your application is ready to be super scalable. But like, when we are thinking about scale, we have a customer that really utilizes the cloud that for most of his month, they are working on two servers and they need about 60 to 80 servers for just two or three days over the month. So they are launching those instances automatically, just paying for the usage for those three or four days, and then they get back to a cheaper operations, something that you cannot do with physical hardware.

[00:06:29] Joe Colantonio All right. So it almost sounds like it needs to be built from scalability from the start, almost like something needs to be testable. You need to build it in something to be scalable. It'd be better if they built with scalability in mind. How do you get developers then to make that mind shift so that they are thinking about scalability from maybe the beginning? Think they should do a little POCs first before they commit to an architecture before building it out and then worrying about scalability?

[00:06:55] Jacek Marmuszewski I think the scalability part comes with experience that developer have. So at some point he just writes scalable software and builds scalable systems. And for those who are starting, this is where we actually come in as Let's Go Develops. Our goal is to join companies as early as possible. When you are thinking about the design, we can talk with you about the initial architecture that you have and say that this part, this is your database, it will not be scaling well. You should treat it as a rather hardcore component, but those parts you can actually scale up and down. Let's move the logic from database to applications. You would be surprised how many people are still writing a really hardcore SQL queries or use a functions in databases to do some kind of business logic. And this is the stuff that doesn't scale well. The sooner we are involved, it's the DevOps culture. You get the infra guys and your business guys on the very first meeting. And you ask them what we should build and how, and let's have a brainstorm and figure out how the application should look like in the end.

[00:08:02] Joe Colantonio Awesome. I started my career as a performance tester. How do you test the scalability to see if it's performant? Do you put a load on it with the tool? How do you normally do it? Or do you just know best practice that once it does have load on it, that it's going to behave correctly and scale correctly.

[00:08:18] Jacek Marmuszewski Yeah, when it comes to checking out the application, we usually use something like K6 to generate a little bit or a lot of load to understand how application behaves. And like in many cases, when you are thinking about scalability, you may test it with the numbers that you have in mind for your final product, but the bigger thing is to actually understand when you have a bottlenecks. Like disabling the auto scaling completely checking which component breaks first and why and understanding this flow, fixing it or upgrading this part and checking what are the next things that will go down. This actually gives you quite a lot of understanding how the application will work under load and allow us to build monitoring, alerting around the application that will also do an automatic scaling and downscaling of the app itself. And this is how we start. So like when we get a black box. K6 and talking to developers, understanding the flow is the key. And like for a day-to-day operations, we put a lot of metrics in place. Like you don't need to do that much of performance tests per se, like the synthetic ones, but you can get a lot of potential issues from production or from metrics. And you can have theories, validate them and actually upgrade the system whatsoever.

[00:09:41] Joe Colantonio Nice. This may seem like a weird question. How many people know they have an issue? Like how hard is it to say, Hey, I have a company, let's go DevOps. You have an issues, you may not know about it. How hard is, it like, do you have to give them awareness? So they know, we know we have scalability issues as it, but you actually still have to educate them, Hey we know you have this issue that you don't know about yet. Does that makes sense?

[00:10:04] Jacek Marmuszewski The question may be asked, how are we reaching a companies or clients as Let's Go DevOps? I think that's extremely tough question because the answer is I don't know. This is the stuff where I have a lot of metrics, but still couldn't figure out all of the stuff. But there are mainly two cases. Like what I personally love is to get in touch with early stage companies and just say that, okay, we will manage your infrastructure and all the boring stuff. This means that infrastructure, IT support, security, compliance, blah, blah blah, everything that is not custom designed for your business so that the owners of the company can actually focus on building the company and building the product. Like usually in startup, you have two or three of those guys. If one of them is focusing on building infra, that's like, I think it's a waste of his time and potential of the companies. They can actually launch faster to production if we take care of our building else. It's usually a robust setup. We can understand a lot about the business to build a platform for them. And this is one of the approaches. And in here, you get this learning curve of what this means to be scalable and so on as a add-on to the initial services that come in to help you with initial traction. The second option that is not that fun is when somebody calls me in the middle of the night and says that I just figure out that we have a scalability issues. And I usually ask, so what's the case? Well, it doesn't work anymore because we hit some kind of the limit. And those are usually a second cases like where the client actually feels a huge pain of system that is not running and wants to do something with it. So we have quite a few of those companies that actually came to us with a fire on hand asking for help.

[00:11:57] Joe Colantonio I'm actually getting a similar issue. I have a PHP work or memory limit I keep hitting. My developer has no idea. That's probably definitely a scalability issue. That's a hot mess for sure. What do you do with people that have a hot mess like that? Do you just tell them how it's too late or do you actually, how hard is it to fix the issue in a way that's going to be in a timely manner? Because I guess, a lot of those would be deep rooted issues, right?

[00:12:22] Jacek Marmuszewski So yeah, usually we just jump in, we take the best of our company and try to fix it on the fly. This usually means that we are by design getting a lot of permissions to a lot of the systems. So like the NDA comes first and then we try to figure out what can we do with an issue. We were actually called twice or three times to the live production incident with all the paperwork being signed in a couple of minutes. That was pretty insane from my perspective because usually the paperwork takes months. In here it was minutes. And yeah, we sit until we finally figure out what's wrong. We get dirty, we get our hands dirty, and we start going through metrics, logs, servers, whatever necessary to actually get the issue fixed.

[00:13:10] Joe Colantonio Nice. Talking about paperwork and all that, how do you balance maybe speed and agility with all these new security compliance, probably concerns that companies have?

[00:13:21] Jacek Marmuszewski That's a really tough question. Especially for the young companies, everyone needs to understand that if you have an investor and the runway for three, six months, you need to deliver product. And usually the security is not that important. I probably don't, shouldn't tell it on the record, but like that's the reality of young startups. Then you also have some kind of feature delivery that is in many cases more important because still you don't have a real user base or you have just a couple of users. The thing how we try to approach the security stuff is that we build mechanisms to actually control and make the stuff secure from the very beginning. So like all of our designs, infrastructure, CI/CD pipelines, and so on. They all have some kind of knobs to tickle if you want to enable security and disable it. And in many case, for the first couple of months though, are pretty low or medium set. Not to, you know, say they are low. They're medium set. And when the company matures to the point that they start realizing that okay we have exposure, we have really valuable data, we actually have traction and we should take care of security, it's not that we are coming to the system and coming there and it's like nothing is here. We need to rebuild everything.

[00:14:41] Joe Colantonio Right, right.

[00:14:41] Jacek Marmuszewski There are those knobs already prepared there. We can just enable a couple of switches, tune it a little bit up and we go from medium to high in a matter of days usually, because it's more about education than actually setting up the technical part.

[00:14:55] Joe Colantonio And I guess that's another good reason of working with a company like yours because you may not start off with security, but you've already thought about it in the backend where when it does become more of a priority, then it's ready to flip the switch then almost it sounds like.

[00:15:07] Jacek Marmuszewski Exactly. It's about cutting corners, but cutting them in the smart way that you don't cut the flesh too deeply. It's just a small wound for the product itself. And the fact that we actually launched a couple of startups in the past and we know how to work with them, we try to add additional of this knowledge that we gathered over the years to build it better every next time.

[00:15:31] Joe Colantonio I don't know how common this is. I know a lot of companies, I've read a few articles where they went to the cloud and then they realize how much it costs them and like cost less to go back to bare metal. Is there anything people can do to maybe help with their cloud costs to optimize when they're scaling to make sure they're not just spending more money? Make it just going up and up because they don't have a good plan in place, I guess, for scaling.

[00:15:55] Jacek Marmuszewski Oh my God, there are so many things that I can talk now. I'm actually afraid that I will take the rest of the show, but when it comes to going to the cloud, one thing that I usually start with is when you are thinking about the cloud the same way you are thinking about bare metal, like it's you pay premium for a lesser level of stability in the cloud. That a little bit of phenomenon because like even EC2, if you compare Hetzner offering with EC2, EC2 is, well, far more expensive and not that stable. But the truth is that cloud allows you to buy resources just for minutes for the time you really need them. So like from the very beginning, from your design, if you are designed for those bumps that go up and down in the traffic, this means that you don't need to buy hardware that will facilitate your peaks, you can actually buy the hardware when you need it. And it starts with this simple approach. Then we have a lot of stuff where in the AWS, you can, for example, develop really quickly and use services where you pay per request. And those are usually cheap in the very beginning, but when you reach a certain scale, they became extremely expensive. So like in the strategy, if you're doing POC, it's good to use stuff like AWS Lambda to check out if you really have any kind of traction. But when you reached a certain level, It's far more cost-wise to move it to some other stuff of compute. And I don't want you to give you a exact hint where, because it can mean that you will go to ECS or to EC2 or to Kubernetes, depends on the business and depends on the scale, but like it starts with this, you have also a lot about data, how you handle it, how you move it from hot and cold storage. Plenty of options where you can upscale and downscale the infrastructure. In the way it will be cost effective. There is one story that I want to mention here. Like we have one customer that works with us for around 10 years. It predecessors the Let's Go Devops founding and they were focused on optimizing the cloud spend extremely. They're very focused from the very beginning. The business had a pretty low margin, especially in the very beginning, so the cloud spent was actually quite a part of how much they were getting money. And we did a lot of cool stuff and we estimated that the infrastructure cost from the well-designed architecture to what we actually managed to do by doing cloud-native approach from the very beginning is about 80% reduction. So they are paying only 20% for the infrastructure compared to the full AWS without a lot of cool features like spot instances and so on. And the AWS was still five times cheaper than what they had on their previous hosting. From their perspective, the infrastructure is actually running on pennies, not on the full price that they had with their previous hosting provider.

[00:19:06] Joe Colantonio Nice. I guess to add on costs, how soon do people need to worry about, or do they even need to worry, about like a multi cloud strategy? Sometimes you have outages of AWS and then whole applications are brought down because they don't have like a failover. Is that something you recommend or you still think stick to a single cloud vendor and you probably, that's like a one off once in a great while, maybe it goes down.

[00:19:28] Jacek Marmuszewski That's a good question because like two or three months ago, we had an outage of huge portion of internet. Not necessarily AWS, but yeah, a lot of internet went down. And when it comes to multi-cloud strategy, we have opportunities to work in fully multi- cloud approach. We had a GCP and AWS being a hot and the traffic switch was pretty much seamless between those. But amount of work that we need to put in and amount of things that we need to resign from or like features that we need to resign from AWS and Google. Like maybe let's try to rephrase it. If you have a really cool offering in AWS and the same kind of offering in Google, usually they will have 10, 20% of really similar features, additional 80%, it's something that is cloud specific more or less either in configuration or those features are just missing in the second cloud. So with multi-cloud approach what you can do is you can stick only to those 20% of common stuff that you have available in both clouds. The biggest pain point for us was actually managing firewalls because you had security groups on one hand and then you have those stacks for firewalls on the other one. You cannot just merge them together. So you need to build some kind of a layer of translation. You probably are getting back to the scenario where you are doing IP ranges for different services. So like you're getting back to the 80s when it comes to managing infrastructure. If you really want to go multi-cloud, it's a huge cost. The biggest question that I ask my customers is like, before you think about multi- cloud, tell me what will happen if the half of the internet dies because this is what we are talking if the GCP dies or Google or AWS dies. Nothing is working. All the pages that you are visiting every day are not working and now the customer tries to go to your page. If Google is down, probably if you'll not be able to search it, but like he goes to your page and sees that it's down. Will you think that like, oh, the guys don't know how to build the infrastructure and the services down? Probably not because most of the internet is not working. So like whatever. And this is the case of actually AWS region going down. Probably half the services for the people are not there. In many cases it's not that big of a deal when it comes to availability and you can actually live with the fact that you die with half of the internet. That's okay. The biggest issue or something that I usually recommend clients is that think also about the disaster recovery so make sure you can restore in other cloud. Probably it will not be an hour, maybe a week, but let's assume that for some reason AWS screwed up and melted in in server room. I want to have a backup in secondary location, probably secondary cloud, but this is hot and really cold approach to just the data. I think this is cost effective and cost wise. You should really think if you want to be online where everything else is dead and offline because that's not that easy answer.

[00:22:43] Joe Colantonio Absolutely. That's a good point. A lot of people probably lulled into a false sense of security. I don't know how many people still think of a backup strategy and everything, but that's I think that's good point for sure. So we mentioned security already. How much security is part of DevOps for you? And does AI make it more difficult? I know there once again was an attack recently. Someone just used AI and it was able to like hit a bunch of different websites just because a new was able to go through all the vulnerabilities really quickly.

[00:23:09] Jacek Marmuszewski When it comes to security, I actually have a title of DevSecOps. So the security is a really important part of our job, especially if you think about young companies that don't have a CISO role and so ever. So the Security part lies a little bit on us, at least we think so. And we try to influence and build it from the very beginning. This is really cool when the CISO actually comes to the company and he's usually like, okay, I am a CISO for a couple of startups already and I know I will be in the really deep, deep hole. And then he comes and it's like, no, the first floor is actually pretty, pretty okay. The second one is half built. Now you can come in and start venturing the place because like we try to build it from the very beginning. When it comes to AI at this point, I haven't saw any increased number of attacks that are driven by AI because if you think about it, like even in the past we have a lot of script kiddles that were downloading-there's a addition to Metasploit called Armitage that actually launches everything against the target and so a lot of this kind of traffic I'm not sure if AI maybe made it a little bit more sophisticated because the Armitage was a really dumb way of firing everything. With AI, you can probably ask a lot of questions to drive you into a really good direction. But the biggest thing that I'm actually afraid when it comes to AI, we are allowing more and more agents to run in the infrastructure. And I'm more frightened about the attack vector for those agents because you can actually send documents or prompts that will be targeting artificial intelligence living in your infrastructure so you can actually make a bad actor inside of the infrastructure. And that's I believe more scary from my perspective.

[00:25:03] Joe Colantonio All right. Now you just freaked me out. So if someone worked with a company like yours, then would you point that out? Like, Hey, I noticed you're starting to use these AI agents just a heads up. This could be a concern down the road because I don't know how people don't figure that out until it's too late, it's almost unavoidable.

[00:25:21] Jacek Marmuszewski So in here we come with a set of best practices. If you come to me and say, okay, I want to install this agent and it will be operating on this particular set of data. It's like, if you're going through, I don't know, scrapping other web pages and I ask your AI, it will go down to the agent, scrap something that's publicly on the internet. Like really from my perspective, let it go. Maybe it will generate some costs when it comes to traffic. Well that's it. We start to make those conversations where you want your agents to be hooked to a lot of data sources, especially if they contain critical for the company information or PII's or something like this. So in many of those POC designs that we saw in many companies, is like you put a single server and say, okay, so we had a silo for each of the application and we had the database that's connected to single application. But now let's take the AI. Let's open up all databases to it and let it digest all the data to learn and to produce answer. In here, we start the discussion about what is the potential impact? How can we prevent it? What measurements we have built around the system to make sure that if the AI goes rogue, we either catch it beforehand or even be able to block it so it will not exfiltrate the data or not do any harm to any of the systems.

[00:26:48] Joe Colantonio You mentioned PII, and my notes have something about WorldCoin, and that got me thinking WorldCoin, PII privacy. Are you concerned about digital identity and privacy from cloud? A lot of people when they hear cloud, they think, oh, I can't do it. I need an on-prem type of solution. Do you have any thoughts around WorldCoin and how maybe that raises any questions around any of these issues?

[00:27:08] Jacek Marmuszewski I think for WorldCoin, they are doing a really good job when it comes to privacy. And this is one of their mission statements that the privacy is the key. And probably I cannot talk much about the details of the project, but I can tell you that when it come to how far the rabbit hole goes on AWS, how secure you can be with compute and with data that you have, I think we reach quite far into the rabbit hole. Starting, even if you think about the standard approaches, the AWS enclaves for encrypting everything and plenty of stuff that you can pick from AWS. And AWS claims and has a lot of documents showing you how compliant and secure they can be. And one of the things that often comes to my mind, we had this conversation with one of our colleagues. If you think about a physical server room that you have in your building. If I want to get to your hard drives, I need to come to your building, take those out, if you go to AWS server room, you have probably a million of drives. Finding the data there is actually a lot more complicated than you think. And a lot of the concepts that AWS is building into their servers, cause like they, even the nitro machines, the hardware components are actually making sure that you cannot mess with other VMs, you cannot mess with other's data. And AWS, in my opinion, is doing a really good job when it comes to allowing you to use a security features that are extremely enterprise when it come to design and execution.

[00:28:48] Joe Colantonio Okay Jacek, before we go, is there one piece of actionable advice you can give to someone to help them with their DevOps efforts? And what's the best way to find or contact you or learn more about Let's Go DevOps?

[00:28:58] Jacek Marmuszewski I would probably guide you to my LinkedIn. So like you can ping me with a message if you want to talk about anything, or you need to know the advice or maybe a mentor, like if you want the mentor and you want to talk about your designs, like ping me, we can get a virtual coffee and actually go through the issues that you might be having. I'm happy to share my knowledge, especially with the young companies. Like I really root for the best for startups. So like if your starting up, just come with technical questions and I'm really open to answer those.

[00:29:29] Joe Colantonio Awesome. We'll have a link for that in the comments down below.

[00:29:32] All right, before we wrap it up, remember, frustrated users quit apps. Don't rely on bad app store reviews. Use SmartBear's Insight Hub to catch, fix, and prevent performance bottlenecks and crashes from affecting your users. Go to SmartBear.com or use the link down below, and try for free for 14 days, no credit card required.

[00:29:54] And for links of everything in value we covered in this DevOps Tool Chain show. Head on over to Testguild.com/p195. So that's it for this episode of the DevOps tool chain show. I'm Joe. My mission is to help you succeed in creating end to end full stack DevOps toolchain awesomeness. As always, test everything and keep the good. Cheers.

[00:30:16] Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation, testing, and DevOps. If you haven't yet, join our vibrant community at TestGuild.com where you become part of our elite circle driving innovation, software testing, and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real world challenges, we're excited to collaborate. Visit TestGuild.info to explore how we can create transformative experiences together. Let's push the boundaries of what we can achieve.

[00:30:59] Oh, the Test Guild Automation Testing podcast. With lutes and lyres, the bards began their song. A tune of knowledge, a melody of code. Through the air it spread, like wildfire through the land. Guiding testers, showing them the secrets to behold.