Scaling ML Infrastructure Like a Pro with Elliott Clark

[00:00:00] Joe Colantonio Get ready to discover some of the most actionable DevOps techniques and tooling, including performance and reliability for some of the world's smartest engineers. Hey, I'm Joe Colantonio, host of the DevOps Toolchain podcast. And my goal is to help you create DevOps toolchain awesomeness.

[00:00:19] Hey, in today's episode, it's a must listen if you're trying to scale machine learning infrastructure. Well, burning it all down. Hey, I'm Joe, and our awesome guest today is Elliott Clark. Someone who doesn't just talk scale, he's lived it. As you'll find out in this episode, if you don't know, Elliott has built one of the largest online transaction processing systems on the planet. Help shape Apache HBase at Cloud or before the IPO, and most recently led the ad infrastructure team at Meta. Now, he is also the founder of Batteries Included, rethinking how platform infrastructure works, but this time, as his own boss and working with nodes. Really excited to hear more about this. If you're wondering or really want to know how to do a work at scale and how ML systems survive real world pressure or why infra teams keep repeating the same mistakes. This one's for you. You don't want to miss it. Check it out.

[00:01:10] Hey, before we get into this episode, I want to quickly talk about the silent killer of most DevOps efforts. That is poor user experience. If your app is slow, it's worse than your typical bug. It's frustrating. And in my experience and many others I talked to on this podcast, frustrated users don't last long. But since slow performance is a sudden, it's hard for standard error monitoring tools to catch. And that's why I really dig SmartBear's Insight Hub. It's an all in one observability solution that offers front end performance monitoring and distributed tracing. Your developers can easily detect, fix, and prevent performance bottlenecks before it affects your users. Sounds cool, right? Don't rely anymore on frustrated user feedback, but I always say try it for yourself. Go to SmartBear.com or use our special link down below and try it for free. No credit card required.

[00:02:06] Joe Colantonio Hey, Elliott, welcome to The Guild.

[00:02:10] Elliott Clark Well, thank you. Thank you for having me.

[00:02:12] Joe Colantonio All right. Before I get into it, is there anything I missed in your bio that you want The Guild to know more about?

[00:02:16] Elliott Clark I think for people that are kind of DevOps and testing, and stability interested, one of the other positions that I had at Meta was I was a tech lead of a team called DevEfficiency, which basically means how can we make developers faster without breaking things. It was a lot of testing, performance, and deployment related stuff. That was also part of my scale journey was learning and breaking those things quite well.

[00:02:44] Joe Colantonio All right, so let's put you on the spot there. What are the top two or three things you implemented with that to help with dev efficiency?

[00:02:51] Elliott Clark Well, can't talk about all of them, right? Meta keeps tons of it for themselves.

[00:02:56] Joe Colantonio High level, high level.

[00:02:59] Elliott Clark High level? I think a lot of stuff that would test shadow environments with real production type traffic. And that was one of the big things that I've learned is just how powerful your production traffic can be in making sure that you don't break things in actual production.

[00:03:21] Joe Colantonio How do you get there, though, when I used to do performance testing, I used to test and staging, test and go to production. Bam! but it's that's been years ago.

[00:03:27] Elliott Clark Yep.

[00:03:27] Joe Colantonio I assumed you have a dockerization, ephemeral environment to spin up and completely mimic a production environment.

[00:03:35] Elliott Clark Yeah, Facebook has a lot of environments. And so they have something where they will capture incoming traffic for different services. And then you can replay it. While I was there, we had an open source project called Treadmill, where you could take thrift type requests or other requests and replay them. And so if you can replay what's going on in production. You then have a lot of different things you can do. You can do performance testing. Do I get through this production run fast enough? Great. My production won't go slower because I have introduced a new bug. You can also sum up the total results of a replay. I replayed an hour. In that hour, were the ads that we would have recommended what we wanted? Or were they x, y, or z, you can kind of. Go back and look at what would have happened if production had been using this version. And that becomes super, super powerful. You have a baseline that you can compare against before you even go in production. And I think the real insight is just how different production traffic is than what developers assume it's going to be. You all write a test, a unit test that says, well, this is how someone calls my thing. But you don't really know, and there's always a new thing that you learned, a new something, and so keeping up with what's actually happening in production and using that in test environments was super, super useful and super powerful.

[00:05:06] Joe Colantonio That's a good point. I guess if someone had like a microservice infrastructure, you may call a service and I know it has dependencies and other services until you use a tool like sounds like Treadmill to find these. Very cool.

[00:05:16] Elliott Clark Or your dependencies start sending different results. They have a new version. And so what that new version sends to you, your behavior can be different and you want to notice that before you go. You want all of those things to kind of surface in rise before it rises in production.

[00:05:39] Joe Colantonio Interesting. Could you use some of these tools in production first to capture this? Before you even start building your requirements or developing to know exactly what you should be building off, that makes sense. A lot of times people build anticipating what it should be or how it's going to be used in production. But if they listen to production now, maybe they can anticipate some issues before they even start coding. Does that make any sense?

[00:06:01] Elliott Clark I don't know about before you start coding, but definitely you can use it before you're too far along. A lot of performance, a lot of, hey, I'm doing this, you replay it locally. And does it hit the same problem? Do you still see the same slowdown or can you get faster? Meta and a lot of the bigger companies I've been at all use compiled languages. And so figuring out what the compiler is going to do for you. And what that end result of your code is going to be is a lot harder to reason about. It's not quite as straightforward as, this will make me faster. It does make the whole program faster.

[00:06:39] Joe Colantonio Nice, you obviously inherited probably a lot of systems are already in place. You had no control over. I think you're known for more of building scalable infrastructure from the ground up. If you're on a greenfield project, say, how do you know what infrastructure to use? Do you do a little beta test between different infrastructure or different technologies? How would you get started, I guess?

[00:06:59] Elliott Clark Yeah, sure. For me, the kind of way I choose the default infrastructure is I go with what the community recommends the most, right? So you're going for a database, pick Postgres. You're going SSL. You go with Let's Encrypt. You just kind of choose the default and it gets you out there enough that you can stub your toe. The real engineers change because they have a complaint about something, right. That wasn't fast enough. That didn't solve my whatever. But you don't have that insight of what's not going to fit your problem until you actually go out and try it. Choose the default, use that to build enough where you can start saying, this is what didn't work for me. And then you can go search for the kind of more cool or fancy or shiny things because you have a need. You have a thing that pushes you to it.

[00:07:52] Joe Colantonio That's a great point. It's like doing a little POC, not just sitting there like playing around for months, just trying to figure out the perfect infrastructure.

[00:08:00] Elliott Clark Yeah, you won't be able to say what you need to tune until you know, you won't able to choose and say this trade off is worth that until you know what the pain is.

[00:08:14] Joe Colantonio And if you're using more common tools, does it help you to know how to design for scale or does it all depend on your particular application? Is they're all different, I guess, from one to the other.

[00:08:24] Elliott Clark I really think that a lot of it comes down to learning your tools and so choosing the ones that the community has already kind of banged on means you don't have to go learn that, hey, this thing can't actually scale for me. You can say, hey I've had a problem with, and I use Postgres right now for a lot of JSON and stuff. And so I can go find a blog post or a developer that has talked about how you can optimize JSON storage on Postgres. If I tried to do that on a database that has a 100 people using it every day, I probably don't go find that blog. And so my path to go find that optimization is much, much harder. I have to start reading code. I have do all of those things rather than finding an expert that can already point me in the general direction. At some point, you still are past the blog point and you've got to go deeper and deeper. That's still always there. But I'd rather that softball, easy automation. Be something that I can go have the best already kind of found and pointed out for me and learn from the giant to have already come before me stand on their shoulders.

[00:09:33] Joe Colantonio Nice, I don't know how many people do think the scale is easy now, you just throw up more virtual environments and there you scale. I'm sure it's a lot harder than I obviously there's other breaking points, I'm sure, but is that too simplistic to think?

[00:09:45] Elliott Clark No, I don't think it's that simplistic but I think scaling often comes down to actually a problem of scaling people and scaling people's understanding. We've gotten to a point now where, yes, if I need a second or third read replica on a database. There's no can this be done, right. You know this can be done. But the hard part then comes in knowing what all of the things you've just done really mean in production. Right now I have to wait for log replicas. Now, I have to keep monitoring this and that. And those kinds of things are the things you have to teach people. That is where scaling really comes in, is teaching your team how to move correctly and stably around different code bases is super challenging.

[00:10:31] Joe Colantonio I would think it seems the softer skills or the non-technical skills are the hardest to scale. Any pointers on how to do that with the team get them up to speed or get them all on the same page?

[00:10:43] Elliott Clark Yeah, so I think a lot of it comes down to culture, right? You've got to have a culture that encourages bringing people along and encourages people to be okay with failing. So that means that your new engineers can come on and feel okay trying some experiment and bringing it back to the team and saying, hey, I tried this. It didn't work at all. And they've got a feel acceptable saying, this isn't going to work. I need some help. Whereas, if you have a culture that says, hey, you failed, it's your problem, that's real hard to kind of scale systems because people end up hitting a lot of brick walls on really hard technical challenges and they need help. And if they can't go ask for help because they feel ashamed or whatever feeling inside, then you can't teach them. And then the other part is just rewarding teaching, rewarding people that are the fonts of knowledge, rewarding people that have gone out and experimented and played with it, let them kind of teach people, give them an hour long lunch session where they get to say, hey, I played with this new JSON parser, or I played with this way to compile ML models to a new model format. Even if it never makes it into production, it never changes your software. The fact that your people are now thinking about these new things needs to be rewarded. Finding a way to reward your engineers who have learned and then wanna teach that gives you a culture that is okay with scaling into these hard problems and doing that in a way that can go sustainably.

[00:12:24] Joe Colantonio Love it, love it, great advice. I just want to pivot a little. I usually start off this question. I don't know why I did it today. I love speaking to founders. I'm always curious to know why start a certain company. Why did you start Batteries Included?

[00:12:35] Elliott Clark Wanted what I didn't have. I was at Facebook before it was called Meta for seven and a half years. And while I was there, they have an amazing set of infrastructure teams. And I was on the database team for a while. So you want a database, you click a button that's multi-continent, PAXOS. And you just, you clicked a button to get it. And so I knew I was leaving Facebook and was starting to explore different startups that I ended up finding out were not my path at all, right? I didn't want to build those. But in doing it, I kept running into the same problems of, okay, I'm going to build this, I need a database. Well, I kind of want some monitoring on what I'm writing, and then I want to be able to version it and all these kinds of tools that for me were just so intrinsic to how I built things. But putting all of the open source solutions together ended up taking so much of my time. And when I was trying my second or third go at a startup software, I got annoyed doing the same thing just a little differently again and again. I wanted an automated solution that gave me all of this open source that I had at the big companies with minimal friction. And so that's kind of where I got to, is I was just annoyed doing the same things over and over again to bring up a production environment. I wanted to automate that like I would have as a PE at Facebook, so just did that and did it with open source and off we go.

[00:14:10] Joe Colantonio Absolutely, so what are some of the pain points then and maybe current cloud native solutions that you're solving with this model? Obviously, you can help easy to get up to up and running really quick, I guess. Any other types?

[00:14:22] Elliott Clark Up and running. And then also, infrastructure does well when it has somebody that loves it and follows it and updates it and does all of those automations, you get a new thing. But a lot of companies just don't have time for that. The database team doesn't get days and days to try a new version and do those things for mid or small size companies. But at the large fangs of the world, you get that easily. And so I wanna basically give all of that automation to everybody and say, hey, I can upgrade your Postgres every time there's a new version for you, but we will already have done the integration tests. We will already written the automation so that version A to B goes through or SSL search. Let's make sure we always get them renewed on time or like all of those kinds of things need a team that's watching it. And I wanna give that because the way that I've seen it, given in large companies is just a bunch of automation that gets kind of sprayed to tons of computers. Why not be able to do that in open source and spray that that love and that expertise to all of the rest of the companies that are wanting to build on open source but unable to.

[00:15:38] Joe Colantonio Nice, nice. How do you keep track of all the open source packages, some that may have security things that are built into it and you've already had a package full of like, how do you track of open source projects, things that break in, things that have security risks that are introduced to things like that?

[00:15:51] Elliott Clark We have a giant registry of every, we're Batteries Included, so of course, they're batteries because I can't pass up a pun, right? So we have a giant list of all of the software and versions that go along with each battery. And so we have just a single place to go up change it, and then our full test suite runs, redeploy, upload, do all of the thing with a new version so that we can kind of test each one individually. And the hope is that we stay kind of like a rolling distribution. We stay as close as possible and push those at a weekly ish cadence to you.

[00:16:27] Joe Colantonio This is great. I used to work in a regulated environment every quarter. You have to write down all the open source up things you're using. This is great, I'm using Batteries Included. They have a list right here. There you go. All set.

[00:16:38] Elliott Clark Yeah, we wanna basically depend so closely on open source that you're able to say, here are the shoulders of giants that we're standing on and you can thank them, right? Without Grafana, we wouldn't have our dashboards or without all of these things. And so being able to, say, here's what you're running, off you go, also gives you some agency to say that, if something breaks, you can go to the Grafana project and write up patch and say here's the thing that would fix the version that I'm using off you go and that kind of close tie with open source just gives so much power and control to people that I think it's worth the community.

[00:17:18] Joe Colantonio Nice. What have you discovered after releasing what you thought would be why you envisioned what Batteries Included would be and what it's become?

[00:17:25] Elliott Clark I think I didn't really fully realize how many open source projects straight up depend on the cloud. S3 being so ubiquitous to being able to run things in production was quite a surprise to me. I kind of assumed I would get around to durability and storage as a later feature and then kind of found that like. Nope, that's not possible. You have to start talking about S3 and where do you store large files or those kind of things kind of early on earlier than I thought.

[00:17:59] Joe Colantonio How does that work with redundancy? Say S3 goes down. I guess you could use that as a dashboard while it went down because these three services that we're using are.

[00:18:07] Elliott Clark Right now, we use S3 mostly for backups. So if that happens, your backups are delayed or your logs aren't archived back to S3. We try and keep Postgres on just hardware, hard drive kind of things. So less dependencies in production is always better, and less distributed dependencies are always better.

[00:18:29] Joe Colantonio How much is it machine learning involved in this process?

[00:18:33] Elliott Clark For Batteries Included? A little less. That's, we're just getting to that right now. That was part of reason why I wanted to start Batteries Included is the tech stack that you need to get to a real well-running machine learning environment in production is huge. You need so many different things to be able to go back and find what happened, what changed, what's different, that I didn't want to put machine learning in until later. We're just now about to release a new version that will include the full Bragg stack so that you can keep it all on your hardware or in your cloud environment rather than sending your data to OpenAI or somebody else. You get to keep it all internally. That one's just coming because I just now feel like we have enough. We feel good enough about Postgres that we can use that as a vector database. We feel enough about our load balancers and networking that we feel like you can actually trace all of the calls to see where things were going, that you now feel comfortable enough to do the next step of complexity.

[00:19:37] Joe Colantonio Nice. Do you have, like, a checklist if you're going to start implementing ML into your software? You may want to try these five things first or go in this order or anything like that.

[00:19:45] Elliott Clark Yeah, I would say just I'm always a fan of going and trying, right? So for me, I love sending people to try ML from Jupyter Notebooks. It's just so easy to get started. It's so easy to get a is this working? Is this not you get to the place where you can start complaining about what's working or not means that you have some perspective into what your actual end goal is. And so for me, that's kind of it. I love those. And then a vector database is kind of become needed for larger enterprise workloads, and off you go with those.

[00:20:19] Joe Colantonio Are there any misconceptions you think teams have about implementing machine learning that makes them like, I'm not going to use machine learning for my large production environments because of X, Y or Z?

[00:20:29] Elliott Clark I'm actually not seeing them not implement it. I think the most controversial thing I'll try and say is, I don't think most teams should be shoving LLMs in most of their stuff. They don't really understand the risk that comes with that, and they don't understand the production performance implications of just asking someone to hold on and wait until the LLM is finished reasoning about things. And so seeing everyone rushing for the new shiny without really having a this is what I expect to get from it. This is the thing that will be better, not just a little bit cheaper or easier for me today. But what will actually be better is kind of one of the things I wish I could bring back and say no, but make it better. Not just a little bit cheaper and easier for you right now.

[00:21:17] Joe Colantonio Absolutely. So where do you see the future of infrastructure, especially with ML? Do you see like, I know it's hard to predict three to five years. I'm always curious to know, you seem to always be learning. Are you looking at things down the road that you think, oh, this could be something that people should stop paying attention to now?

[00:21:33] Elliott Clark Yeah, I think I would start looking at different architectures for ML, right? So right now, everyone is kind of all in the, we're text LLM with maybe some image that we can also, but ask it in text. That's the kind of machine learning that's applicable to just kind of all of society, but I've seen how powerful ML can be when it's tailored to your business. When it's recommending what you should see in your feed, when it's deciding which ad you should show, when it's choosing an automated placement group for where you should put your compute or those kinds of things are much more specialized. I see the world going to a lot more than just LLMs. And so exploring what those other types of learning or statistical learning can do for you in other areas. Being combined with LLM, right? So LLM does your describe or your whatever, and then you hand a lot of that information off to another more specialized model, I think will become a pattern that we see. But that takes a while till teams and companies can get the, how do you train a model? How do you version the model? How do push it out? How do revert it? How do keep the data tagged of where each of those things came from and what they mean, that kind of thing.

[00:22:55] Joe Colantonio Nice. I've been working on this with a lot of I do news stories on a modal context protocol. Everyone's come out with an MCP. Is this applicable at all to infrastructure, MCP? Do you see that as something that's new that could help better with infrastructure?

[00:23:08] Elliott Clark I think MCP could help infrastructure. I think MCP can help in that we spend a lot of time trying to explain things to a computer. I'm going to write a ton of stuff into my shell to explain that what I really want is a list of the running services that look like this. And so using LLMs and MCP to kind of bridge the gap of what information is out there and how do I explain to a compute what I want to see those things are much easier for humans to explain in text. I can tell you what I want, probably before my brain can even start saying, here's the five commands you need to pipe together to get that. And so being able to give that to people that maybe don't have that command line experience or they're in an environment that they're not so familiar with will open up a whole lot of new things. I think the MCP and that kind of environment will be less useful for the super experts, right, the ones that spend their entire day on one package, I doubt they'll be doing as much MCP day to day, but using it as a way to kind of interface with things that you're still learning, or maybe you don't have as much on the top of your head. I think those will be super, super useful.

[00:24:28] Joe Colantonio If someone's listening to this, like, oh, I need to give Batteries Included a try. Is there a paid version? Are there free versions? Is it like a starter plan? How does that work?

[00:24:38] Elliott Clark Sure, I try and say that I want to be the Robin Hood of infrastructure. I want give everybody what the big boys have. We wrote our own license. So under a certain number of computers, it's always free and is free by default. Then after that, we will have a paid version, but for right now, we're just completely free to everyone to go just try it out. We will be a usage base. You know if you have 700 batteries with a million computers running it we're going to charge you a lot more than someone that has you know 10 computers running it. But it's also open source right so it's on Github, if you really just want to go play around and try it, our license has a time delayed Apache to it. So after a while whatever out and been released becomes just Apache 2.0 for you so you can play with it on your computer on your own. And make changes to it just like it was Apache. So you get kind of the best of both worlds, which also means that we as a team have enough funding to come in and be that infrastructure company that goes and does those updates for you. We care about the new security patch, making sure SCO is fully TLS for you, doing those things. You get that knowledge that we have to move you forward since we're giving it all away. If it's old enough, well, also being usage based so that we don't charge you a million dollars because you went from the free tier to the pro tier.

[00:26:07] Joe Colantonio And that's probably a lot of the grunt work that no one wants to do. So it's a perfect, right?

[00:26:14] Elliott Clark Right. I think just integrating so many of those systems is the grunt work that doesn't get done well, right? A lot of people will have something where they're like, I need to get monitoring up, right? And so they get one Grafana dashboard, and they call it done. And then three versions from now that dashboard has metrics that are no longer in this new version. And no one goes back re does it because there's nothing there's no one that loves those dashboards anymore. They've moved on. There's no one that's checking to make sure the logging version still work. So that kind of integrating and checking that the integration still work is kind of what our bread and butter is what we do day to day.

[00:26:55] Joe Colantonio Like I said, there's a lot of things going on with DevOps and ML. Are there any other cool things you see that could be applied with machine learning directly into DevOps efforts that people are probably struggling with right now?

[00:27:05] Elliott Clark I think one of the biggest things that you can do to make your DevOps and ML experiences mesh well is really talk about your data acquisition for what's happening in production and tying that in to the model version that you're currently using. So I have a request that comes in. It shows some page. Tie in that page. Had model X, Y, or Z that was used to generate the help description or the whatever. Using and tying what models are running all the way back also gives you, oh, that request was super slow. Cool, then we can tie it back to a model version. Or lots of people are complaining that they're not seeing the right content or whatever. Tying that back, not just to, hey, it had some ML in there, but exactly what model, exactly where it came from, allows you to see how models are actually impacting your customer experience without you having to go do a lot of finger in the sky guess and check kind of which way is the wind blowing.

[00:28:18] Joe Colantonio How do they know those ML models? Is it because Batteries Included keeps track of that as well or is that?

[00:28:23] Elliott Clark Right now, Batteries Included doesn't do that. We don't have a lot of that kind of ML stuff tied in yet. That has to be a lot more specialized to the application. What is my application that I'm doing running and that's the kind of special sauce that companies will have to produce themselves. We hope to make that super easy, so that all the logging and all of the tracing can easily go from there. But currently, we don't yet.

[00:28:48] Joe Colantonio How much is open telemetry help with that all those logs and machine learning?

[00:28:53] Elliott Clark Open telemetry works and does a lot of that, assuming that you put in the correct metadata for it. So you've got to make sure that when you log your request ID, that you also log that same request ID on your microservice that calls out to an ML model. So it's on the developers to make sure that they tie in the right stuff to open telemetery. But if you do that and when you do that, it really pays off a lot.

[00:29:22] Joe Colantonio Okay, Elliott, before we go, is there one piece of actionable advice you give to someone to help them with the DevOps infrastructure efforts and what's the best way to find or contact you?

[00:29:30] Elliott Clark Sure. The best advice that I can have is explore new ways to test, right? Find new ways to break and find interesting things about your software. Teaches you what to expect if something happens and also can proactively get you fixing it before you run into it in production. I've had great experience with throwing fuzzers at things, throwing replay and production traffic at shadow environments. Just doing production-like runs on test things breaks so much, so go try it. And the best way to find us right now is we're on GitHub, so batteries-included. We have all of our issues in open source right there. We also have links to Slack or email me, any of those things. But yeah, GitHub's the greatest way to find us.

[00:30:23] Joe Colantonio Love it. We'll have links to everything we talked about in this episode down below. All right, before we wrap it up, remember, frustrated users quit apps. Don't rely on bad app store reviews. Use SmartBear's Insight Hub to catch, fix, and prevent performance bottlenecks and crashes from affecting your users. Go to SmartBear.com or use the link down below, and try for free for 14 days, no credit card required.

[00:31:10] Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation, testing, and DevOps. If you haven't yet, join our vibrant community at TestGuild.com where you become part of our elite circle driving innovation, software testing, and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real world challenges, we're excited to collaborate. Visit TestGuild.info to explore how we can create transformative experiences together. Let's push the boundaries of what we can achieve.

[00:31:53] Oh, the Test Guild Automation Testing podcast. With lutes and lyres, the bards began their song. A tune of knowledge, a melody of code. Through the air it spread, like wildfire through the land. Guiding testers, showing them the secrets to behold.