Preparing Dev Leaders for the AI World with Will Blackburn

[00:00:00] Get ready to discover some of the most actionable DevOps techniques and tooling, including performance and reliability for some of the world's smartest engineers. Hey, I'm Joe Colantonio, host of the DevOps Toolchain Podcast and my goal is to help you create DevOps toolchain awesomeness.

[00:00:18] Joe Colantonio Hey, is AI freaking you out, your developer, or maybe you're a dev leader? Well, you're in for a treat because today we'll be talking with Will Blackburn all about preparing dev leaders for the A.I world and a bunch more. If you don't know, Will is the co-founder of DevClarity, an AI powered one on one tool for software engineering managers and Will previously went for a first employee at a CTO at a financial technology startup that successfully sold in 2022. Really knows his stuff, really excited to get his input around AI and how it's changed things. You wouldn't miss this episode. Check it out.

[00:00:51] Hey, before we get into this episode, I want to quickly talk about the silent killer of most DevOps efforts. That is poor user experience. If your app is slow, it's worse than your typical bug. It's frustrating. And in my experience, and many others I talked to on this podcast, frustrated users don't last long, but since slow performance is a sudden, it's hard for standard error monitoring tools to catch. That's why I really dig SmartBear is Insight Hub. It's an all in one observability solution that offers front end performance monitoring and distributed tracing. Your developers can easily detect, fix, and prevent performance bottlenecks before it affects your users. Sounds cool, right? Don't rely anymore on frustrated user feedback, but I always say try it for yourself. Go to smartbear.com or use our special link down below and try it for free. No credit card required.

[00:01:47] Hey Will, welcome to the Guild.

[00:01:51] Will Blackburn Hey Joe. Thanks for having me. Excited to be on today.

[00:01:54] Joe Colantonio Awesome. Awesome. So, hey, what got you into AI to begin with?

[00:01:57] Will Blackburn Yeah, I think just as a technologist, it's something that you're always trying to do is stay on the latest trends and be looking at the new technologies that are coming out. And so no doubt we had this large explosion of large language models over the past few years. You start asking yourself, wow, what is this technology? And I've always been a user of technology. No matter what wave is coming out, whether it's crypto or this latest AI trend. I'm a big believer. And the only way to understand it is to get your hands on it and try it out. And so as I began to use large language models, really that chatbot moment that I think was a lot of people's first experience with generative AI, you're like, Wow, this is interesting. And so since then, I've continued to stay engaged with it and have been blown away by it while also have seen some of the limitations that generative A.I. has.

[00:03:01] Joe Colantonio I want to jump into some of those limitations and where you think it works. But first, so you were a CTO in 2022 ChatGPT and they came on November 20th, 22. So as a CTO, how much do you have a pulse on like when it came out, it was probably just when you were about to leave the startup.

[00:03:17] Will Blackburn Yeah.

[00:03:18] Joe Colantonio Did you see it as like, my gosh, this could have helped us or as a CTO, this really can help me with this at this company?

[00:03:24] Will Blackburn Yeah, it's a great question. We at that company, we were already going through an acquisition process and we could see that the company was going to be acquired. I'd gone from the first employee, a founding employee to chief technology officer. We had a team of 35 on the development of product team. So and yeah, I was feeling that, wow, this is coming out, Copilot was just getting released in some way or people could get it into their code editors. And I think I felt what a lot of development leaders have felt, which is on one hand, wow, this could be this really amazing tool that increases our productivity and lets us write good code. And on the other hand, while this is also scary and do we really want our code base, our data going into these models, what does that mean? And also, are developers going to use this tool in the right way or are they going to read the code that AI is outputting or are we going to really reduce the quality of our code because of AI? I felt a lot of that at the time.

[00:04:34] Joe Colantonio Gotcha. So for current CTOs, what would you recommend then? What's the approach you would give them to? Because they probably get pitched all the time with the AI promises, like I don't know now at this point what's real and what's not?

[00:04:47] Will Blackburn Yeah, I think the world has changed a lot in 2 years. And I think now the models and the way that these tools are available, you absolutely should be exploring these tools and using them inside the development team. And that doesn't mean that you just give everybody a subscription and you turn away from it. It means that you work with your developers, you work with your other development leaders, you work with your DevOps team. We're going to have to be supporting all of this code that is getting pushed out by the software development team. And you establish best practices in your organization around the right way to use these tools, what tools you're going to use. And then you're constantly learning from them. One of the main suggestions I make when we're talking to the development leaders is set up a Slack channel that is specific for A.I coding and tell your development team to put in your wins, but also the challenges that you had with a I like. Where did you see AI go off down some rabbit hole that was just a complete waste of time and was going to produce the quality of your code or our technical debt. Share that and have other developers add to that so you can all learn together.

[00:06:06] Joe Colantonio Love it. Awesome. If I'm a developer, then I'm listening to this. Obviously, they've probably been using codepilot for a while now and all the other things. How do you see changing or affecting software developers or software development?

[00:06:21] Will Blackburn Yeah, it's a great question and I think that the change that's happening in software development has already begun and we're going down a path that we're not going to come back from. I like to tell this story that there's this developer and she got told by her boss that tomorrow we've got this new tool that we're going to be able to use and it's going to change the way that software development is done forever. And you're going to be able to be more productive and write better code. And so what's that developer taking? She is worried about her job. She's worried about what's going to come tomorrow and how that's going to impact what she does day to day. Maybe she liked the way things were done before. And so this developer, she goes home that night. She doesn't get much sleep. She comes in to work the next day and there in front of her is magnetic tape. And this is actually 1955. And her company has just moved from punch cards to magnetic tape. And I tell that story because while we never had this transition from not being able to use Generative AI to Generative, AI, being such a core part of software development, software developers have gone through transitions before, whether it's from punch cards to magnetic tape or high level programing languages, which let's so much of the underlying machine language could be handled by compilers and not by us. We've gone through transitions like this before. And so while this is a change and it can be scary, I would say we should lean into this and it is going to change software development forever.

[00:08:12] Joe Colantonio Yeah, for sure. Definitely agree with that. Do you think it's under hyped, overhyped, or properly hyped? I think it's under hyped because I think it's going to be a paradigm shift. And I think a lot of people are burying their head in the sand kind of, maybe I'm wrong, but that's the pulse I get.

[00:08:27] Will Blackburn Right. I think when you look at it from a headline standpoint or a news standpoint, like that's maybe where it's overhyped. But to your point, you like you get to talk with development DevOps leaders all the time that we serve development leaders as well at have clarity. And when we talk with them, I am surprised at how slow people are moving in some of these things, even just to try it out. I'm saying, you should be using AI agents right now to write all your code. But it does feel under hyped in a lot of ways. And I think to your point, this is a big paradigm shift. And whether we like it or not, this is changing the way we're going to write code. And so I completely agree with you. When I talk to developers, they don't want accept that and a lot of circumstances.

[00:09:17] Joe Colantonio So who you think's in a better position for this, though? You would think it would help junior developers. But it almost sounds like I saw news stories. I don't know where it was where top college grads with computer degrees used to always be in demand and always get a job. And now for some reason, there's a lot of resistance. I know if it's the economy or if it is A.I.. I guess what I'm asking is where do you see like how it's going to do you see and impacting newer developers or maybe is it going to really be a good benefit for people that already know development and make them more like higher level developer leaders almost?

[00:09:50] Will Blackburn Yeah, it's a great question. And when I've studied this topic and again, talks with people are much smarter than me, I've kind of developed like two ways of thinking about this. I think one of the interesting things when we just think about generative AI in general, but it also applies to code output, is that they've studied who this helps the most. It really helps bring up that average player to that A-plus level. Like that is where you see most of the gains. And of course there's diminishing returns at the top. And so if you already have an excellent developer, you know, it's harder to get more of a return out of them. I think that that is just an interesting thing to be thinking about. But then we do have to think about like how this is going to actually be used. And one thing I have heard is that development leaders, CTOs like myself in my former role, they are more reluctant to give AI to a junior developer and let them, go loose with it and start writing a lot of code. And I think that right now is what is hurting junior developers. Now, I think that that will change over the next six months as companies figure out the right way to roll this out to their organizations to institutionalize best practices with it, to make it part of the Dev review process, that DevOps process, if you're tracking a door like metric, like how can you understand what AI is doing to those kind of key output metrics that you're tracking? Once that happens, I think junior developers are going to be seen as that, hey, maybe they're not, the skill level that a senior is. But if we augment their work with A.I., we can actually get a lot more out of them. And then, the last point I would make is if I was a junior developer, I would be looking at this as like the greatest learning opportunity of all time because if you're using a tool like cursor, which is an AI enabled editor and you figure out how to use it well, where it can understand your code base and have the context there and give it official documentation. And instead of saying like, Hey, I just wanted to write code for me to say like, Hey, I wanted to be a personal teacher to me as I learned this new pattern, as I think about antipatterns to avoid is just an amazing opportunity. Instead of having to go to StackOverflow and copying an answer from two years ago, now you can get something that's generated for you, for your code base and then have official documentation cited by your editor. That's incredible.

[00:12:35] Joe Colantonio I love that. I was actually spigots. A good friend of mine, he's been software developer, he's a CTO now for over 30 years and he talks about Cursor how he used and he was blown away by how quickly and easily he was able to get up and running with an application. Here's someone that has 30 years of experience developing and he says that he omits it. How do you make sure junior developer doesn't get discouraged, I guess? Like you said, it's a great opportunity, but how do you not see it as, my gosh, this is like I'm going to be replaced. Why even do it? Why even learn?

[00:13:06] Will Blackburn Yeah, it's a great question. And like I can say all those things and then, I'll talk to a developer in person that can't find a job. And so, my words don't mean much when I can't get them a job or help them land a job right now. I again, optimistic on this as a whole. And I don't think it's going to be something where in the short term. You know they're going to get a job to only get replaced a few years later. I think that you do have to lean into this, a question that gets asked a lot or something people like to say is that AI is going to be like a net job creator. That's fun to say and it's nice to say and people like to hear it. But when you zoom in and you look at the individual, people are going to get displaced and it's going to change industries. And so you've got to be as the individual, you have to understand this technology and then use it as a differentiator. And that is what I tell the people that I meet in person, which is, hey, find a way to differentiate yourself in the market and use AI as a tool to help you do that. And then go explain to that CTO that is blown away by cursor and say like, Hey, I'm a junior developer, let me show you how this is helping me level up. Let me show you some of the limitations I found in that and how I'm applying this as a junior developer who hasn't been out there in the market and seen a lot of things yet.

[00:14:28] Joe Colantonio All right. So I think we touched on a lot of things. But for example, chat should be really in its form now. It's been only around for two years and it seems like we've had a wave. Do you see the wave continuing or do you see a coming down now? So in the next 6, 12, 3 years, it's more now of how do we now leverage this or do you see it like I guess I hate asking these questions because you can't see the future per se, but where do you see us maybe in three years, you see it just us now adopting to this world, or do you see evolving even more so in three years like I have no idea at this point where it's going.

[00:15:02] Will Blackburn Yes, it's a great question. So one way I think about this question is like, okay, how much are we going to see the models continue to evolve? And then kind of to the second part of your question there, what's the implementation going to look like? How is our workflow going to change? And I'm a believer that like even if we're like done from a model evolution standpoint, we have more than three years of work to do to figure out how to use this and really impact our workflows. I think there's a lot of interesting things going on right now. You've got things like copilot which are just extensions inside of your editor, and then you've got things like cursor, which is a step up from that and that it's a full editor that has been AI enabled. And you start to get into these agent, these agentic workflows where they do more things for you without you having to be involved in every step of the way. We've got new tools like Bolt, Bolt.new or v0 by Vercel where these are app builders where you can build a full stack app into ..... These are just examples of some of the ways that we're using generative AI right now in the coding process. And I think that even if the models stop evolving at this point, we've got to figure out how to use these better, how to make it more functional for the developers, the things that developers hate about it, where it starts to hallucinate or uses a class that doesn't even exist inside your code base. Like there's still so much opportunity to improve the workflow that we have right now. I personally don't think we're done from a model evolution standpoint. I'm interested in seeing what happens when we get more code specific models, if that's necessary those are just some of the interesting things that I'm thinking about. But yeah, I think that really what we're going to see over the next few years is how does it just become a part of everyone's day to day SDLC software development lifecycle process?

[00:17:10] Joe Colantonio 100%. You mentioned DevClarity a few times. Obviously, you are CTO and then you start a company. I assume you start a company due to maybe problems you've been seeing and how it could solve AI. I guess before I assume things, what is DevClarity?

[00:17:24] Will Blackburn Yeah. Thanks for asking. DevClarity really came out of my experience as a CTO, so I went from founding employee and just a software developer to eventually being a small team lead of 5 developers and then 15 developers and finally Chief Technology Officer managing 35 developers. And I made a lot of mistakes in that process and luckily had a great leadership team and great developers to help me learn and grow from those mistakes. And I encounter a lot of pains as a development leader and really like thought, Wow, it's amazing how much work I'm doing and Excel and OneNote. And so from that experience set out to create a platform to help development leaders make that transition from being a developer to being a manager and then to help development leaders up the chain to better lead their organization. Today, we help engineering manager that kind of first level software development leader. We help them better lead their individuals on their team with a dev specific 1 to 1 notes that really creates structure. And oftentimes you've got a developer who's spent 10,000 hours talking to a computer and giving them commands, and then the next day they're a manager and now they have to talk to humans and try to get them to do things. And it's not easy. Humans have a hidden compiler where you can't always be sure that the code that you gave them, the instruction that you gave them, was processed in the same way that a computer was processed. And so we are really creating a platform to help in that journey and to give them structure and leading their team. We believe that with AI, developers are actually only going to become more important when companies realize that they can do 3x, 5x, 10x more and there's more opportunity, they can serve their customers better than ever before because they've got these large language models to help them better understand what's going on. We believe that deaf leadership will only become more important because the output of developers is leveraged and managed by these leaders.

[00:19:46] Joe Colantonio Alright cool. I want to talk a little more about that, but I was talking about that like how does as a leader or CTO, how do I start measuring performance of what metrics I use for my team with AI now? Because you just mentioned 3x2x. Should I be looking at okay, I have developers, I know before AI they were producing this amount of code. Now, with AI they either should like is that a metric you take up you should start using or any type of metrics that you would think with AI now is really going to change?

[00:20:14] Will Blackburn Yeah, that's a great question. One of the things that we have really tried to do is understand how developers are judged and how development leaders are judged. If we're offering a product to them, we need to understand like, how can we say that you're going to get a return on using DevClarity and pretty much universal. The thing that we've heard is that development leaders are just judged on outcomes and those outcomes are usually delivering on the roadmap. If you're creating any sort of product, any sort of service, that's really what you're judged on. You've got things like Dora metrics and different tools like that that kind of talk about the health of the organization and lets you see things that happen. But it's just it's really tricky to gauge developer output for a metric standpoint. And I think that's why so many leaders use outcomes such as, you said we were going to do this by then and did we get that done? And so I think as AI comes out, there will be even more of a request from leadership, whether that's technical or non-technical leadership, to say like, Hey, how can we measure our output? I think things like Dora is a good place to start. I am always very critical is not the right word. I think looking at metrics from an absolute standpoint is the wrong thing to do. I think looking at relative metrics and trends over time is the right thing to do. So if I can start measuring, what is are committed versus completed over a sprint look like for how many story points we said we were going to do. If I can start measuring that now, I'm more interested in the trends that happen over the next few months, over the next six months, as we adopt A.I. to give some sort of narrative around how this is helping as opposed to saying like, I'm going to look at an absolute metric like PRs, merged and make a judgment off that. I would much rather look at the trend and then you can look at things relative across particular teams or particular team members. And you've always have to lean on the leaders to provide the context around those numbers.

[00:22:34] Joe Colantonio Great insight. So I guess that implies not only its metrics, but is there risk to over relying on say AI for inter-personal aspects of team dynamics? So you see a trend, but you have to have discussions, I guess, with the developer as a human to see what's going on. Do you see people all over relying on the AI maybe to make these types of decisions?

[00:22:52] Will Blackburn Yeah, I think what we've seen is that really the only thing that's been available to development leaders have been quantitative things. It has been how many commits people are making, how many pull requests they're closing, how many comments they're leaving, how many reviews they're doing. And so it's always been this quantitative thing. What we're trying to do with DevClarity is actually use A.I. to bring in some of the qualitative things that development leaders just don't have good visibility into right now, or that it's really time consuming and manual to do. And so if we can take your 1 to 1 notes over the past 90 days and if we can provide you context around trends, sediment , different things like that, then that is something that's now a new tool to help you have a more complete picture. Don't just use the metrics and also, don't just rely on an AI generated sediment trend. But when you start to put that together, it begins to give you a complete picture and it builds confidence in yourself as a development leader that you understand what's going on, that you know where to drill in and where to spit your top ice.

[00:24:03] Joe Colantonio Nice, could it also help get insight to maybe a team that's not as forthcoming. You know how you have like us every sprint? Like, how happy are you? Like thumbs up or thumbs down. But then if you have note and everything, you can try to say, okay, they're saying thumbs up. But I'm noticing that there's an issue here that maybe isn't being bubbled up to me, that maybe you should help not to be punitive, but maybe to fix before it becomes an issue.

[00:24:26] Will Blackburn Yeah, absolutely. Again, two thumbs up, thumbs down. Other metrics are great. Oftentimes they're lacking indicators and not leading indicators. And so that's another great thing that you can get if you have structure around talking to your individual developers and seeing that information. Oftentimes, the developer in the project knows what's going to go wrong before anybody else does. And so if you've got a structure and a process to help get that information up to the development leader, then they can help that developer make the right call, make the right decision unblock something for them. And then to your point, you can really marry the qualitative and quantitative data, like, okay, this sprint, we had 30% more thumbs down. Well, why? Now, if I have qualitative data that I can put with that system, I can see what happened. Maybe somebody was off for vacation and we knew about it, but we didn't adjust the story points that we were going to assign to them. So that crunched the rest of the team members. Or maybe it's a problem vendor that just you always run into issues when you're working on that vendor's integration, that type of information can flow out of the system much more clearly.

[00:25:44] Joe Colantonio Nice. I always love speaking to founders of solutions and products. I'm just curious to know when you created DevClarity and now that it's in the Wild, is anything you hear that's feedback like, Well, I didn't think of that or wow, it really helped his team do X that people may get a sense for like all right sounds cool but like any case studies or any other feedback that you be getting how it's helped teams.

[00:26:06] Will Blackburn Yeah I think, probably the thing that surprised me the most is as a founder of a solution that you've built kind of for yourself, like I experienced these problems, I was thinking so much about me as the individual, and we've got a company that we put out a case study for, and they're managing about 35 developers, which is very similar to my team and my last company. This is part of an organization, and the thing that surprised me was how much they said it helped their dev leadership team better communicate with each other. And so a cool side effect of the platform is that if you're an engineering manager and you report to a VP or a director, you can actually tag them, you can tag them in an issue that's come up with a particular developer or with a particular team. And they said like that was a game changer for them when they were able to surface issues from the bottoms up and really say like, Hey, I know we've always kind of heard murmurings about this process or this issue that always runs on Tuesday nights, that disrupts our sprints. And now here is like a very clear statement from a developer about that. Being able to share that or share a mocks the engineering managers team. That was a really cool side effect that was kind of emergent and our platform, we didn't think too much in designing that feature in it and really we were led there by our customers.

[00:27:35] Joe Colantonio Nice. All right. Here's a little switch up here. Just came to my mind here. All right. So AI, you were founded up piece of software, obviously, it's very cool. It does a lot of things. I'm not saying it's easier now to create software, but as a founder, as one that's created something, how do you not have be worried that someone's going to use AI to say, Hey, build me an application like DevClarity and then you lose all your IP or something like that. How do you stand out if someone's like, I built software now, not only as a developer my afraid of either, but now as a company, I'm afraid because it could just mimic what I'm doing and kind of recreate it.

[00:28:09] Will Blackburn Yeah, that's another good question. I'm a first time founder, but I was founding employee in the last company and I was there day one. And that experience and over the past year of getting DevClarity off the ground, it just it really shows you how much execution matters and that ideas don't matter that much. And not only does execution matter, but having clear purpose and a vision for the future and a mission that your own is just required to be willing to do the hard work that it takes to build a company and to grow your customer base. And so you become less worried about people stealing your idea or running off with something. Sure, there's a threat out there, but I don't believe that anybody else is on the exact mission that we're on and has this set of unique characteristics that we are putting together to form a unique group of values and mission. I don't think that there's anybody else out there that's sure somebody could come and copy what we're doing. But six months down the road, we are marching in a way, in a direction that's going to be really hard to continue to copy.

[00:29:25] Joe Colantonio Great advice. Okay, Will, before we go, is there one piece of actual advice you can give to someone to help them with their A.I DevOps efforts and what's the best way to find contact you or learn more about DevClarity?

[00:29:36] Will Blackburn Yeah, I think I always like to leave with a piece of advice which is use the tech, use the tech, and that's how you're going to understand it. Do the work to form an opinion. And maybe your opinion is we're not ready for it or this is not good for this reason and that's okay. But make sure that you've done the work to form an opinion. And through that, you will learn a lot. And I think it is the only way that you can really set yourself up for success in the future is by spending the time to use the technology to try it out, to see what's good about it, where the limitations are, and ask yourself, are those limitations always going to exist or are they today's problem that will be solved in a month from now. That is my advice and recommendation. And then I am active on LinkedIn. I post most every day and said that's the best place that you can get a hold of me personally. Should have direct messages opened up where anybody can message me. And so LinkedIn.com/in/WillBlackburn and then you can find more about DevClarity and also contact me through DevClarity@devclarity.ai

[00:30:54] And we'll have links for this and much more in the comments down below. All right, before we wrap it up, remember frustrated users quit apps. Don't rely on bad app store reviews. Use SmartBear's Insight Hub to catch, fix, and prevent performance bottlenecks and crashes from affecting your users. Go to smartbear.com or use the link down below and try for free for 14 days. No credit card required.

[00:31:20] For links of everything of value we covered in this DevOps Tool Chain show. Head on over to testguild.com/P175. And while you're there make sure to click on the SmartBear link and learn all about Smart Bear's Awesome solutions to give you the visibility you need to do the great software that's smartbear.com. That's it for this episode of the DevOps Tool Chain Show. I'm Joe, my mission is to help you succeed in creating end-to-end full-stack DevOps toolchain awesomeness. As always, test everything and keep the good. Cheers.

[00:31:53] Hey, thank you for tuning in. It's incredible to connect with close to 400,000 followers across all our platforms and over 40,000 email subscribers who are at the forefront of automation, testing, and DevOps. If you haven't yet, join our vibrant community at TestGuild.com where you become part of our elite circle driving innovation, software testing, and automation. And if you're a tool provider or have a service looking to empower our guild with solutions that elevate skills and tackle real world challenges, we're excited to collaborate. Visit TestGuild.info to explore how we can create transformative experiences together. Let's push the boundaries of what we can achieve.

[00:32:37] Oh, the Test Guild Automation Testing podcast. With lutes and lyres, the bards began their song. A tune of knowledge, a melody of code. Through the air it spread, like wildfire through the land. Guiding testers, showing them the secrets to behold.