How to Create Testing Magic with Zero-code with Shubham Jain

[00:00:04] Get ready to discover the most actionable end-to-end automation advice from some of the smartest testers on the planet. Hey, I'm Joe Colantonio, host of the Test Guild Automation Podcast, and my goal is to help you succeed with creating automation awesomeness.

[00:00:25] Joe Colantonio Hey, it's Joe, and welcome to another episode of the Test Guild Automation Podcast. And today, we'll be talking with Shubham all about how to create testing magic with zero code using a tool I just learned about. And I think it's really going to help you with your automation, especially with API testing. Shubham is actually the maintainer of Keploy, which is an API testing platform. API testing is becoming more and more popular, and I think this is a solution that's going to help everyone be more efficient with their API testing. He is the former consultant at AWS India, contributed to XWiki, and he also has led data and platform engineering teams across the world. He loves talking about developer productivity and next-gen problems and solutions, and he's going to share one of those solutions with you today that I think you really gonna get a lot of value from. You don't want to miss this episode. Check it out.

[00:01:14] Joe Colantonio Hey Shubham, welcome to the show.

[00:01:17] Shubham Jain Hi Joe. Excited to be here. Thanks.

[00:01:19] Joe Colantonio Great to have you.

[00:01:20] Shubham Jain Thanks for having me.

[00:01:21] Joe Colantonio Yeah, absolutely. I'm excited to learn more about Keploy. Before we get into it though, is there anything I missed in your bio that you want The Guild to know more about?

[00:01:29] Shubham Jain Well, I think that covered most of it. Something I think interesting would be I've been part of Google's Summer of Code. It's a very interesting program run by the team at Google. It got me to open source, both as a contributor as well as data summer mentoring for Google Summer Code and XWiki. So I think yeah, that has been really phenomenal. And the team at Google Summer of code is pretty cool. I really enjoy, still part of them, part of the program with Keploy.

[00:01:55] Joe Colantonio How do you become part of that program? Do you submit an application and then they go through it?

[00:02:00] Shubham Jain Yeah. So you can apply as an organization. And once the organizations are picked. Only, it was only open to students. So students apply. Now I think it's open to more people. Then the organizations pick based on the contributions and the student's profiles.

[00:02:14] Joe Colantonio Awesome. So here's a dumb question right off the bat. What's the difference between a maintainer and a contributor? Since you are the maintainer of Keploy. You are the lead guy that says no or yes to the final build?

[00:02:24] Shubham Jain Kind of yes. Yeah. The maintainers obviously have more access, more involved in the decision-making process. Once you've done a certain level of contributions and you control a big part of what's going on. So yeah, that's basically the difference between a maintainer and a contributor.

[00:02:40] Joe Colantonio Awesome. So what are you excited about Keploy that you want to be the maintainer and contribute to it?

[00:02:45] Shubham Jain Well, so my team started it. Since we started we had to be the maintainer to enjoy.

[00:02:51] Joe Colantonio Why did you create it then? What was the main issues or are they like these open source solutions everywhere, are there solutions vendor base but not meeting the need of this particular pain point you were saying?

[00:03:02] Shubham Jain At my previous teams, what we were doing was we were mostly focused on R&D building out, trying out new ideas, taking things from ideation to users in less than two weeks. And that's when the way our team worked, and not essentially the way we worked back then was we would change things. We would deploy nearly on a daily basis. Whatever was out there and whatever we could try in terms of testing, it required a lot of effort. We were like, okay, we want to test our application works. We want to make small changes and deploy immediately. So can I just use my production traffic or what my users are doing and validate if I broke anything really quickly that eventually led to Keploy? Turns out everybody has. I mean, it's a very common problem. Everybody wants to make small changes and validate is this application still works. And yeah, so we open-sourced it and now here we are.

[00:03:55] Joe Colantonio And if someone asks you hey what is Keploy? What does it do? What's the main tagline or the main feature that it provides developers or testers?

[00:04:03] Shubham Jain It's the only EVP-based solution which can capture like live traffic and generate test cases and data mocks, which you can run a test. Like I said, It's all open source.

[00:04:13] Joe Colantonio Completely open source so everyone can use this. And is there a paid version or no? Okay, nice.

[00:04:18] Shubham Jain We are working with select companies as beta partners to try out deploying more complex scenarios where we have streaming or we have async workers and things like that. But yeah, as of now, what's open source is mostly open source. It's all built on top of this.

[00:04:34] Joe Colantonio Nice. I'm trying to visualize this for the people that are listening here. Is it like a sniffer in production and it actually listens to real traffic for your application without you guessing? And then it will generate API tests for you that mimic exactly what's happened in production. Is that the main use case?

[00:04:51] Shubham Jain Yeah, exactly. That's what it does. It can be anywhere, right? It could be somewhere on a dev's laptop where I'm developing an application I can call the API, and so generate a test or in any live environment or a production environment where you could do something like that.

[00:05:05] Joe Colantonio And this is important because as we go cloud native, we may be using services and APIs that we don't even know about. Like we may have a feature on our application like submit and Lord knows what it's calling behind the scenes. So actually having this in production is able then to create those tests for you and know what endpoints to interact with automatically.

[00:05:24] Shubham Jain Yep, that's exactly right. What's crazy is we have 15 users who have been using different SDKs, which they did not make. Turns out a very simple operation was doing like 50 HTTP calls or 50 database queries. Those were largely hidden. And the tests that they wrote earlier were mostly using mocks. So they would mock the entire interface out and never know about it. When you use Keploy, you're able to see everything that the application does. Record everything like database queries, and external calls, and rerun them exactly same. And see if the application still works. So apart from testing, it gives you a deeper visibility into what should be happening.

[00:05:59] Joe Colantonio All right. What do people have to do to get this? So they have to like instrument their application with open telemetry or something. So you can get this type of data in production, like what needs to be implied before they can use a solution like Keploy.

[00:06:10] Shubham Jain Nothing really. I mean, so something I mentioned eBPF. We use eBPF to trace system calls and instrument applications without any code changes. So using Keploy just you download Keploy. You do Keploy record and just run your application. And that would generate the test sandbox.

[00:06:26] Joe Colantonio All right. So for folks that don't know, what is eBPF then? An interpreter for languages or something?

[00:06:32] Shubham Jain eBPF is Extended Berkeley Packet Filter. It allows users to trace system calls. So for example, if I'm using Linux, my application would be using system calls do different things like repacket, execute any piece of code, and get access to memory to network. So using eBPF, we can instrument or change or understand the application behavior without actually injecting the SDK inside the application build. Traditionally, it has been used for observability, especially by companies like New Relic. They have a project called Pixie. It's also been used heavily in security because you can observe applications without needing to change the code. So yeah, we used the same technology to go one step further, not just observe but also simulate. And yeah, and so kind of use that for testing.

[00:07:22] Joe Colantonio Nice. So I know one of the benefits is eliminates the need for data dumps. Like through data dump stubs or mocks for dependencies. How's that done? So you're using a third-party dependency. It's in production. It's obviously, interacting with it. But then you go to staging a development and you're like, I don't want to use that service. It's going to charge money or it's not available. How does it get rid of mocks then or not in production?

[00:07:42] Shubham Jain Yeah. So it's simple. Once you record something, Keploy essentially acts like a proxy. You can capture from any environment, let's say production. And then you can rerun it literally anywhere. So essentially when you use something like Keploy, environments don't matter at all. You could capture different things from different environments, run them together in a single environment, and all that environment would be running in the application in Keploy. So no databases, no external services, no internal services. Typically what this helps people achieve is they're able to within five minutes run maybe 500 integration tests in their CI. And if you use something like GitHub actions, you just the entire thing runs inside of actions container. So you don't need any environment to run any of this. So yeah like a use case.

[00:08:29] Joe Colantonio Nice. Who's the main use of this is it would be a developer before they do a main commit they would run it through spin up a I guess an environment run all the tests. Okay, green promote to the main build?

[00:08:39] Shubham Jain I think that's the most popular use case we're seeing as of now because Keploy also integrates with integration testing libraries like JUnit, Pytest, and CodeTest, adds to your code coverage. So essentially, normally writing an integration test is hard because it requires effort, plus it's slow, it's expensive. Keploy kind of helps you create those integration tests that also add to your test coverage, which are insanely easy to make. You just need to use the application. As a developer, I can run it as part of my CI, but while that's the most popular use case you're seeing as of date, I believe it can be used further in other use cases. Maybe debugging or augmenting the testing process, especially when it takes a lot of time to write a lot of time to execute the existing tests in any testing infrastructure.

[00:09:26] Joe Colantonio All right. Let's dive into that a little bit more, how you can use it for the testing process. So you'd run it in production. You'd have your tests. How would you know what's being covered or how much coverage you have? So you have existing functional UI test in there slow and like, oh boy, we could probably get a lot of this API testing, but I have no idea what is being touched by this UI. How do you know? Okay. I'm good. I have these 50 API tests and I know that they can replace these lower or slower-running UI tests.

[00:09:54] Shubham Jain The way that works is Keploy would collect records test it can also capture the coverage information that okay what coverage in my code is this able to achieve. What Keploy can then do with it can use that and figure out okay, what are the top 50 test cases from, let's say 10,000 or millions of test cases that I should run? Yeah, I mean, that's what I can prioritize and set up running billions of requests. And since Keploy is able to also capture the data calls, it can actually isolate those 50 requests and actually run them in a lower environment. So that's basically the very, very high-level idea.

[00:10:30] Joe Colantonio Do you have any use cases of customers? I've done that. And you're like, wow, that actually saved me a lot of X amount of time. I didn't know you were able to do that.

[00:10:38] Shubham Jain Yeah. So we keep hearing about different users from the community. For example, one would be Nutanix. So traditionally they would need to spin up virtual machines. And Nutanix has their own data center products and they have their own managed services. So all of them take time to spin up. So one interesting use case for Keploy is that that traditionally, I mean, that takes time and effort to maintain as well as run. They were able to step out most of that and use Keploy as a proxy server and run the tests in like miliseconds, compared to something we should take like hours because it has to orchestrate everything and then do the validations.

[00:11:18] Joe Colantonio Does this save money on environments then or containers or runtimes, because you don't necessarily need thousands of containers spinning up and spending a lot of time being open? Does that make sense?

[00:11:29] Shubham Jain Yeah. Yeah. Yeah definitely.

[00:11:30] Joe Colantonio All right. What are some other features then that we may have missed? You mentioned debugging. How can help with debugging? Does it bubble up insights or does it like how hard is it to get the data? I'm sure it collects a lot of data. And then you're like, okay, like how do you know what's relevant? What's not?

[00:11:45] Shubham Jain Yeah. So it's quite simple. Since it's again, depending on how somebody configures Keploy. So if I would use Keploy to capture something like capture tests in a live environment, and I would like to debug a particular request, I just need some way to get some kind of request identifier, maybe some kind of trace ID that we see in Opentelemetry. I can search that in Keploy. That shouldn't give me that exact particular request along with everything that happened with respect to that request. So yeah, that would essentially help me isolate that particular thing and replay for debugging.

[00:12:18] Joe Colantonio Have you seen people use this for performance testing to say, hey, how does this service handle billions of requests? I don't know, almost transaction-based. Well, actually having to go into production doing it the nuts like that.

[00:12:31] Shubham Jain Yeah, that's a use case I've been thinking about. But I think that as well as chaos engineering, chaos tests, and doing different instances of application and see how it acts under stress. I mean, we have not done that yet because we feel that the tooling needs to mature further before we can get into performance testing, because I feel we would need to really because we are emulating dependencies or emulating databases. The ideal way this would work is when you record, you would also need to record those failure scenarios. Or we would need some way for Keploy to emulate or make a database failure set of failure goals. So those things can be explicitly done. But I think it's still early stages for Keploy for that particular use. And I think it's a long way to go.

[00:13:17] Joe Colantonio Is it possible to overuse deploy where you just completely rely on it and say, I trust it completely. I don't have to do any full-end testing on a full-blown environment. I don't know. Is that a concern?

[00:13:30] Shubham Jain I don't think as like in the current state, it can replace everything that we have currently because there's definitely value in testing with real endpoints. I feel the value is more towards getting a quick response and being able to cover a lot more use cases than just core flows. So yeah, I think end-to-end tests, having real network interactions, those are valuable. But people tend to prioritize the core flows because we have a lot of stuff to do. I feel there something like Keploy can add value. But yeah, there's definitely still value in running real network interactions.

[00:14:06] Joe Colantonio Yes. I'm thinking like my old job, developers don't want to wait 3 to 4 hours to end test to finish before they get the okay to submit. So this seems like a good compromise. We run all these tests. Looks good. Let's put it in the next stage of the pipeline. Then we'll run maybe our end-to-end test overnight or something to do the extra coverage. So it seems more like an assistant rather than a replacement, then?

[00:14:29] Shubham Jain Yep. In fact, we have seen use cases where the way the team started using Keploy was they would convert the existing tests to Keploy test and the Keploy test would run much faster, like in minutes compared to hours. And then they started adding more tests to Keploy. So essentially, most of the scenarios that would be caught by the regular testing pipeline would caught by Keploy first, unless there's something very weird happening in the network. Something which has not happened before at all. Maybe you updated the database or something external changed until something that very weird thing happened. It would be caught a lot earlier and a lot quicker. And then there's always the other pipeline.

[00:15:05] Joe Colantonio How do you convert test to Keploy? Yeah, the existing test like you just mentioned.

[00:15:09] Shubham Jain Yes. You run the existing test suite and you record everything with Keploy and yeah.

[00:15:15] Joe Colantonio Cool. It's that easy. So it doesn't matter if someone used Selenium, Playwright, or Cypress, it's just going to pick up the traffic and almost act like a sniffer. Like I said and automatically generate.

[00:15:24] Shubham Jain Yeah.

[00:15:24] Joe Colantonio Nice.

[00:15:25] Shubham Jain Yeah, exactly.

[00:15:26] Joe Colantonio You mentioned it's not fully mature yet to handle. Maybe performance testing may not even be something that's on your roadmap. Anything like AI or anything like that. You see being baked in in the future, I know. Yeah, it's a hot topic. Just curious to know if you're messing around with anything funky over there right now.

[00:15:41] Shubham Jain We have done a lot of experiments. Most LLM models today, actually, they're not really reliable as a test generator per se. So they could generate something really funky, something they would imagine libraries. They are definitely not reliable. But what they're really good at is reasoning. And given a particular input, they could probably think of some really interesting edge cases. Something I've been experimenting with and trying with some of a beta customers is okay, you're using Keploy to capture and generate test cases. Normally, you would have to do it yourself or get a like user traffic to do it. What if you could take your API specs, Swagger, Open API 3, give that to an LLM model? Ask it to generate different edge cases based on that and use Keploy to actually generate the real tests. So that way the syntax will always be correct. The mocks which are recorded by Keploy are real interactions which happened. And Keploy can also, based on coverage, filter out all the noisy and in faulty tests. I think that can be a combination we're trying. That's where we are with AI.

[00:16:40] Joe Colantonio Awesome. I saw on the website something called Keploy's accurate noise detection feature. What's that?

[00:16:46] Shubham Jain Oh yeah. We've had this from the beginning. What we do is whenever we're recording something a replay, it could be noisy parameters. There could be timestamps, it could be something which is what we always change regardless of application behavior. What Keploy would do is a good run. Whenever it would capture something, it would rerun it again. But when it's relaunch, it will mock everything out. Similar to what it would do during testing. And then it would assert, okay, did anything change? Are the timestamps different? Are the any field were changed? And I know the application is still correct because it's the same application, same version. So that way I can find noisy parameters not compare them during testing.

[00:17:25] Joe Colantonio Awesome, awesome. You mentioned CI/CD, how hard is it to integrate? It sounds like easy but I don't know, like how hard is it to actually get it into an existing software development lifecycle to plug it into as a plug-and-play? Basically.

[00:17:38] Shubham Jain It's a plug-in-and-play. Just add the Keploy test as a stage. Just one line and they are good to go.

[00:17:44] Joe Colantonio Nice. I always ask when it's open source. How's the community? Sometimes you get great solutions, but you're like, oh, this community's dead. It's going to die. I'm not saying that's gonna happen with Keploy. How's the Keploy community? How active is it?

[00:17:56] Shubham Jain I think it's really interesting so far. We get different kinds of users. Most of them want to maintain test coverage because that's part of the job. Recently got a lot of community members from Japan. I think there's a Hacker News equivalent there. And somebody mentioned Keploy. And yeah, we got a bunch of users from there. I think so far the the community hasn't really good. We got a lot of feedback. And yeah, so far great, I think the community is still there.

[00:18:23] Joe Colantonio Being a contributor and a maintainer yourself, how do you get people to contribute to open source? I mean, a lot of times people make complaints in the community, but it's like not many people actually take up the hard work of actually implementing something. Do you have any tips on how to get started contributing, or how hard is it to contribute to the Keploy project? If someone says, hey, I'm a master developer, what can I do to help around here?

[00:18:45] Shubham Jain What we typically do is we would mark, we would label solutions and good first issues. Whenever we get an issue from a community user, we'll try to break it down into different tasks, which are relatively small and can be done by somebody who is not an expert. And Keploy is also written in go. And we focused a lot on readability. So contributing is typically relatively easy. We already document most of our methods and ensure that okay, if somebody wants to contribute they can go to the good first issues label. They can get started. So we get a lot of contributors, especially during GSoC and Hacktoberfest.

[00:19:19] Joe Colantonio Awesome. So applications are always changing. How do you make sure your tests are up to date in Keploy? Is it just a matter of do you regenerate tests often? And because it's so fast to create tests like how do you make sure your test a fresh up to date, I guess with the latest and greatest releases?

[00:19:33] Shubham Jain Like I said, since it's easy to create, they're easy to update, so if something fails and you would get a diff, the user can decide, you know what? If this is a new expected behavior, you tell Keploy to normalize it. And yeah, that's it. Your tests are updated.

[00:19:46] Joe Colantonio Nice. Cool. If someone's listening to this like, oh, this sounds great, but maybe I want to get some more in person and see it in action. Is there any events going on or online resources they can use to learn more about Keploy.

[00:19:57] Shubham Jain The first place to start would be Slack channels. I mean, we have a community there and we are active. The second place would be like if you wanted to have a more direct conversation, the best channel would be our community calls. So we have that monthly. Yeah. Anybody can come and share their experience. If they have a use case they want to get answered. That's under the channel. So I think those two are the best channels from the community.

[00:20:20] Joe Colantonio And we also have a lot of folks in India. I believe you're based in India, and I know there are a lot of meet-ups that go on that you'd like to attend. Is there any live events that if someone is in that area? I know India's huge, but any live events they could actually attend if they're hearing this. Like I said, there are a lot of guild members from India.

[00:20:36] Shubham Jain We have an event coming up in collaboration with Google to focus on open-source contributions and getting started with open-source in general. It's in collaboration with the Google startup team as well as the Google Summer of Code team. It's happening on 29th Feb in Bangalore. So yeah, I think that would be a very nice event to check out, especially if somebody is interested in testing as well as open source.

[00:20:57] Joe Colantonio Okay, Shubham, before we go, is there one piece of actionable advice you can give to someone to help them with their mocking kind of Keploy automation testing efforts, and what's the best way to find contact you and learn more about Keploy?

[00:21:10] Shubham Jain Best way to connect, I mean slack. We are available and I'm also available 1 to 1 and advice. I think that would be to keep mocks realistic to record real behavior. I have faced a lot of problems when I would assert my mocks, or when I would make mocks myself, which are most of the times not true. I mean, they're not real behavior. So I think that us bit me a lot. I think, yeah, let's keep it as real as possible.

[00:21:34] Joe Colantonio Awesome. Head on over to Keploy.io and definitely check it out. It's also on GitHub actually, and it has 3,000 stars. So it seems like it's well like so definitely check it out. Thank you so much for joining us today. I think it's a great solution that people definitely need to try and get their hands on right away. So thank you so much.

[00:21:50] Shubham Jain Thank you, Joe.

[00:21:51] Joe Colantonio Awesome.

[00:21:52] Joe Colantonio Thanks again for your automation awesomeness. The links of everything we value we covered in this episode. Head in over to testguild.com/a486. And if the show has helped you in any way, why not rate it and review it in iTunes? Reviews really help in the rankings of the show and I read each and every one of them. So that's it for this episode of the Test Guild Automation Podcast. I'm Joe, my mission is to help you succeed with creating end-to-end, full-stack automation awesomeness. As always, test everything and keep the good. Cheers.

[00:22:28] Joe Colantonio Hey, thanks again for listening. If you're not already part of our awesome community of 27,000 of the smartest testers, DevOps, and automation professionals in the world, we'd love to have you join the FAM at Testguild.com and if you're in the DevOps automation software testing space or you're a test tool provider and want to offer real-world value that can improve the skills or solve a problem for the Guild community. I love to hear from you head on over to testguild.info And let's make it happen.