246: Chaos Engineering with Tammy Bütow

Tammy is an Australian and enjoys riding bikes, skateboarding, snowboarding, and surfing. She also loves mosh pits, crowd surfing, metal, and hardcore punk.

• Back in 2010, the Netflix team created something called Chaos Monkey. Chaos Monkey became really popular because it was right when Netflix was moving to the cloud. And they did their massive migration. What they wanted to do is be able to make sure that whenever engineers were building something on the AWS they didn't just think that all of those machines will be there all the time.
• It's really exciting sine Gremlin is the first company to actually build something like that that has a web UI that has an API that allows you to programmatically do all of your Chaos engineering experiments including disaster recovery testing which we call in region fall over in the chaos engineering world.
• We did actually recently build a new product call Alfi which is application level fault injection. And that's really a new concept. It's much more advanced. So usually what people do is they start with infrastructure levels so we collect that Ilfi (infrastructure level fault injection) where you are injecting failure on the actual servers or the containers or network related failure. But there is the idea to have actually injecting failure into your application.
• The idea here is you can do very precise fault injection so you can actually match against any attribute you're already using you can really do precise scoping of attacks to things like for example for custom IDs, location, device types after you integrate Alfi into the application. And that means that you can create really small glossed radius chaos engineering experiments.
• One of the big things that we've been focusing on over the last year is really helping people learn about how to get started doing chaos engineering because technically it can be quite complicated and a little bit scary before you start doing. But once you've been doing it for a long time you'll become very good at it and you will no longer be scared to run chaos engineering attacks.
• This is one of my big cases advice and it is going to sound a little scary but I like to say that it's important to focus on critical systems not low hanging fruit. And that's my personal advice and the reason is because if you focus on a critical system you inject failure for a critical system then you're able to show people why it's important and you can actually show them the positive impact that you can have because your more likely to be able to identify important things that you need to fix.