About this Episode:
Are you struggling to find information on how to use Sysmon for your security efforts? In this episode, Carlos Perez, a Research Team lead at TrustedSec, will discuss the TrustedSec Sysmon Community Guide. Discover why Carlos created this guide, and how it helps empower defenders with the information they need to leverage this great tool. Also listen in to hear about Carlos’s extensive knowledge gained in working to detect attackers.
TestGuild Security Testing Exclusive Sponsor
Micro Focus Fortify is the recognized market leader in application security and is the most comprehensive and scalable application security solution that works with your current development tools and processes. Try it today
About Carlos Perez
Carlos Perez is the Research Team lead at TrustedSec. He has been active in information security for 20 years. He also contributes to open-source projects like metasploit and backtrack.
Connect with Carlos Perez
- Company: www.trustedsec.com
- Website: www.tenable.com
- Blog: www.darkoperator.com
- Twitter: Carlos_Perez
- LinkedIn: carlos-perez
Full Transcript Carlos Perez
Joe [00:01:35] Hi Carlos! Welcome to the Guild.
Carlos [00:01:38] Hey! Happy to be here.
Joe [00:01:40] Awesome. Great to have you on the show. Before we get into it, is there something I missed in your bio that you want the Guild to know more about?
Carlos [00:01:45] Well, I live in Puerto Rico, which is kind of cool. Not many people here from the island are very active in the Security field, at least from the island itself. I've been doing Security now for 20 plus years. I actually started in 98 working for the government and I've worked for companies like Digital Compaq, Hewlett-Packard, Microsoft, Tenable Network Security and now here at TrustedSec.
Joe [00:02:09] Awesome. So you've been in the game for a long time. Just curious to know as we enter the New Year, as we recall this is the beginning of 2021, are there any trends you see that are different from when you started or that you thought would be different, are things still the same, people are creating secure enough software?
Carlos [00:02:26] I would say 2020 was special because for the first time due to Covid we had to deal with a mobile workforce, a brand new mobile workforce. Many companies out there actually simply had a simple VPN so they can have like three or four users when they needed to connect. And all of the sudden now they were thrown into the middle with all of their workforce now having to work remotely. And they went like, “How do we do this?” And cloud kind of became this humongous thing where companies start moving to Azure AD and instead of receiving a laptop and prepping it and sending it to a user, all of a sudden they were having Best Buy or having D&H delivered the laptop straight to the user. And now we have to connect it to the cloud and manage it from the cloud. So for anybody in the industry that was not used to that environment, which is a great deal of people managing clients in the cloud, it was a very daunting task to learn, adapt. So you started looking at organizations and knowing which ones actually had people with a kind of like a growth mindset instead of a static mindset to be able to adapt to this time.
Joe [00:03:35] Absolutely. Now with that switchover were there any new security issues then that came up with people working remotely from the cloud that makes software more exploitable or networks more exploitable because people are working remotely, because people didn't think about this type of work kind of conditions?
Carlos [00:03:53] Yeah, especially when it comes to phishing attacks, social engineering types of attack. Many companies actually depend on having the control of controlling the traffic outside of the network. But now the users are at home. So how do you control that traffic? “Oh, I have to force him in the VPN.” But what if they don't bring up the VPN? How do we look at that traffic? How do we patch these machines? So right now it's been kind of like a bit of a struggle for people to figure stuff out the beginning of the year. Now, to meet (??) year people were getting kind of like on track and now they're addressing a lot of that stuff. Some are still struggling, especially because as (unintelligible) surface and software didn't patch them, now people don't have the filtering, don't have the protection of a corporate network and are more exposed. So companies have had to adopt what we call no trust model, which is something that Google does where it forces every asset needs to be fully authenticated. Stuff is in the cloud and the client has certain software that connects to other cloud infrastructure to report stuff, because now those machines do not connect to the AV (??) server, the AV server is on prem and if they don't VPN you can not communicate to it. You cannot get new signatures.
Joe [00:05:08] Very cool. So I guess I don't know if this makes any sense because I'm a newbie here. How about people using maybe not company sanctioned (??) hardware to access software in the network? Can that caused security issues? I know I've been hearing a lot about maybe chips being in other countries, being kind of exploited by shadow governments to kind of create havoc as well. Is that overblown? Is that a thing?
Carlos [00:05:31] Well, it depends on your threat model. If you're somebody that a government is going to go after, for example, you work with DoD or you work for a company that has ties to DoD that should fit in your threat model. All of those supply chain attacks, like who makes my software? Who makes my hardware? Which ships it? At which point does it come through? Because we've seen this kind of attack, it's not new. We've seen it. For example, I remember when CCleaner got infected a bunch of people got owned by it. Another example, when Snowden put out that the NSA intercepted Cisco and Dell shipments to put chips into those and back doors into those. Same kind of behavior that many other countries actually do, for example, today is making the rounds, a dissident in Russia, all of a sudden he had to take his iPhone in for repair and they went like, “What is this?” And when they showed it to him, he had a GSM bug, physical bug, a small board with its own GSM antenna embedded into his iPhone that somebody installed at one point or another.
Joe [00:06:37] That's pretty crazy.
Carlos [00:06:38] Yeah, we think of it as, “Oh, no, that only happens in movies.”.
Joe [00:06:41] Right.
Carlos [00:06:42] Well, it does happen in real life.
Joe [00:06:45] Yeah, what's scary I guess is it was found by mistake so how many people are intentionally not being found? I guess that brings me to how do you know? Another thing I've seen a lot is people usually don't know they're being hacked or they're being compromised until after the effect. Is that part of your threat model as well or being aware of your risk, how to harden systems?
Carlos [00:07:04] And the way you do it is you monitor your system. You build what is known like a standard behavior of the system. That is one of the things that, for example, you mentioned the system on guide . That is one of the things that we cover for certain event types that you monitor on the computer. Our recommendation is, “You know what, install the software, enable it to capture these types of events. Look at it after a month, probably after a patch cycle so you can capture word stuff that software does as it updates and build a known good list of behavior of the machine and then capture but doesn't fit in.” That would be kind of like a good way of doing it is like when you go outside and you look out of the street for your walk, you've never seen a van with tinted windows and all of a sudden now you're seeing the van with tinted windows every other day. That would make you suspicious. Same thing with your machine. If you see something that is not normal, not part of the normal behavior you go like, “That's suspicious.” And that's something that then you investigate.
Joe [00:08:02] I like that analogy. So we're talking about Sysmon. Before we dive into it, that's one of the main reasons why I want to set this interview as well as I heard a lot about this community guide. I've heard that Sysmon is a great tool but it's kind of difficult to learn because there was no documentation out there. So I guess, first off for the folks that don't know, what is Sysmon?
Carlos [00:08:19] Sysmon is a software that is written by Microsoft, part of this internal tool suite that is one of the side projects of Mark Russinovich. He is the CEO of Azure. And the tool itself is a freeware without support but what it does is that it enhances the event log. It allows you to create rules and tell it, “Hey, any program with a command line that matches this, log it. Any program that access this registry key, log that. If I have a program accessing the memory of another program, log that.” And it allows you to create a very fine tuned series of behaviors of tactics, techniques and procedures and embed them into a config file that you feed to the tool. So you're able to capture that. And then as another tool ingests all of those logs they're able to take actions and go like, “Hey Sysmon detect that something that it says is suspicious. Let's take a look or let's alert.”
Joe [00:09:17] So what kind of tool will injest this then? I heard a lot of things like AlexStack or does it matter? You're just feeding a database and then feeds a dashboard that alerts people proactively.
Carlos [00:09:26] Yeah. You can actually see those locks into AlexSack, Graylog, any system (??), and any platform that can injest logs and you can set rules and alerts on Sysmon will work because it actually adds or in fact, it actually extends the already existing Windows event log. So anything that can consume the Windows event log can leverage Sysmon.
Joe [00:09:48] Nice. So I assume everyone that has a Windows environment should be using this. Would this be appropriate for anyone or should every team that's worried about security, that has a Windows based environment should they be using Sysmon?
Carlos [00:09:59] Yes, I believe they should. But there's a caveat. If you're going to use it, you need to spend the time to tune it, to learn it. It is not one of those tools that you simply double click it, next, next, next, and then a (unintelligible) good at the end of the machine and now everything is super fine and it will take everything. It needs to be tuned constantly because the tags change, behavior change, and people need to know about it to be able to know how to configure it, to know what is amissed when it reports stuff. So if you have a security team or SOC that actually can spend the time to do threat hunting, it is very useful. If you don't have the resources to do it, it just adds noise to the environment. For example, I had a hospital that decided to downscale and part of them downscaling, sadly, was the Security team. Security team went down from six people down to two. Now that person for some reason also had to be in charge of the AB (??) of updating it and also part of the change management process and it had so many tasks. Adding Sysmon to him next without a system that he can automate or have the knowledge or the training to build upon it would not be useful for them. It is one of those things that you do need to spend time to learn it and to play with it.
Joe [00:11:15] All right. So the follow up question then would be what would you recommend then if someone's like, “Oh, this sounds great, but is there a roll up plan usually recommend? Is this someone with a certain skill set you recommend? If you don't have them, then you probably don't want to start with this right away.
Carlos [00:11:29] Typically, you need to know what you want a log. What is the behavior you want to log? And then you build out. You should also feel comfortable with XML because the configuration is an XML file and typically what you do is you grab a version of Sysmon. You identify what is your install base and the most common systems. You grab a set of test systems and you deploy it to those and you deploy the configuration. You capture data and across a schedule every three weeks, every month, you go back into the logs, you look what you're capturing and you start tuning that configuration, either first section of your environment or user, and then you do the final deployment to everybody else once you feel comfortable with it.
Joe [00:12:13] Nice. So you did write a guide, the community guide as I mentioned. I think you wrote it because there was hardly any documentation on this. If someone read the community guide would that help them implement it or is the community guide just for someone that already has the skills to kind of fine tune it the way you think needs to be done?
[00:12:27] It will actually help them to implement it. I even included scripts and commands of multiple tools, what to be careful with. You can shoot yourself very easily if you're not careful. For example, in setting you set it to capture all drivers are getting logged and all communications out to the Internet. And all of a sudden when you open your browser, it floods with 60, 80 logs all at once, just by double clicking and opening Chrome or opening Firefox or Edge. Because when you open a page, you don't know that it's making so many outbound connections all at once. And you can very quickly over text the CPU if you're not careful. And that's why I wrote the guide so I can tell you, “Hey, be careful here. This other place configure it this way. On this one you want to just grab this configuration from this page and you should be okay. And this other, no, let's take our time.” And I give them kind of like an idea of the process so they can follow it so they can adapt it to their own environment.
Joe [00:13:23] Very cool. So since you've written this, I believe you also, if I read it correctly, did you create some extra modules to Sysmon as well? And if so, can you tell us a little bit more about what those modules are and how to help folks that are currently using Sysmon?
Carlos [00:13:35] Yes. In fact I actually wrote the BS (??) code extension which is also the code extension that allows you to have snippets to help you write those rules a lot faster. And I also wrote as a personal project with a couple of friends of mine from Microsoft and FireEye and some other companies I wrote a Powershell module called PSGumshoe, which is for when you're doing insert response. But after doing several trainings for MARFORCYBER, Marine Force Cyber Units that do specifically CPTs, which are known as cyber protecting teams, which are the guys that do all of the insert response, I gather all of their feedback on their use of Sysmon. And I created a bunch of functions or commands for the command line in PowerShell that you can use to create (??) a filter and help you even build new configurations for Sysmon.
Joe [00:14:24] So I know there's a lot of different areas in Security. Where does this tool fall? Is it part of threat hunting? Is it part of Red Team, Blue Team? Where would this be used?
Carlos [00:14:32] It would be Blue Team.Blue Team, threat hunting. Yeah, I've seen environments where, for example, they'll go in and they'll identify a series of TTPs, tactics, techniques and procedures from an actor in the network and they'll quickly deploy a new configuration of Sysmon with rules for those TTPs to find out on which machines they are currently in. So you can use it that way for threat hunting actively, also passively, because you can go into the logs and look for those TTPs in the logs and say they must be here, here or there. So it's more of a Blue Team kind of tool. On the Red Team side, typically what they do is they work on ways on how to bypass those rules. Now, something I tell in all of my training classes and to customers is that the vendors control two important things for attackers. They control their toolset and their tempo. What tools can they use and how fast they can move in their environment? If all of a sudden you have a hardened very well configured Sysmon ruleset and they see that ruleset and they know Sysmon is there, all of the sudden you've taken off the table a bunch of their tools and tactics and techniques because they're going like, “Uh, it got to move slower now because they can see my actions.” So all of a sudden you impacted their toolkit and how fast they can move.
Joe [00:15:47] That's interesting. So I wonder, there must be like I said I'm new to Security, is there like a visual that has like this set of tools carry (??) these type of exploits and you could see where maybe you have some weakness based on that type of visual?
Carlos [00:16:00] Yeah. In fact, there's a project called the MITRE Attack Framework that covers most of the techniques that you're seeing out there. And you can actually grab a bunch of those techniques and you can start building your own configurations on top of it. There's also a project by a good friend of mine. His name is Olaf. He's going to kill me because I'm forgetting his last name. It's Olaf Hartong. He is in Austria. He has a Sysmon configuration project where he keeps a bunch of small snippets of configuration from Sysmon that you can copy paste. And he has reference to the MITRE attack framework on many of them.
Joe [00:16:38] Oh, that's cool. So I guess so that's it sounds like you could share snippets almost like custom functions or between each other. So are there any functionalities then in Sysmon that you think folks don't know enough about or maybe don't use enough that you think would really benefit them?
Carlos [00:16:53] For example, there's a new one, which is for process tempering that will actually tell you if somebody is using process along (??). So one of the Red Team tools that sadly many APT groups now have adopted is Cobalt Strike. Cobalt Strike when you try to execute .NET binaries or you're trying to run certain commands, certain actions in it, it will start a new project and due process hollowing on it. This new event type that was added on Monday will actually detect that behavior. So that is very useful. It is out there. I encourage people to start testing it and start deploying that one. One of the things that I like is that it makes the life of the threat hunter a lot simpler and easier because typically before Sysmon, when you are doing hunting, you were going by process ID. The thing is that many people don't know that every time you start process an ID gets added to it and those IDs get reused. So when I'm doing (??) the logs and say, “Oh, I want everything that was created by this process the ID is 1886 and I run a query for everything 1886 I'm also catching a bunch of other processes that are reusing that ID. Sysmon introduced process (uintelligible) and machine (uintelligible) that actually are unique to that process across your entire network, which actually makes threat hunting a lot easier, especially when you're trying to do that initial scan across your network to find where they can be in actions.
Joe [00:18:16] Cool. So earlier you gave the great example of when you're trying to look at your traffic or anything out of the ordinary and you did an example like a yellow van. If you saw a yellow van in the neighborhood with tinted windows you never saw before. How do you create that, though, in Sysmon? It seems like it's collecting all this data you put into dashboards, but how do you filter to find those type of anomalies that are real things that you need to investigate more?
Carlos [00:18:37] Okay, so with Sysmon one of the things you can do is you can create a rule that catches everything. Once you create that rule that catches everything and you run it for a period of time, then you can create your logs and build a known good and that known good you add into another rule group that you call exclude. Like, “Hey, these are my typical processes with their command line. Exclude this from being logged.” Now, everything that will be logged now is what doesn't fit that known normal, for example. And that's where it's going to pop up. And you can look at it and see is it a change in the tool and the way it behaves? Or is it something nefarious or something that actually triggers alerts in your head? Like, “It shouldn't be doing that. That command line looks kind of funny.”
Joe [00:19:23] You also mentioned in the beginning that it's not just plug and play, you let it go and it does its thing that people need to be able to adjust over time based on different attacks and I guess things that are happening in the field. How do you know when to change what you're doing already? I don't know if that makes sense. But you have like, in Security is there like a monthly sprint or some sort of scrum meeting. Everyone gets together and says, “Okay, here's what we have and what do we need to improve it? Or is there any change that we need now to introduce or change within our current type of strategy?
Carlos [00:19:55] Typically, initially it's kind of like meeting every other day to a week. And once it is stable, it is more something that you can actually do bi weekly or monthly. And typically you are going to involve two different teams. One of them is an operations team that can tell you, “Hey, Sysmon is impacting performance over here. Or we're collecting too much garbage or we need to deploy a new version. When should we deploy it? How should we deploy it?” And that's mostly on the operational side or your DevOps team. And then you have your SecOps or Security operations team that will go like, “Hey, we've been gathering threat intelligence actions that actors have been doing either via Twitter, via Recorded Future, threat intel feed, or any other free open source threat intel or even Twitter is one of the best so far for me. I have kind of like a list of people that are constantly hunting and putting stuff out. And I'll just simply go and look at them and go, “Huh, I never thought of that technique or I didn't know the Russians were doing this. I didn't know that this crew from the US is doing this or this group from Ukraine is doing this other thing.” I could track that. Let me write a quick rule and they'll gather all this threat intel and then decide what applies to us, what doesn't apply to us, and let's update the rules.
Joe [00:21:09] Okay. Nice. So I always think of Twitter is just wasting time, but I guess there are people actually posting things that are really helpful. Like that's a good, good kind of hack I guess. For sure.
Carlos [00:21:18] You need to curate it.
Joe [00:21:19] Yeah. So I guess this is probably off of the deep end here, talking about curating and knowing what's real, especially nowadays, you know, how do you know if something's really been hacked, especially when it comes to elections and people's confidence in elections? One side always says like, “The election's been compromised by Russian hackers.” Then four years later, I don't think there's been any new technology or effort to try to avert this. And people say, “Oh, no, this has been compromised.” Like, how do you know, I guess, what's real when it comes to security? I don't know if that makes sense.
Carlos [00:21:52] Yeah, at the end, it's show me the proof.
Joe [00:21:55] Yeah.
Carlos [00:21:55] Like I need that proof. Like, for example, any system can be hacked from your iPhone, your Android phone to our laptop, to our computer. Anything can be hacked. Now, the thing is, you need that proof that says, yes, I was hacked, especially when it comes to something like elections, because one thing that people forget is that when you go in and challenge it and you say, I won all of the voting records so I can check them by hand, is that votes are supposed to be confidential. People are not supposed to know who voted for who, only to tally the amount of votes. So too much info opens the door for in any government change that it becomes authoritarian to go like, “These are the guys that voted against me or these are the guys that have this view, I'm going to go after them.” So that's why the voting system has been set up where the only confidentiality (unintelligible) that it's important is that X amount of people voted and they voted and this is the tallies of it. But there's no one to one like Carlos voted for this candidate. Joe voted for this other one. And that's why they're built the way that they're built. Now, when people say it's being compromised. And so far, everything I've seen is like we think it is this way because it doesn't behave the way I think it should have been. I go like, “Well, that's an argument that if I play with my wife, I'm going to get in trouble.” If I go like, “Hey, honey, it was supposed to be great and it ain't or this was supposed to taste bad because it looks bad or stuff like that.” It is not what we think could be. It is what can actually be done or has it been done. For example, Krebs, the guy that ran CISA, he actually did a bunch of good stuff that us in the information security arena were very happy with. He worked with DoD. So as soon as the Koreans, the Chinese or Russians were taking action, he was very quickly putting a lot of those malware samples out there and helping coordinate that stuff, because now you burned that tool all of a sudden you may be able to identify every other place that they have been going into. So that was a unique to him. He started working there. As soon as he started the campaign, he had a group of threat intel analysts that were going in and analyzing all of the data and everything that was set. And he created a page and started saying, like, “This is fake, this is real, this is fake, this is real,” and start squashing that stuff. So it wouldn't grow because a lot of this psyops campaign psychological operation campaigns actually depend on putting out there uncertainty and doubt. So as soon as you address that stuff and this is how we know that it is that way, you kill that. Now that sadly, some politicians on both sides go like, “Oh, no, but my feelings say that it should be that way.” Well, yeah, I was actually asked talking about actual factual proof. They monitored those systems. They checked the versions of the systems. They as soon as something came up and sounded suspicious, they had people go in and check stuff. So he knew that everything was secure and everything was not tampered with because they did the due diligence to look into that stuff.
Joe [00:25:07] Absolutely. And I don't know how my mind works but as you were telling the story it reminds me of another story that's completely different that you were talking about in one of your YouTube videos about you doing some sort of work for someone. And they claimed you broke their system down because you were doing work on it and talking about proof, this is what came. Why I thought of this is you said well, you track everything when you're doing type of engagement so that you have hash tables and you have timestamps rather than to say just to show proof that, look, you're saying this and it's false because here's proof that I did this, this and this in this certain time spent. Could just talk a little bit more about the points of that, I guess?
Carlos [00:25:43] Yeah. If if we look at it from the Pen Testing side or Red Team side, every time I do an engagement, my different platforms, what they actually do is that they track every command that I issue and it has a timestamp and what is the output of this command. I remember on one of my first Pen Test back in 98, 99, I remember I had this lady at a government agency say that I burned down her CRT monitor. It was that long ago and I was like, “No, I did not. For starters, there's no way of me issuing a command that will burn your CRT and bring it down. It's just not possible.” And I had a log of all of my actions. I had configured in my CMD.exe a prompt that included the time of each one of my commands since I was issuing them back there in Windows 98 and it was keeping all of that data and I go like, “This is the command I issue. Look here at this hour. At this hour, when you're saying that you had this problem, I was not in your system. I was in this other system. So there's no way for me to have done that. In fact, I had no interaction with your system.” Same thing when you're doing forensics because you have to take it into court. You need to know specifically each and every command that you took inside of the machine, because in forensics, there's this principle called the Locard exchange principle. Edmond Locard is the father of modern forensics. He was a French cop. And the principle actually boils down to any interaction between two objects or two subjects is going to leave a trace. So with the absence of trace, there's an absence of proof that there was an action.
Joe [00:27:20] Perfect. Okay Carlos before we go, is there one piece of actionable advice you can give to someone to help them with Security testing efforts? And what's the best way to find or contact you?
Carlos [00:27:29] Yeah, in Twitter I'm carlos_torres. Or you can contact also the company TrustedSec at www.trustedsec.com. I'm the lead for research there, so I get involved in engagements kind of like a support to engagements when the Red Team needs a new implant or a new tool for an action, or the instant response team finds new TTP that they need us to reverse engineer and help them work out stuff there.
Joe [00:27:57] Great! And Carlos do you have one piece of actionable advice you can give to someone that they can take away and implement right now to help them with their Security efforts or Security testing efforts?
Carlos [00:28:05] If it is somebody in an enterprise, I would say just like it is corny, but know yourself. Know what are your systems. A lot of people believe it or not do not know how many machines run in their environment, how many users they have. And even if you don't have that basic piece of information but clear inventory of what you have, you can now go to the next step, which is to know how do they interact with each other? Because then you can group them and be able to take action. So knowing yourself by having a clear inventory of your environment, it's critical. And that's one place that I see so many fail.
Rate and Review TestGuild Security Podcast
Thanks again for listening to the show. If it has helped you in any way, shape or form, please share it using the social media buttons you see on the page. Additionally, reviews for the podcast on iTunes are extremely helpful and greatly appreciated! They do matter in the rankings of the show and I read each and every one of them.
