[Music] hi my name is marcusynovich welcome to sysinternal's tools overview i started cis internals back in 1996 with my friend from graduate school bryce cogswell back then it was called nt internals a few years later we ran into sysinternals but the goals of the site have always remained the same from the start which is to make available tools and information to help you understand the way windows works or troubleshoot windows when it's not working like it should the principles around the tools have been that they're single file that they're intuitive to use and that they don't leave behind anything on your system after you've finished using them while the tools from the start originally supported windows 95 windows 98 windows nt and today support 32 bits versions of windows 64-bit versions of windows arm 64-bit versions of windows single file means that you can download one tool and run it across all these various systems it will dynamically detect what it's on and for example if you're on 64-bit windows the 32-bit main executable will extract 64-bit versions launch them and then when they're terminated clean them up when you go to the assist internals website you can see the downloads links over here on the left side clicking on that will take you to a full list of the tools and at the top you can see that there's various sweep bundles this is the main suite bundle which includes the 32-bit and 64-bit versions of the tools you can see that there's a nando server bundle for nano windows server version nano server and then there's another one that just has the arm 64 versions of the tools clicking on the suite will open up the suites contents and you can see that there's close to 80 versions of the tools now one of the tools that's my favorite is the one that i've been using to highlight and zoom and it's called zoom it zoom it lets you zoom in static view statically as you can see you can see that i'm able to draw freeform draw shapes like arrows like squares and even like circles you can change colors by typing various keys to represent different color like g for green y for yellow b for blue you can even make the screen completely black and then draw free form there you can type t and then type and it also has live zoom capabilities so you can actually zoom in and move around and interact with the screen while you're zooming now in this talk besides just giving an overview of the site like i just did i'm also going to highlight some of the most popular tools and tell you just a little bit about them and why you might be interested in them following this session there's additional deep dive talks by experts on each one of these tools and let's start by looking at process explorer a lot of people have called process explorer task manager on steroids so if you've used task manager to identify what's consuming a lot of cpu or disk or memory process explorer does that but much more and gives you much more detailed information about those processes on your system one of the most common uses of process explorer is to find out what has a particular file open and for example if you're trying to delete it you might be wondering how do i delete that file and that you can do that through a file search capability here if i go to find file handle or dll and type in zoom it for example you'll see that zoomit. exe is in use that file by the zoom it process and you can see that the system process has a bunch of handles open to it as well as the 132 process has opens to the thread threads open and system and zoom it now when you open process explorer you see this convenient process tree which shows you parent-child relationship between processes if the parents and children both are still in existence when you hover the tool tip over process you can see its command line you can see its path you can see services for service hosting processes like this service host and then there's columns that will show you cpu usage memory usage both virtual and physical description of the image the company name but you can add dozens of additional columns that show you very conveniently additional information about those processes all of this information like dpi awareness what process protections it has whether control flow guards active are available also in the process properties when you go click on the process properties and look at the image tab for example you'll see there again the command line current version things like what user context it's running in here the control flow guard and other process protection settings and then you can see additional tabs that show you information about performance you can see graphical views of the performance private bytes i o cpu usage disk and network counts cpu gpu graphs if it's using the gpu what services are active in it it's threads information about which threads where they're starting their thread ids how much cpu they're consuming any tcp endpoints they've got open the security context of the that the process is running in which is represented in something called the token its environment variables and then what strings it has in the image in memory or on disk and this can help you identify the purpose of a process that you might be unfamiliar with the other things process explorer can do is show you information about the handles and dlls that are loaded into a process if you go and show the lower pane you can see here is the handle view which shows you the operating system resources that the process currently has open files registry keys and so on if you click on modules you'll see mapped files as well as dlls dlls being those executables that are it's running code from and map files being files that like these natural language string files that it might be loading to represent different resources and then finally in the threads tab you see additional information about the threads much more than you saw in the threads tab in the process properties as additionally you can see these convenient system views at the top which will show you the most active by cpu consumption it'll show you the system commit usage the physical memory usage over time in a summary view here of all of those stats across the system and this is just dipping our toes into the things that process explorer can show you one of the most powerful tools in this week and really the when when you ask me which is the one that i've used the most besides zuma it is process monitor process monitor is a tool that captures file registry networking and profile information it shows you full thread stacks it shows you flexible it has flexible non-destructive filtering it can capture literally millions of events shows you to process trade the running processes that have existed at all times during the execution of process monitor including ones that have terminated and you can also capture all of the activity through an entire windows lifecycle from very early in the boot process all the way through shutdown it is so powerful it's showing you what's going on in the system that it is the number one tool when it comes to troubleshooting in fact it solves many cases where you think process monitor is not going to show us the answer but it turns out that it does me and dave solomon when we were working on uh troubleshooting in our classes that we'd teach the predecessor to process monitor was called fileman and regimen and we similarly founded those tools solving so many problems that we'd on it unexpectedly that we came up with the phrase when in doubt run fileman regiment of course when process monitor me and bryce delivered that shortly after we joined microsoft the catchphrase turned into when in doubt run procmond so think something's happening on my system what's going on let's run procmod if i go take a look at process monitor here i've got an active trace that i'd captured just a little while ago as i was preparing for this session and you can see here in the display the time of day the process name the process id the operation here's a create file the path of that particular operation and if it's a file operation this would be a file system path the result and then additional attributes here you can see the desired access the disposition whether it was an open or creating new file you can see the attributes the share mode and so on lots of information about exactly what's happening like i mentioned if you go to the event properties and go to stack you can actually see the full stack that led to that operation which can give you in from information about what exactly is executing this operation and why it is and you can see it's resolving symbols here to show us the stack and the symbol names like path is directory is a call that's being executed here to see if this path is actually a directory or not the reason that i've got this open and this trace captured here is i was downloading the system journal suite and here you can see the full suite of tools like i showed you earlier and i was copying it over my tools directory which you can see right here which is where i got my local copy of the tools i ran into this error here as i was doing it the action can't be completed because the folder file is open in another program close the folder or file and try again well first this message is a little bit vague what is keeping this file or folder open and why and if i do this again you can see there it was part way through the copy operation but this error just continues and i had no idea at the time i ran into this what was causing it so i ran process monitor typed in an additional filter just for the tools directory you can see here path begins begins with c tools include which includes only references to the tools directory and now i can immediately see what was going on these can't delete errors correspond to that error message in that dialog box and what explorer was trying to do is to delete zoomit. exe as we saw earlier in the process explorer find dialog zoomit.
exe in the tools directory is actually opened by zoom itself because i launched it from there and so that's why i couldn't delete it so here i'm i can immediately understand what that source of that error kind of vague error message was and if i wanted to get past this now i could just simply exit zooma. exe and that error would go in and i'd be able to finish copying so again process monitor to the rescue here on solving a kind of unexplained situation the next tool we'll take a look at is called autoruns it is one of the most popular tools in the site because it gives you information about what is configured to start and run automatically on your system in far richer detail and comprehensiveness than any other tool it knows of literally hundreds of different locations you can see here as i've got it running that it categorizes things into various groups the everything tab shows you everything that it's aware of that is configured on this system to automatically launch whether things that are built into windows or added by third-party software on this machine but then you can drill down into specific categories like what's configured to launch when you log on to the system what's configured when explorer launches it like explorer context menu handlers that you can see here schedule tasks services that are scheduled to run drivers codecs boot execute print drivers what's configured to launch as part of winlogon the logon manager interactive logon manager and much more then you can see when we click on something down here at the bottom you can see additional information about it like the publisher the size the time it was built and the path to it one of the key characteristics of auto runs is helping you identify what's been added to your system after base windows components run and for that you can go to the options and do things like hiding microsoft entries or hiding windows entries here that will just let you focus on what third party software is configured it will also in the case of something that is foreign to you like what is this doing on my machine let you go search for it online or submit it to virustotal a free service run by google that has files that are submitted run through dozens of antivirus tools i've done that earlier with zoomit. exe and if i open this report up you can see that no security vendors flagged this file as malicious there were 67 virus scanners run on the zoomit.
exe executable whether it was at the time that i submitted that or previously by somebody else and no hits were found and so this is a good way to help you identify is something malicious or not on my machine next tool i'll talk about is called proctum this one's a little more esoteric really for people that are trying to understand what is crashing on their system and perhaps in a position through use of debuggers to understand why or to be able to submit a crash dump to a publisher of that software to help them understand what happened so process explorer and task manager both have ways that you can create a dump of a specific process but i created process proc dump back when i ran into situations where i would have something that would spike the cpu on my machine and i wanted to understand what was causing the cpu spike it also uh one of the challenges that microsoft support ran into is outlook having hangs so capturing a dump when a process hanged and so this called for a tool that you could run capture a dump of a particular process on demand but also be able to set triggers that would give you the ability to say only capture a dump if the cpu goes above this limit capture a dump if there hasn't been an exception capture a dump on a termination and proc dump has those kinds of triggers built into it plus much much more further you can install proc dump as a postmortem debugger meaning you can register it on the system so it captures a dump of any process that has an exception on the machine and that lets you get exact kind of dump information that you want because proctem supports various sizes dumps from a full dump which is the entire memory and context of the process down to minidumps which are just small summaries basically of just the core context information and the memory pages that are referenced from that core context you can also leverage dump extensions to create custom dumps that for example microsoft support does to capture the most relevant parts of for example one a piece of software that has large amounts of memory in use and some of that has interesting information for dump analysis and some of it doesn't let me just give you a quick example of proc dump in action if i go and go to my tools directory and say proc dump minus c20 to say if this process is cumulative assuming more than 20 of the cpu then create a dump of this process cpu stress which i've got running right here and put it in the temp directory so now we see if that process executes more than 20 of the system cpu for a threshold of more than 10 seconds then it will create a dump and here's the format of the dump name which is all of this is completely configurable i've got it paused right here but when i activate these threads cpu stress being another tool from sysinternals just to show you to generate cpu activity on your machine you can see now proc dump is seeing that it's above the threshold and here we go 10 seconds of above 20 and now we've got our dump listed so a great tool to help you troubleshoot your own processes and diagnose them if you've been if you're a software developer or to be able to capture dumps that you can submit to the software publisher now the next tool that i'm going to show you isn't actually a tool it's a collection of tools it's the ps tools suite of tools and the reason it's called ps tools is because the first tool that i wrote as part of this back in the late 1990s was pslist a tool that would show you much more than the tools that were built into windows about which processes were running from the command prompt of course i had a process explorer that would show you information about the running processes in graphical mode but if you needed to very quickly from a camera inline c what is what processes are running which ones are consuming cpu ps list was the tool for that further ps list would let you look at the processes running on remote systems as well so from a single system you could see across your network all of this the process activity across them from a simple command line tool and the reason it was called pslist is because it would list processes and like the ps tool in unix which i'd used in grad school so as i started to add tools to this with the same capabilities single file executable able to run both locally as well as across remote systems and having the same kind of way of being able to authenticate with username and password to remote systems thus began this collection of tools with all the ps prefix to identify them as having those characteristics and you can see that there's now over a dozen of these tools in this collection the most the newest one being is being ps ping but the most popular tool by far is ps exec ps exec lets you remotely execute programs remotely interactively as yourself or an alternate credentials and lets you do things like troubleshoot what's going on and basically remotify any local command like ipconfig by itself you can't run against a local machine a remote machine but using ps exact you can and command prompt doesn't let you by natively run against the local machine but with ps exact you can so if i type mr surface 4 which is my laptop machine next to next to me and i say i want to run command. exe psx is going to start a remote service on that machine it's going to connect create an interactive session by connecting the standard and standard op name pipes that command prompts have and that will let me interact with that machine remotely so here in a few seconds when the connection is finished i'll be able to type hostname and see that sure enough i'm interacting with a command prompt that is actually not this machine which when i exit this ps exec and type host name we'll see that the hostname is actually mr game and not mr surface the final tool that i'll drill into is called sysmon what cisbond does is monitor and report key system activity that are related as security relevant events into the windows event log it's a tool that i created back in 2014 when john lambert who's in our microsoft threat intelligence group came to me said that we had malicious activity on the microsoft corporate network the attacker was dropping files and then shredding them so that we couldn't see what tools they were using and he wanted both additional information about which processes were running on the as the attacker was moving around which wasn't available in the built-in windows event logging at the time but he also wanted to be able to capture those tools so that we could see what those tools were made of so i wrote sysmon thomas garnier joined me shortly after the first version and added additional capabilities like capturing clipboard activity now these special features of capturing shredded deleted files and clipboard activity were something that we didn't introduce into the public version until just the last few years in fact just within the last year but they were in the internal version and ones that we would share with key partners now it's all all the capabilities are in the public a version of sysmon and sysmon has become one of the most popular system security monitoring tools available for windows it's literally installed on hundreds of thousands or millions of machines and corporations across the world and including at some of the most security sensitive organizations like government agencies one of the key capabilities uh value propositions at sysmon is that it gives you very detailed information about things like process create network connections but it also has very rich filtering capabilities so you can create monitoring rules that will only have sysmon collect information about what is interesting or relevant to you and ignore the rest and this is what made it has made sysmon so unique and so popular one of the things that you are also available now is a whole ecosystem has been created around sysmon of people sharing information about how to configure and deploy sysmon including a couple like a couple out there that have created community configuration files that have hundreds of rules in them to that are even linked with miter uh detections so that you can just simply take those configurations deploy them on your corporate network know that those have been validated and are running at massive scale and also designed to capture very security relevant information i'll just quickly show you cisco in action here because sysmon when we run it you can see that i've already got it running here and the config file i just ran it from the command prompt and just ran it enter to install without any specific configuration in that mode it already collects some interesting information like process termination and process creation with detailed information as well as a few other types of information so if i go into the sysmon event log here you can see process create process terminated rules and for process create by double-click you can see here's sysmon which i just ran just a second ago with the command line the current directory the user the logon with the terminal session id as well as the hashes that i've configured to capture here the sha 256 hash which is put the default hash that's collected plus the parent process and child process one of the unique features of sysmon here that is a little bit different than the built-in windows monitoring is that every process is given a unique identifier the process grid that sysmon assigns and so this lets you even if processes have the same name to identify the direct linkage for instance a process to instance a process so i can see that the parent process grid of this sysmon instance is that process grid i can search the log and find where that process was created to know exactly what that was and here you can see the full list of information that's captured by sysmon from process create all the way to file deletes that you can just log file deletion or process tampering which is process hollowing detections and much more now that basically has been a focus of the system journals tools on windows me and bryce actually wrote some sysinternals tools for linux back in the early 2000s filemon which was a tool we originally created for windows we made a version of it for linux but because of the way the linux kernel evolved it became hard for filemon to keep up without being completely broken and so we we eventually deprecated that tool removed it so there were no system turns tools for linux for a long time up until the last few years and over the last few years we've introduced three system journals tools for linux all versions of the windows tools but designed to work on linux the first one is proc dump there at the bottom it creates the same kind of principles creates process dumps based off trigger information then process monitor which is something new where we're releasing process monitor activity for linux capturing system calls letting you do the same kind of filtering you can for windows but aimed at console interfaces and then finally sysmon for linux and the benefit of sysmon for linux is that unlike the other tools that are available for auditing system events system security related events on linux sysmon for linux has the same exact schema and configuration file and log file schema as sysmon for windows and that means that if if you like many have both linux and windows in your environment you can use the same tool with the same kind of design for schema and capture like process creation capture across all your different operating systems so that brings me to the conclusion of the overview hope you found it interesting hope you found the tools compelling and want to explore some of the other tools of the 80 and the suite but again stay tuned for follow on expert deep dive sessions into those tools that i highlighted as i went through the the talk i showed you kind of a real world case the unexplained a case that explained is a series of talks that i ran uh on tech ed and then ignite sessions as well as other conference sessions where i would collect people would send me troubleshooting cases like the one i just showed you and with log files and screenshots to demonstrate how they solve those tools with cis internals i've got the collection of those kits unexplained talks if you go to system channels. com you can go find those sessions everyone has different cases with different ways that people use the tools to troubleshoot so it's a great way to become an expert in trouble using the tools to troubleshoot and if you want to go a little bit deeper the official book on system journals tools is called troubleshooting with the assistant journals tools which is not only about troubleshooting but about the tools themselves and it's authoritative it was authored by myself as well as a friend aaron margosis it covers every tool every feature with tips on how to use the tools and has full chapters on the tools that we covered and more and has 45 example troubleshooting cases in text form some of those captured from my blog some of those brand new so that's also a great rich source of information and finally i'm excited to announce and instead of having to always keep up with systemturtles.
