Transcription performed by LeahTranscribes[FULL TRANSCRIPT]
JACK: Hey, it’s Jack, host of the show. Let me ask you a question; whose job is it to keep the roads you drive on safe? [MUSIC] Is it the driver’s sole responsibility? What about the car-makers? Are they responsible for keeping the roads safe for other drivers? What about the cops? Maybe they need to come by and watch everyone and make sure they’re obeying the law and keeping everyone safe. Or wait; maybe it’s the job of the civil engineers, the people who design the roads. I mean, a crazy, curvy, bumpy road with a speed limit of 100 miles per hour is obviously not safe. It must be their job to design it to be as safe as possible, right? Whose job is it to keep the roads safe? All these people. We need drivers to drive safe, we need cars to be built with safety in mind, and we need cops to catch people who aren’t being safe, and we need civil engineers to design us safe roads.
I think this analogy applies to keeping our networks and computers safe, too. We need users to be smart at what they click on and do, and we need software makers to design the software to be secure. We need the cops to arrest people when they break the law, and we need groups who set up industry standards that guide us to safety. We cannot rely on one person to keep our networks safe. It takes all these people to always be vigilant to keep our computers safe. This is a story about what happens when a software maker finds a bug in their own software and what those effects were. Specifically, this is a story about when Microsoft found a massive bug in Windows which paved the way for the largest worm in history.
JACK (INTRO): [INTRO MUSIC] These are true stories from the dark side of the internet. I’m Jack Rhysider. This is Darknet Diaries. [INTRO MUSIC ENDS]
JACK: I’m very excited about this episode because I think this is a rare story to hear. We aren’t going to hear it from the hacker who found the exploit, and we aren’t going to hear from the company who got hacked by this exploit. Instead, we’re gonna go right to the source to hear the story of one of the most notorious bugs ever, right from the horse’s mouth; Microsoft. [SKYPE] Hello!
JOHN: Hey Jack, can you hear me?
JACK: Yeah, I hear you pretty well. What kind of mic are you using?
JOHN: This is a headset. It’s actually made by Microsoft.
JACK: This is John Lambert.
JOHN: My name is John Lambert and I work at Microsoft.
JACK: He’s been with Microsoft a long time. While he was there, he spent ten years working on a team called the Trustworthy Computing Group.
JOHN: That was a group that was created after the early worms of Code Red, Nimda, Blaster, Slammer, and it was designed to help improve customer trust in Microsoft products by fortifying them from a security privacy and reliability perspective. The domain I worked in was security and so a lot of that focus was on finding and eliminating vulnerabilities and designing and coding from the products, and working across the Microsoft product suite; not just Windows or Office but all of the products to sort of build the techniques that security researchers had familiarity with and then inculcating that ethos and know-how inside of the Microsoft development cycle.
JACK: Whoa, what an interesting role to have, right? His mission is to improve the trust people have in Microsoft products by making them more secure. I just looked this up; in 2008, Microsoft had 91,000 employees. With that many employees, yeah, I guess it makes sense to make [00:05:00] a team to improve the trust of their products. Oh, and in case you’re wondering, in 2019, they had 144,000 employees. Now, of course, the biggest Microsoft product that John would work with is Windows, the operating system itself. John got the ability to examine Windows in unique ways, by looking at the source code and building relationships with developers and conducting security tests against it. It’s not really his job to make sure all the bugs are squashed or to find the bugs, but he was certainly involved in getting all the teams together to help find the bugs and get them fixed. Okay, now when a team at Microsoft finds a bug in Windows, they give it to the developers who then work on a fix and create a patch for it. They issue these patches on Tuesdays.
JOHN: [MUSIC] Every Patch Tuesday which is every second Tuesday of the month, we will create a number of security bulletins. The bulletins essentially are the advisory that describes here are the vulnerabilities that we are fixing in say, Internet Explorer, or Windows, or Office. Every bulletin may have one or more vulnerabilities that is being fixed. Those have individual CVE identifiers that security professionals are familiar with. But the bulletin is essentially a grouping of them for a product. The bulletin is rated critical important, moderate; the level of severity.
JACK: Now, these advisory bulletins put out on Patch Tuesday might have a name like MS07-029 or MS08-067 and if you see something like this, MS07-029, it means the advisory was published in 2007 and it was the 29th advisory of the year. For this story, we’re going to be talking about the security bulletin MS08-067 which means we’re talking about this bug which was found in 2008. Vista had just come out the year before but its adoption rate wasn’t that strong, so the majority of Microsoft Windows customers were using Windows XP still. Take a guess at how many lines of code it took to create Windows XP. You ready? Forty-five million lines of code. [MUSIC] Of course, that was spread out among many teams. Trying to keep a program that big bug-free is, well, in my opinion impossible.
It’s just way too big to keep it bug-free. There has to be some function or routine in the code that didn’t get tested properly or coded properly and has a glaring bug. Or sometimes it gets tested and coded properly but there’s a bug that’s not so glaring in there. It only appears under weird circumstances, like only when the memory is filled and it’s only a certain millisecond on a certain time of day. Weird stuff happens when you have a codebase that big. All this is to say because Windows XP was forty-five million lines of code long, just by that size alone it must have had a lot of bugs. But before we get into MS08-067, maybe we should talk about a bug found the year before called MS07-029.
JOHN: Okay, so that MS07-029 was a very important – it was sort of the moment of insight, if you will, for all the things that came after it. What MS07-029 was, that was a bulletin that corrected a vulnerability with Windows DNS. When Microsoft became aware of it, a customer that was being exploited in the wild contacted us and said somehow, we just got attacked. Here’s an attack tool that we were able to find associated with it. We think it’s exploiting a vulnerability somehow. That goes into MSRC.
JACK: [MUSIC] MSRC is the Microsoft Security Response Center. This is the team of security professionals within Microsoft that is in charge of making Windows secure and looking for bugs and getting them fixed. They took this report they got and tested it to see if you could hack into a fully-updated Windows computer with it. It worked.
JOHN: Then, they investigated and found there’s a zero-day in Windows DNS that is being used. They put out an advisory, they put out a bulletin. What I was doing, is I was looking at this crash dump system that Microsoft has, this Windows error reporting, and I was working with some engineers that were looking at the crash reports coming in associated with Windows DNS. The DNS product was otherwise very reliable and hardly ever crashed. The only crashes coming in at that point, when we looked at them, were actually attacks in the wild. When we started to examine them, we could tell that actually, we could have known about this attack in the wild much earlier than the customer contacting us if only we’d been able to pull, you know, these needles out of the massive haystack of crashes coming into Microsoft.
JACK: Do you ever use Windows or Internet Explorer or [00:10:00] Microsoft Word and the app suddenly crashes and it says, ‘Do you want to send the crashed data to Microsoft?’ This is what’s known as W-E-R; WER.
JOHN: Yeah, so Windows Error Reporting, WER. It’s also known internally as Dr. Watson. It’s really a technology that goes back to the late 90s. Both Office and Windows were independently working on how do we get better data on how the products are crashing before we ship them? Then after we ship them, other than customers calling customer support, how do we get data about how they’re faring in the wild? Both Office and Windows had built features to collect crash reports at scale that could be automatically submitted by customers and then run analysis tools against them to try to root cause and bucket them to say okay, is this a new bug we don’t know about or is this an existing bug that we do know about happening again? Those two efforts came together and that feature was built into Windows and that was called Windows Error Reporting.
JACK: Yeah, that little box that pops up when the app crashes will ask you if you want to send the crash report to Microsoft.
JOHN: If they say yes, then that communicates with Microsoft and the back-end system decides is this something we already know about or is this something new? If it’s new, it starts to prompt the user to potentially upload more data so we can root cause the issue further.
JACK: Now, remember I said that XP had forty-five million lines of code. A program that big is impossible to be bug-free. Yeah, I don’t think it’s just a dozen bugs in a program that big. I don’t even think it’s just hundreds of bugs, or thousands of bugs. I think there’s more like tens of thousands of bugs in a program with forty-five million lines of code in it. At the same time, Windows was installed on a billion computers in 2008, so Microsoft was seeing millions of crash dumps a day from these computers. Let’s try to be a project manager for a second here and come up with a strategy to tackle these millions of crash dumps a day. We have a few options; first, let’s just filter out all the known bugs we’ve already fixed. People’s computers are crashing but they just need to patch it and it won’t crash anymore. Okay, that’s out of the way. But now when we try to prioritize what to look at next, it’s not so easy.
It might make sense to tackle the bugs that show up the most but these might be very low-severity, maybe not as important. Maybe there’s something with a more high severity but not as many crash reports. Then you might decide to look at the highest severity crash reports instead. Or maybe some apps are more important than others. Like, if MS Paint crashes, it might not be as big of a deal as if the whole computer was crashing. Or maybe you can look at the easiest-to-fix problems first; get those out of the way. It’s really hard to know what to prioritize here so this might give you a better understanding of how hard it would be to look through millions of crash reports a day to figure out what to fix first. But it was still very important for Microsoft to collect these dumps and analyze them. At this time in 2007, John was starting to look at these crash reports to try to make sense of them.
JOHN: [MUSIC] They go up to a massive automated system. These tools run against them at scale in a completely headless way. They’re sort of bucketed and binned in a way that makes it easy for engineers to know is there something new coming along or how active, how prevalent is this issue? Then a wide variety of engineers across the company, if you work on Office, you look in the Office crash buckets and see what is hot for you there. If you work in Windows or any other product, you can kind of see how is my product crashing? Engineers across the company were using this to fix bugs. For example, in Windows Vista, from the Trustworthy Computing side, they did all kinds of static analysis tools to find and remove reliability issues. I think they fixed 100,000 bugs through those efforts. Through Windows Error Reporting they found another 5,000 bugs that had escaped all of those processes that they went and fixed. Every service pack, every new product from Microsoft is more reliable because they’re finding and fixing all these bugs that are manifesting in the wild. I was there on the side saying this is a fascinating telemetry system. How do we look at this from a security point of view?
JACK: You said 100,000 bugs?
JOHN: 100,000 changes to the product that were done because of – not all of those are bugs. One way to think about it is there’s coding practices that are outdated. Developers will know about these things as calling these unsafe string functions like strcpy and so forth. Instead of trying to figure out which ones are vulnerabilities and which ones are not, they said let’s just go remove all of them from the product. That’s a massive amount of code changes. Many of those are not bugs. Many of those are not vulnerabilities but it’s an easy way to just go say look, we’re going to have a new standard of engineering. We’re going to get rid of that whole class of thing by tackling that. That’s an example of some of those changes, [00:15:00] yeah.
JACK: Yeah, but I mean, are you exaggerating when you say 100,000, or…?
JOHN: No, no.
JACK: Jeez, can you imagine trying to keep something like this secure? To fix 100,000 things in the code? It just sounds like madness to try to complete. I guess this is why they needed 91,000 Microsoft employees to tackle huge issues like this. These crash reports were really helping them identify the problems that needed fixing. But even though these might be bugs, they might not all be security problems. Like, if you click Print and nothing happens, that might be a bug in the code, but is it really a security issue? A hacker probably can’t use that to take control of your computer or hack you, but John was looking at this and wondering if there was anything in these crash reports that is something a hacker could use, or maybe signs that a hacker caused a crash.
JOHN: Some vulnerabilities are discovered by attackers and exploited in the wild before Microsoft is aware of them. How could we go find that before customers contact us? There are a lot of reasons that exploits actually fail in practice, and that was the idea which is instead of trying to find exploits when they’re working, sometimes they don’t. There’s a bunch of interesting reasons why they don’t. By studying the causes of failure of those exploits in the wild, that would lead us to potentially discover them. Some of the reasons they fail are because the exploit was written for, say, an English language version of Windows and it’s run on a Chinese language version and they’re slightly different internally. A variety of reasons that exploits would fail in practice. I studied extensively all the different patterns of how they fail ‘cause that was what I needed to be able to understand to find them.
JACK: [MUSIC] The hunt was on for John. He wanted to find traces of hackers in the WER logs. But like I was saying, there are millions of error logs a day, so this wasn’t going to be easy.
JOHN: I think back then it was over a billion a month. Recognize that it’s like, how could you look through a billion a month? Because of the way Windows Error Report buckets the issues, you don’t have to look at every single one one-by-one. You’re really trying to find new ones, new ones that are interesting in a different way. Then another way to think about it is there’s only a certain number of ways that zero days are going to show up and affect an application. People can send malicious documents that’ll crash Word, so Word is interesting to look at. People can send exploits through the browser, so Internet Explorer would be interesting to look at. Then, if you think about even inside of Microsoft Word, if an attack is gonna happen, likely it’s gonna happen when the user is first opening the document.
It’s rare that some exploit is designed to work when the user’s been in Word for an hour and they finally click Tools > Options > Spellcheck > Add-In, and then it goes boom. Those are not the paths where an exploit’s gonna work. It’s gonna be in the File > Open code path. That further narrowed down the places that we needed to look. Exploits fail in certain specific patterns that are the kind of patterns that we knew to go sifting over. We were able to work that funnel down from a billion reports a month to millions that seemed interesting to hundreds of thousands that had other clues in them and so forth, and whittled the funnel down.
JACK: John dedicated a lot of time to try to find this unknown vulnerability that some hacker was exploiting. When we come back from the break, John finds something that he’ll never forget.
JOHN: [00:20:00] On September 25th, I remember opening a crash report for the Service Host process that we have seen many millions of crash reports for already because a couple years earlier there was this other vulnerability, MS06-040, that had been adopted by bots and worms to spread. It was causing millions of crashes against machines that had not put on that patch. But this one was a little different. [MUSIC] First of all, it had an exploit. It had exploit code in it, okay? That didn’t mean it could be new; it could be an old one. It was an exploit on the stack which is a critical part of memory that tells you that exploit was trying to be activated right now. It had a difference in it, what I call an egg hunt, which was an exploit technique that I had never seen before for any exploit in MS06-040. This was either a new strain of that or something new for that service host.
This egg hunt is an exploit-writing technique where, as the bad guy, imagine you’re gonna go scale a wall and you put all your attack tools in a – your thieving tools in a bundle and you throw it over the wall and then later you go scale the wall and go get that. An egg hunt breaks the exploit into two pieces to serve a purpose. It had that technique in it so that alone drew my eye to look at it. Then, the odds of there being a buffer over on an exploit in the same area as this MS06-040 just seemed unlikely and yet I felt like if it was new, this was really important. So, I just tried to stick with it and do what I could to rule in or rule out whether this was new. One of the most stubborn clues was in a crash report, it has information about every DLL that is loaded into it. The DLL that had MS06-040 was loaded in it. That is this netapi32. The version information told me it was fully-patched so it clearly could not have been exploiting that vulnerability because that vulnerability did not exist in that version of the product.
JACK: John looked at the logs here and it looked like a hacker was exploiting two processes. One was basically injecting some hacker tools into the system, hiding an egg if you will, and the other was going in and using those tools. A strange combo but like John said, it was sort of like throwing your tools over the fence and then jumping over the fence to get them. The two processes being exploited here were SVC Host, or Service Host, and netapi32.
JOHN: Yeah, so sometimes when you are writing an exploit, you have constraints that you have to work within. Once your exploit starts to run, you have a number of steps as the attacker you’re gonna want to do. Typically, that involves downloading some external piece of malware to that system and then launching that, and then the rest of your attack, then you’re a resident on your computer and you can do whatever attacks that you want to do further from there. But you have to get it going and sometimes that shell code, as it’s called, to get going, can’t fit within the constraints of the vulnerability.
The buffer, for example, needs to fit and is just too small a box to package all those instructions with it. So, an egg hunt is a technique that is designed to solve that by first sending over some data and interacting with the vulnerable program in a way that doesn’t use the vulnerability at all. It just tries to get data and memory that is that bag of tools, that shell code that they are going to use later. The only thing that they need to get when they actually run the attack is a very small piece of shell code that basically goes and searches memory to find that bag of tools.
JACK: What a tricky exploit. Because there’s two parts to this bug, it makes it really hard to find. The more John looked at this particular crash report, the more he started to realize this was actively exploiting an unknown vulnerability in Windows which makes it a zero-day bug.
JOHN: [MUSIC] This, in a way, was the hardest moment of this entire thing for me because I clearly had enough evidence to say this was a new attack, a new vulnerability that we did not know about, and to get the company to act, you need to actually pinpoint the vulnerability. We have to know what’s the code that’s wrong? Otherwise, nothing can happen. I pored over the code and pored over the code and I could not figure it out. At one point I decide the clock is ticking; [00:25:00] this is a potentially really bad situation. I need to go ask for help if I’m gonna figure this out. I walked over to the office of Andrew. Andrew was a colleague of mine. We had worked together previously, actually. Andrew had looked at many crash reports for security objectives for a long time.
He knew what this hunt was like, which was in a lot of ways, it can be a very frustrating hunt and you can spend hours looking at something you think is real and then it’s not. What I wanted is, hand this to one of the MSRC engineers and they will go figure this out. He was reluctant to take an engineer off of an existing confirmed vulnerability that somebody else had reported and was in the middle of to go potentially chase a ghost. He took it on to go look at it, to, I think in his mind say look, this is a false positive. This is not a real issue. Case closed. There was some tension in the air with me really pushing a busy team to go look at something that likely was not going to pan out but I felt like it could be important enough; it’s worth doing that, Andrew wanting to protect his team but do the due diligence to make sure we got the answer right.
JACK: Andrew got a copy of this crash report and the code for these processes. He started analyzing all this. He spent a couple days looking it over and working on it, trying to find what the crash report means. There’s an egg somewhere in the code but neither of them could find it. But this error report did say that there was something causing a crash. Now, keep in mind, John had only seen one crash report from this. This wasn’t a widespread issue so both John and Andrew weren’t entirely sure how much effort they should spend into looking for something that only happened once on one computer. But if there was an unknown vulnerability, both John and Andrew wanted to find it and fix it, so Andrew kept looking.
JOHN: Then at one point he stops by my office and the look on his face told me everything. The look was the look that a security researcher has when they found something. There’s a goofy, happy smile that is also full of holy cow, I can’t believe how serious this thing is that I just found. [MUSIC] He said I found a vulnerability. When I heard that, I knew everything was gonna – the next two weeks of my life were gonna be completely different because this vulnerability was in an area that would allow an internet worm to be written against it.
JACK: The vulnerability they found allowed an attacker to take remote control of a Windows computer. No need for a username and password, no need for RDP to be enabled or anything like that. Just give me full control of that computer and now I can issue my own commands on it, see any files, do whatever I want on that computer as if I’m right in front of it. Of course, to top it off, it’s a vulnerability in a fully-updated version of Windows. In the security world we call this kind of vulnerability an RCE which means you can do Remote Code Execution. Like, a hacker can execute whatever commands they want on the victim’s machine, and this is the worst kind of vulnerability. It’s the most critical, the most severe because you absolutely never want just any Joe Shmoe executing commands on your computer which has full defenses up. If you can run whatever commands you want on someone else’s computer, it pretty much means it’s your computer. But to top it off as if this highly critical and severe vulnerability wasn’t enough, this exploit was wormable.
JOHN: Wormable means that you have a vulnerability that an attacker can write an exploit for and it can propagate across the internet, exploit that vulnerability, and then turn around and continue to repeat the process and propagate further. It becomes a viral outbreak and it is the most damaging, devastating, disruptive kind of attack that can take place. The world had seen many worms in the form of Blaster, Code Red, Nimda, Slammer, Zotob. We knew how devastating these attacks can be; entire businesses are disrupted, systems are taken offline, network traffic gets clogged with worms replicating out of control, using up all available bandwidth. We knew what the potential was. We didn’t think that that state had been happening yet but we knew that was possible.
JACK: When Andrew came to John’s office and [00:30:00] told him this vulnerability is real and wormable, the two immediately jumped up and sprang into action.
JOHN: Andrew and I both were on the engineering side and we knew we needed to go activate the crisis response part of Microsoft. He and I immediately leave my office. We walk down the hallway to the office of the crisis manager. His name is Phillip. Phillip has worked on many of Microsoft’s most important security crises and he’s chatting in his office with a colleague. We show up. He knew that wasn’t a good sign. We look at him and say we need to talk. He looks at his colleague and says I’ll talk to you later. Then I said we have a zero-day. He just knew by the two of us showing up in that fashion that something bad was happening.
I mostly had two emotions going on; one is we need to get all of Microsoft engaged on this that can do something about this. That is an impetus to go make sure you’re informing people so they understand the severity right away and they can begin the mobilization process. Then the other side of me was what is really going on out there? I have one crash report. I don’t know if there are a hundred thousand out there just like this and this is a worm that is raging across different parts of the world, or this is something that’s just getting started. I immediately wanted to go and get some better situational awareness about scope and scale to know what we were really dealing with.
JACK: John and Andrew briefed the Crisis Manager on what’s going on, that a very serious wormable and extremely critical vulnerability is present in Windows and needs to be fixed immediately. [MUSIC] While they found this in Windows XP, the vulnerability existed in more than just XP.
JOHN: Yes, and it affected every version of Windows up until we had at that point. Microsoft has a crisis response process that they invoke when one of these things occur. It’s called a CRP, internally. Then a call bridge is stood up and then all of the crisis partners across the company join that call and then Phillip would take them through; here’s a summary of the situation. Generally speaking, if there’s a vulnerability in Windows, teams know what to do. This would involve a lot more teams because they had to be ready for customer support calls. If there’s an attack, what is the malware? What does the threat side of it look like? The anti-malware team will start building signatures for that. We knew we had to prepare data for all of the security partner companies across Symantec, MacAfee, or whatnot for them to help go protect their customers and the ecosystem. The engineering team needs to go okay, what do we have to do to fix this vulnerability? Are there any others just like it waiting that we need to fix at the same time so we get the patch dead right?
JACK: A huge conference call was set up with pretty much someone representing all the different departments of Microsoft. The goal was to get everyone engaged in helping solve this as quickly as possible. This vulnerability was much more severe than any of the others they found.
JOHN: Once I knew that the right people were engaged and working to get the right patch out the door, the thing I could go help on was how often is this happening and where is it happening? I call that dialing into the crash buckets to find any information I could about how often this was occurring. I was able to bring back the situational awareness to the crisis response that said okay, I saw it five more times today. It just spread from Malaysia to Japan to Singapore to whatnot and they can kind of get a sense of okay, it’s the same attack payload communicating with the same IP address and then you’re gonna have to pull it down. But it’s spreading from geography to geography to geography. It didn’t look like a worm but it looked like some set of hackers that were going around using this tool across a variety of targets, expanding its scope every single day.
JACK: See, that’s fascinating. Somebody knew this vulnerability. They found this before Microsoft did and was using it. This wasn’t just some accidental bit flip or something that caused the crash report. This was, actually, somebody had coded this and was using it in Asia.
JOHN: That’s right, so somebody found this vulnerability, probably got the bonus of the year for finding it. Then it was [00:35:00] weaponized and then some group unknown to this day was using it in those geographies where we saw these crash reports coming. The tool was not completely reliable and so the crashes that I were seeing, that’s when it failed. But obviously, it worked and it succeeded a bunch of the time, and I didn’t know how often. For every crash is that one in a hundred times that it’s crashing or is it one in a thousand? What shadow of the real activity am I seeing? That was an unknown that we were working against but I could use this to get a sense of its spread and its reach and its pace.
JACK: This became a top priority for the development team to fix. They dove into the code and fiercely started to rewrite the code to fix this problem [MUSIC] and they did; they fixed the vulnerability. But this is not the end of our story, no, no, no. This is just the genesis. Stay with us because after the break, we’ll hear how this went on to become one of the most notorious bugs ever to be found in Windows. Now that a fix was written and a patch was ready, a decision had to be made about how to release this to the world.
JOHN: Right, so there’s a decision point as part of this response process which is okay, when is the patch gonna be ready? The ideal time to release a patch is on Patch Tuesday ‘cause every IT team in every company knows on the second Tuesday of every month there will be a set of security patches from Microsoft and they know to organize their patching of their fleet of systems and take their down-time and their maintenance time to go put them on. They know some of them will be severe and they should do that right away. IT teams are best ready to consume those updates and put them out at that time. The other option is you just ship it when you’re ready. We call that going out of band. That gets the solution out there sooner but at the same time, none of those IT teams are necessarily ready and poised to go grab it.
We had a situation where there obviously was an attack in the wild that attackers were doing and we wanted to go stop that from happening but we knew as soon as we put this out, copycat attacks would happen by people reverse-engineering the patch and understanding. That would spawn a whole other wave of attacks and so there was this weighing of if you go out of band, you could spawn a bunch of copycats of a very damaging vulnerability when IT teams are not ready to go put it on and there could be more victimization, theoretically. The bottom line in terms of our calculus on it was we had a very severe vulnerability. It was wormable. It affected every version of Windows. We had a solution and we went out of band.
JACK: This vulnerability was so severe that they decided to not wait until Patch Tuesday and just push this out immediately, as soon as they got it. This patch went out in 2008 and it was the 67th patch of the year which famously made this MS08-067.
JOHN: There were two most interesting things that happened after we went out of band. The first one was that attacker where I was seeing crash reports still coming in every day until we went public. As soon as we went public that attack disappeared and I never saw him again.
JACK: How can you trace that? You have a signature for certain machines that kept getting attacked?
JOHN: Well, part of the attack details [00:40:00] that were hidden – so one, the shell code and egg hunt were kind of a fingerprint that let me see, that were consistent across all the crashes I had gotten to date for this issue. All of them were contacting the same IP address of a server in Japan and downloading a payload from it. All of that was consistent. Then the day the patch goes live and we announce it, I see no more crash reports from this anymore. Then there’s this period where, for a number of hours or a day, nothing’s coming in anymore. Then we start to see new crash reports for the same issue that were clearly security companies reproducing this vulnerability.
They had reverse-engineered what we had fixed in the patch. They were writing their own proof of concepts to crash it; not write exploits for it but just to probe the vulnerability to make sure they understood it. We started seeing those crash reports come in from the security researcher side. Soon enough, a few days later after that, we started to see new attacks from the first wave attackers that was clearly different and new than the original ones, that look like bots or botnet programs adopting this as a spreading mechanism. We started to see that wave as well.
JACK: See, this is what I mean by this being only the genesis. Now that the patch is out there, both white hat and black hat hackers analyzed the patch to figure out what this exploit was and how to run it which means now this tool went from being just used by a single hacking group somewhere in the world to now being known by the general hacker community. Within a short time, it would become available for anyone in the world to just download and use. [MUSIC] Isn’t that a strange dilemma or decision to have to make, though? Knowing that if you put a patch out, this reveals the vulnerability to the world for any hacker to use? But Microsoft has a duty to make secure products so they absolutely have to release a patch whenever they do find a vulnerability like this because this has far-reaching effects on helping people stay secure.
JOHN: In the first week it was on Windows Update and within seven days I think we had patched four hundred million machines. This is sort of the awesome part about Windows Update; it’s a system that had been built to patch the Windows ecosystem at scale and this is one of those times when you really needed it. It came through in terms of our ability to essentially inoculate a huge swath of the world against this vulnerability in a very short period of time. It was very effective about that.
JACK: It’s hard to know for sure but my best research tells me there was about one billion Windows computers in the world at that time and this vulnerability affected all of them. By having four hundred million of them patched in the first week, that was a huge win for helping the world become more secure. 40% of the Windows computers in the world were no longer vulnerable to this right away. That’s awesome and amazing.
JOHN: All of this was in October and sometime in early December, a Conficker worm that had been this sort of small-scale thing for a time, adopted this vulnerability as a spreading mechanism and then it began to use that against systems around the world that had not put this patch on yet. If you think about well, what if only 1% of the Windows PCs in the world don’t patch? That’s still millions of systems. [MUSIC] A lot of the damage that was done by Conficker which was significant, imagine what would have happened if it had a half a billion more PCs that were vulnerable and not patched against this issue.
JACK: Conficker was the first big attack to use this exploit. Just months after John discovered this vulnerability, Conficker figured it out and used it to arm itself to infect Windows machines. Because the vulnerability in MS08-067 was so effective, Conficker spread rapidly worldwide, ultimately infecting computers in over 190 countries worldwide. It would eventually infect millions of computers. Conficker was spreading in a terrible way. Think about how horrible this is, for some hacker group to have full control of millions of Windows computers worldwide. I’m talking about government agencies, businesses, home PCs.
The hackers could see all the files on them, run any clients they wanted, install key loggers, take screenshots, install rootkits, or do whatever they want on these computers. It’s so frightening to think about that. [00:45:00] Conficker continued to spread, seemingly unstoppable. By January, 30% of Windows computers still did not apply the patch to protect themself from MS08-067 or Conficker, which means hundreds of millions of computers were still vulnerable to this. Conficker had a field day with everyone who didn’t bother to patch and eventually was able to infect ten million Windows computers worldwide. Ten million. As far as I know, this makes Conficker the largest worm ever, all thanks to MS08-067.
JOHN: It was disappointing because – at the same time a strong lesson because we had put the patch out, right? The job is done. It’s time for the world to put the patch on their systems. That’s the step they have to do. There isn’t anything further that we thought we could do and yet look what happened and what came after that; all of this damage from Conficker. Now, Conficker had other ways to spread, through USB drive infections and scanning file shares and so forth, but still there was a large part of the world that had not patched against this vulnerability. It was a bit of a lesson for just how damaging things can become and how much of the world can be exposed to these attacks even once we think we’ve done our part of the job.
JACK: It’s fascinating to me to see that John here was the one who decided to take it upon himself to look at that crash report and to discover this. If it wasn’t for him, who knows how long this would have stayed out in the wild before being discovered?
JOHN: Yeah, I think these operators using this tool thought they had a really great new attack and probably would have used it if it was more reliable and we had not seen this for potentially years if they were stealthy and choosey about how they used it. Sometimes these exploits we do learn about zero-day have been in use for years in very disciplined ways. The noisier operators are with it, the more likely that some victim is gonna find out about it and somehow, they’ll get a whiff of how the attack is working and the thing can get patched. But I think if we had not have seen this one in this way, very likely this operator could have continued for a very long time with it.
JACK: [MUSIC] Hm, this makes me think about how stealthy some hackers could be. I mean, imagine if the hackers disabled WER or blocked all connections to Microsoft. This would have been an effective technique to keep Microsoft from seeing these crash reports and discovering this. But maybe that’s going too crazy with it. But see, sophisticated hackers take extreme measures to hide their tracks. I mean, that’s almost one of your adversaries, right, are these APTs like the NSA. You’re battling with them like it’s a arms race between you as the vendor and them as the aggressor. Do you feel that way sometimes?
JOHN: It does feel like that way sometimes in the sense of there are people that have weaponized vulnerabilities and are using in the wild. In a way, the defenders that are at Microsoft, Google, and other companies don’t care who these people are or why they’re doing it. We just want to find what they’re doing and take those tools away from them. Every time a zero-day is found in the wild by some defender or organization and it gets patched, that is a happy day for me.
JACK: What a strange thing to think about. Does that put you in deep thought too? The truth of the matter is that the NSA is actively looking for vulnerabilities in Windows so that they can use that against their adversaries. Then here’s John actively trying to figure out what the NSA is up to so he can basically expose their secret weapons. I don’t know what to think about that. I don’t know who the bad guy or the good guy is in this. NSA is supposed to be working towards keeping our country safe but at the same time, they have to develop cyber-weapons to attack other nations, so it almost seems like the NSA would see John and Microsoft as the enemy. John might see the NSA as the enemy. I just never thought about how these two would be battling each other like this. It’s just wild to think about this relationship.
JOHN: Yeah, and then in many ways I feel like I relived this whole moment a couple of years ago when the EternalBlue exploit was discovered. This is one of these NSA tools that the Shadow Brokers group leaked onto the internet. A patch was produced for it; that was MS17-010. A couple months later, the WannaCry worm was unleashed and spread in a very similar way. It had a more destructive payload and [00:50:00] ran across the globe against systems that had not patched.
JACK: Now, if you recall in earlier episodes, NSA actually did tell Microsoft about EternalBlue just before the Shadow Brokers published it to the world, giving Microsoft an early warning who were able to move quickly and patch this before it was released. I’m guessing the NSA knew it was going to be published and wanted to help the world just stay slightly ahead of the game. But that must have been an awkward phone call, or whatever; for Microsoft to get the memo that NSA has found a devastating exploit in Windows and this exploit got leaked to Shadow Brokers and they’re about to leak it to the world. Phew, I don’t know. I don’t know, it’s a weird, tangly mess when you get into the relationships between NSA and Microsoft. Just to be clear, there’s no link between MS08-067 being found by the NSA. Oh, something happened last week that’s kind of interesting, too. Last week was Patch Tuesday and boy, was it a doozy. There was a bug patched.
It was a cryptographic API bug in Windows 10 which basically allows an attacker to pose as someone they aren’t and your computer could send trusted information to it. But here’s the thing; this bug was reported to Microsoft by the NSA. In fact, it was so important, the NSA even held a press conference to urge people to patch. This is very rare; I mean, we don’t know how many times the NSA has reported bugs to Microsoft. They could be doing this all the time. But we do know for sure that there were two times that they did do it; once when Shadow Brokers got the NSA’s exploits, and now again last week. The NSA says they told Microsoft because they want to build more trust with people and help keep computers secure. Hm, really? It could be that.
Our world is changing and new things are happening and the NSA might be doing this from now on to try to keep the country safer by working with vendors to get things patched. In fact, they have done stuff like that for a while but it’s all been small potatoes versus the god-mode bugs that the NSA keeps to themselves. My theory is this; the NSA gave this bug to Microsoft. Why? Maybe it was just to build better PR. Okay. But then maybe the NSA knows something we don’t, like maybe they uncovered a huge man-in-the-middle campaign that some foreign government was doing to many Americans and thought it could have devastating results, so this was their way to stop it. Or maybe the NSA lost another exploit and didn’t want their enemy to have it. I don’t know but I have a feeling there’s more to this story, though.
[MUSIC] Back to Conficker. You might be wondering what happened there. Like I was saying, Conficker infected ten million computers and was growing. However, it was a mystery as to what Conficker actually did or who made it or what it was supposed to do. While it was infecting systems worldwide, it was apparently not doing anything once it was infected. On a periodic basis, the computer that had Conficker running on it would reach out to certain domains to receive instructions on what to do next, but it just didn’t get any instructions to do. Security teams all over the world feared that maybe there were instructions to do on a particular day and in fact, for some weird reason, we thought that on April 1st, 2009, there was gonna be some big surprise that Conficker was going to give us. I remember being in the office that day and setting up a conference bridge in a war room with everyone from IT and looking for signs of Conficker kicking up or something, but nothing happened.
A few companies got really fed up with Conficker spreading everywhere so they decided to do something about it. In February 2009, something called the Microsoft Cabal was formed. This included people from Microsoft, Verizon, NEWSTAR, America Online, Symantec, F-Secure, researchers from Georgia Tech in the Shadowserver Foundation, and so many more organizations. It was a huge list of companies that came together to figure out a way to stop Conficker. They would do things like reverse-engineer Conficker to see how it worked and then write fixes to block Conficker from spreading more. But then, whoever made Conficker would change how it was infecting machines so it could keep infecting machines, creating a new variant of the worm.
It became a game of cat and mouse, with the security professionals blocking it and the worm creator getting around that. At some point, Microsoft said they’re willing to pay $250,000 for information that leads up to the capture of whoever created the Conficker worm. They were taking this pretty serious. The FBI started getting involved and the hunt began to look for whoever was running this worm. The author Mark Bowden did some extraordinary research into this worm in his book which is just titled Worm. He writes that there are a few theories in what Conficker was. One is that is was just a security researcher playing around, making a crazy worm, but never intended to do any harm with it; just trying to see how big it could get. Another theory was that a government created this worm and was waiting for instructions to maybe attack on command or spy on people or something.
[00:55:00] But as more organizations joined the Microsoft Cabal, their more effort put into looking into this, and they may have found the answer. They reverse-engineered the code and did everything they could to trace it back to their creators. They handed this information over to the FBI who then arrested three Ukrainians; Sergey, Yevgen, and Dmytro. These Ukrainians all were millionaires. They drove black Porsches and they lived in penthouse apartments. Their story was that they ran a website with employees and everything and they paid themselves. But they weren’t paying themselves very much. They were arrested on tax evasion charges and the feds seemed to have found some evidence of Conficker code on their work computers but I don’t think we have any idea what happened to them next.
It doesn’t seem like the FBI was able to extradite them to the US and they just disappeared into the Ukrainian courts. [MUSIC] But the FBI also arrested one other guy; a Swede named Mikael. Mikael was arrested in Denmark in connection with this and extradited to the US. The court records don’t say anything about Conficker, though. Instead, the FBI found evidence that Mikael was infecting computers and putting scareware on them. This is where he would infect a computer and say there’s a virus on it and you need to buy this antivirus. But when the victim buys the antivirus, nothing actually gets fixed. The FBI claims Mikael made seventy-one million dollars from his scareware campaigns which is a really big haul.
To get that much, you must have had a lot of infected machines and there was evidence in some of the variants of Conficker that it was capable of running scareware, so it might have been getting ready to launch a big campaign to do just that, but it never did. Mikael got two years in prison for his scareware scams that he was running and it’s alleged that he had ties to those Ukrainians who also got arrested over Conficker. It seems like the best theory now was that Conficker was made by a group of Ukrainian cyber-criminals who may have been planning on using it to send spam e-mails or running scareware to scams its victims, but they never got to it. What’s truly fascinating about Conficker is that it’s still out there, infected on a ton of systems. Even though MS08-067 was patched in 2008, there are still computers out there that are running systems older than that that haven’t been patched still, and the latest estimate is that Conficker is still present on 400,000 computers today.
MS08-067 will go down in history as one of the most notorious vulnerabilities in Windows, ever. The reason for it is because of how effective it is. I personally love playing around with this vulnerability and exploiting Windows computers with it because it’s so easy to do. I want to walk you through how I’ve done it. First, you need a version of Windows from before 2008 which is actually quite easy; you just install Windows XP on a computer and don’t patch it. This will be vulnerable to it. Then you need to run this exploit against it. Now, instead of knowing what shell code to send to the computer and work all that out, there’s a crazy shortcut. It’s called Metasploit. Metasploit is an incredible hacking framework. It has over a thousand exploits all pre-programmed and ready to run. You pick the MS08-067 exploit, then point at your Windows XP machine, type run, and boom. You’re in.
Now, when I mean you’re in, you’re really in. Metasploit has tools to allow you to use that computer you just infected as if you’re right in front of it. You can run any command you want on that computer, all through the command line, take screenshots of the desktop, enable the mic, enable the camera, run a key logger to watch what someone types. You can do all that and more. Metasploit is an amazing hacker tool which is the standard for any hacker to know how to use today. The best part about Metasploit is that it’s free and open-source. Anyone can grab it, study a few commands, and have over a thousand exploits ready at their fingertips. It’s really powerful and fun to play with, and if you attend any ethical hacking training, chances are you’ll be given Metasploit and a system vulnerable with MS08-067 as one of your first hacks you’ll do using it, because it’s pretty easy to use and you can see how effective Metasploit can be.
Because of that, penetration testers all over the world are very familiar with MS08-067 and all have that number memorized. However, it’s now 2020 so MS08-067 is a dozen years old now which means there are far fewer computers running Windows XP or that are vulnerable to this attack. This bug is losing its notoriety. It’s much more rare to find a system vulnerable to this today but they do exist. This story is another example on why it’s so important to update your software as soon as you can. However, it’s not always that easy. Some networks have very strict controls; like, they can’t patch. A patch might break everything. The applications and software running on the computers doesn’t always work with the [01:00:00] latest OS updates applied. This quickly becomes a nightmare.
I recently updated my home computer and a lot of the software that I was running stopped working so I had to wait for each application maker to put out an update for me to be back up and working again. Something like this is totally unacceptable in critical networks like hospitals or power plants. It’s not always as simple as just patching it. Like I was saying at the beginning, it’s a lot of different people’s jobs to keep our network secure. In this case, Microsoft did their job at finding a bug and issuing a patch for it, and now that fix needs to trickle all the way down to everyone’s computers. Because by being on the most up-to-date version, it is like giving yourself a vaccination to render yourself safe from the known attacks in the world. Updating your apps, operating systems and programs, in my opinion, is the single most effective thing you can do to protect yourself on the internet today.
JACK (OUTRO): [OUTRO MUSIC] A big thank you to John Lambert from Microsoft for coming on the show and sharing this story with us. I found it to be really cool. Also, if you want to know more about Conficker, check out the book Worm by Mark Bowden. It goes into a lot of details about it. It’s pretty good. Hey, if you’re all caught up with this podcast and want more episodes, check out the Darknet Diaries Patreon page. You can find bonus episodes there. This show is made by me, the cyber-ghost, Jack Rhysider. Music in this episode was special. Typically, I grab songs from all over but in this one, every single song was created by the top talented Breakmaster Cylinder. Even though hoodies go up and drawstrings get pulled tight every time I say it, this is Darknet Diaries.
[OUTRO MUSIC ENDS]
[END OF RECORDING]