Transcription performed by LeahTranscribes[FULL TRANSCRIPT]
JACK: Hey, before we get started, check out the episode right before this one. It’s called Shadow Brokers. It kind of sets you up for this one. [MUSIC] This is the story of NotPetya and it took place in the spring of 2017. There was some weird tension between the US and Russia during that time. Donald Trump was President of the US and it’s widely-known that the Russians used the internet to meddle with the election. I mean, the FBI has indicted twelve hackers who were working with the Russian government that have allegedly hacked the DNC and Clinton’s e-mail servers which had a critical role in the 2016 election. The relationship between Trump and Putin is weird and mysterious; a ton of allegations are floating around that a lot of back channel support is given to Trump from Russia. But what is clear is that Russia likes to quarrel with Ukraine. They’ve been fighting over things for a long time but in the last eight years, things have really heated up.
Russia decided to take a territory of land from Ukraine called Crimea and besides that, Russia has been deploying troops into Ukraine, pretty much occupying the area. The stuff going on in the Donbass region is just crazy. This made tensions between Russia and Ukraine even more elevated. Now, for the last six years, Russian troops are still occupying places of Ukraine. This was the most blatant land-grab in Europe since World War II and it all happened in the last half decade. But taking over a large region of Ukraine and occupying them with troops was not the extent of what Russia did to Ukraine. There’s so much more terrifying and scary stuff that Russia has done to Ukraine over the internet. In fact, in this rare case, I’ll even go so far as to say this is a cyber-war.
JACK (INTRO): [INTRO MUSIC] These are true stories from the dark side of the internet. I’m Jack Rhysider. This is Darknet Diaries. [INTRO MUSIC ENDS]
JACK: I recently read the book Sandworm. It just came out. It’s so good; so good that I wanted to have the author come on the show.
ANDY: Yeah, I’m Andy Greenberg and I’m a senior writer for Wired Magazine and I’m the author of this book, Sandworm.
JACK: A few years back, Wired asked Andy to investigate whether or not there’s been a hack so devastating that it would be considered a cyber-war. Andy found some very interesting stuff going on in Ukraine at that time and decided to look there. He got to work researching this and was finding the story was just getting deeper and bigger than he expected. He found so much stuff that he decided to not just write a magazine article about it, but instead a whole book.
ANDY: I’ve been working on this book Sandworm since about late 2016. It tells the unfolding story of this cyber-war in Ukraine. In the midst of that, NotPetya happens, this biggest cyber-attack in history. I was kind of primed to investigate that and then I spent probably nine months of the book research time digging into NotPetya specifically, trying to find really everyone who was willing to talk about the experience of witnessing NotPetya unfold, being a victim of this global cyber-attack, experts who pulled apart the code, forensic analysts who tied it back to known hacker crews. This is really the story at the heart of the book that I’ve been working on for about three years.
JACK: Let’s get into what Andy found in his years of research which led him to NotPetya, the biggest cyber-attack in history. Now, I’m pretty sure the goal of this was to create a devasting worm. A worm is a virus that will self-replicate and spread among many other computers in the network, infecting them, too. Then, after it spread, they wanted to take that computer offline permanently, basically destroying it and everything on it. To accomplish this, they needed a few hacker tools. Now, these hackers had a plan for how to get their worm onto computers initially, and we’ll get into that later. But now let’s think; once you get your worm onto just one computer [00:05:00] in a network, how can you get it to spread to many others? Whatever method you use, you want it to work very well, meaning you don’t want it to be stopped by someone who’s just patched their computer or has Antivirus on. No, this worm has to cut through all of that, so the hackers used a tool called Mimikatz.
[MUSIC] Mimikatz is crazy and amazing and one of the most frustrating things I’ve ever seen. I could talk about Mimikatz for hours. It’s nuts. But the skinny of it is this; on Windows computers is a program called lsass.exe. This process is one that’s responsible for enforcing security on Windows computers. Yeah, well, get this; when someone logs into a Windows computer, LSASS stores your username and password in clear text in the memory. Now, this is so LSASS can authenticate that person to other things like shared drives, e-mail, SharePoint, etc, without having to ask the user for their password again and again. This is all fine and good until a French researcher named Benjamin Delpy, or the gentilkiwi, took a look in the memory. He used a tool to examine what LSASS put in the memory and was amazed to see it storing usernames and passwords in clear text, not encrypted at all.
He built a tool to extract this username and password to display it to anyone who wants to see it. That tool he made is called Mimikatz and he made it open-source for anyone to use it. He kept building on it, teaching it how to trick Windows and authenticating in so many other ways like passing hashes and tokens. It’s incredibly powerful and insanely successful because get this; suppose you break into a computer or sit down at someone else’s computer. If you download and run Mimikatz, you can suddenly see every single user who’s logged into that computer since it was rebooted. Not just their username, but you can see their full password, too. On a shared computer like a central jump server, you can potentially get the passwords to a huge number of employees and possibly an admin account, too.
The thing that frustrates me the most about Mimikatz is that for years, Microsoft refused to fix this problem. They just didn’t acknowledge it or understand it. In recent versions of Windows, they have fixed some of it, but Mimikatz continues to evolve, getting around whatever fix Microsoft comes up with. Even today, on a brand-new Windows computer, it’s not secure against Mimikatz by default. This is why it’s such a powerful exploit. Now, once the worm infects a computer and spreads, the last thing it needs to do is destroy that computer. The goal of this attack was to permanently destroy as many computers as possible. The best way to do that remotely is to encrypt everything on it, make it useless unless you have the decryption key. This is typically known as ransomware but I don’t think these hackers had any intension on making money off this. Their goal was to destroy computers and ransomware was just the perfect tool to do that. The name of the ransomware they decided to use was a modified version of Petya.
It’ll infect the system at the master boot record, instruct the machine to reboot, and upon rebooting it’ll encrypt that file system, preventing it from working at all anymore. It’ll then show this screen saying your files have been encrypted and you need to pay to get it unencrypted. Now you combine these two tools into a worm and instruct it to spread through the network. It’s very effective just this by itself. Computers that are fully-patched and updated can get their passwords taken from memory and use that to spread to other computers quite easily. The more systems it gets into, the more usernames and passwords it collects, and it just becomes unstoppable at some point. It could potentially encrypt all hard drives in a network but even though that’s a powerful one-two combo, it might not be a knockout blow. What if those computers it initially infects didn’t have any extra passwords to steal or something? Hm, so another tool was added to this worm, something called EternalBlue.
ANDY: [MUSIC] EternalBlue was probably the most powerful of all of the hacking tools dumped onto the internet by this very mysterious group called the Shadow Brokers. The Shadow Brokers appeared in the summer of 2016 and just started periodically leaking NSA hacking tools onto the internet. These are full, working, zero-day exploits in some cases.
JACK: Yeah, in the previous episode we heard all about what the Shadow Brokers did, but it was their last dump where they handed the world a devastating hacker tool.
ANDY: It included this hacking tool called EternalBlue which exploited a vulnerability in a Windows function called Server Message Block that allows machines to essentially share information between themselves. By exploiting that SMB vulnerability, EternalBlue could basically run code remotely on any Windows machine that was vulnerable anywhere in the world. It turned out that the NSA had actually worked with Microsoft to try to warn everyone about this zero-day when the Shadow Brokers first appeared. There was a patch for this SMB vulnerability but of course, as with [00:10:00] all patches, it was kind of an epidemiological problem trying to get people all around the world to implement this patch. When EternalBlue went public, there were still countless thousands, or hundreds of thousands of machines, really, that were still vulnerable.
JACK: With EternalBlue in the hands of every hacker, the world was about to be sucker-punched in ways it never imagined. EternalBlue is an exploit to get into Windows computers. It just bypasses the username and password altogether and lets the hacker right in. From there, they can look at files, upload things, issue commands, do whatever they want. Yeah, while Windows had a patch for this, not everyone was applying their patches, so the chances of this working – they’re still high, probably twenty to fifty percent, and that just might be enough to get that worm through some difficult places that Mimikatz couldn’t get into. Here’s the combo for this hack; [MUSIC] first, if the worm could get onto a system somehow and then run Mimikatz to get all the usernames and passwords that have logged into that computer, then it could take those usernames and passwords and try to log into all its neighbors’ computers to see maybe it can get into those too, and collect more usernames and passwords along the way.
By golly, with a list of usernames and passwords to try, it would be able to successfully get into a lot of computers to infect them, too. But if it couldn’t login like that, it would then try to use EternalBlue to see if that system was unpatched and exploit it that way. The worm would try two very powerful and dangerous ways to get into every computer on the network. Once the virus tried to spread as far as it could, it would then infect it with ransomware, encrypting the whole thing, making it useless, and then rebooting the machine so it’s unusable. This would be an extremely powerful combo that certainly could be a knockout blow. Now, the target of this attack was Ukraine and the goal was to take out as many computers as possible in Ukraine; businesses, government agencies, doesn’t matter. Everything. Take down all of Ukraine’s network. But how can you target an entire country? This is both a wide-scale attack but it’s also limited in size. They didn’t want it to spread through the whole world, just Ukraine. Hm, this is a very interesting question and something I bet the hackers thought a long time about. They ultimately chose to target a small company called Linkos Group.
ANDY: Linkos Group is a pretty small family-run software business based in a building in western Kiev, the capital of Ukraine, in this nondescript building in a kind of dingy neighborhood on the edges of Podil, a kind of hipster neighborhood in Kiev. In the third floor of that building is a server room full of these pizza box-sized servers stacked up. One of them was responsible for sending updates to MeDoc, this accounting software that Linkos Group sold, their flagship product. It’s really like the QuickBooks or TurboTax of Ukraine. Anyone who files taxes in Ukraine or really who wants to do business in Ukraine uses this software, MeDoc.
JACK: Hm, you see where this is going? MeDoc is like TurboTax but for Ukraine. People who need to file their taxes in Ukraine use this software, so if the hackers could infect MeDoc with this worm, a spreading, replicating virus, then the attack would only hit people who have to do taxes in Ukraine.
ANDY: In June of 2017, a group of hackers took over that update server and they hijacked MeDoc’s update mechanism to push out their own malware. Everyone everywhere in the world who had MeDoc installed suddenly has NotPetya, this worm, installed as well.
JACK: We don’t know how, but they got into that MeDoc update server; maybe a phishing e-mail or something. But it didn’t matter. The stage was set and the biggest cyber-attack in history was about to be launched. [MUSIC] On Tuesday, June 27th, 2017, the virus was placed on the MeDoc update server and an update was sent to thousands of computers in Ukraine. Each and every one of those computers were infected by this virus. The seed was planted and was instantly spreading. As soon as someone got the update, they were infected, and immediately the worm spread to another machine, and another machine, and another, grabbing usernames and trying to log into its neighbor, and then using those passwords it would get along the way to spread to as many computers as it could in the network, as well as using EternalBlue to get into computers it didn’t have the password for.
As soon as it was infecting a computer, it was rebooting it and encrypting it, rendering it useless. In a matter of minutes, entire organizations were seeing their networks just go down, like a shadow being cast on all the computers. Now, all this happened on the day before Ukraine’s [00:15:00] constitution day which is the day Ukraine celebrates their independence from Russia. What was typically supposed to be a slow day leading up to a holiday was a day that some people will never forget.
ANDY: Oleksiy Yasinskiy, this forensic analyst and incident responder for a company in Ukraine called Information System Security Partners, described the experience of going to one of their clients early that morning, one of the very first victims of NotPetya, Oschadbank, this former national bank of Ukraine. As he went in, he described entering a building where everyone seemed to be in a kind of state of shock because all of their systems had been shut down simultaneously. [MUSIC] Around ninety percent of all of the computers in Oschadbank had been hit with this mysterious ransomware worm. It looked at first like a normal piece of ransomware which encrypts all of your files. In fact, in this case, encrypts the entire operating system of the computer.
There was a message on the screens of Oschadbank’s PCs demanding $300 in Bitcoin as a ransom before the attackers would unlock the computers. But Oleksiy Yasinskiy says that he pretty quickly, as he was doing incident response for Oschadbank, could tell that this was something unusual, at least in the sense that it was extremely virulent. The worm had essentially rampaged through Oschadbank’s network until it got access to an administrator’s credentials. Then it had used those credentials to jump out to every machine that that administrator had access to, very quickly just saturating the entire network and shutting it down.
JACK: That day the bank could not do business. The people came to work but their terminals were all encrypted and frozen. Customers and employees were both very upset that systems were down.
ANDY: Every one of these computers that had been hit was completely locked and showing this ransomware screen demanding $300 in Bitcoin before the hackers would decrypt it and give Oschadbank’s staff back access to that machine. But Oleksiy Yasinskiy and ISSP, over the next hours, would very quickly come to the conclusion that this was not really ransomware. It was a destructive worm posing as ransomware. Even if you paid that $300 in Bitcoin, you were not going to get your files back. That was just a kind of thin ruse hiding an act of cyber-war.
JACK: As incident responders investigated this, they found that the ransomware was similar to the Petya ransomware. It was originally thought to be Petya but some additional research went into it and found this is a new strain. It was not Petya. Since there was so many people saying that it was not Petya, that’s the name that stuck for this virus. This would become known as the NotPetya attack on Ukraine. After the break, we’ll hear just how destructive NotPetya became. NotPetya was not just hitting this one bank; it was initially infecting networks through the MeDoc software update and then spreading into hundreds of networks, hitting thousands of computers through the whole country of Ukraine.
ANDY: At the same time as Oschadbank was being taken down by NotPetya, it in fact was spreading across the entire country of Ukraine.
JACK: [MUSIC] In just a short time, in a matter of hours, a massive amount of networks and computers were permanently down, infected by NotPetya. One researcher claimed that over three hundred companies were brought down in Ukraine over this attack. Pretty much the whole country was infected by this in some way; either you personally were down, or your supplier was down, or your neighbor was down, or your client was down. It was a catastrophe.
ANDY: But NotPetya didn’t stop spreading at the borders of Ukraine. I mean, no cyber-attack cares about borders, obviously. Really, any multinational company that had MeDoc installed was also instantly infected with NotPetya. That included FedEx, Maersk, the world’s largest [00:20:00] shipping firm, Merck, the New Jersey-based pharmaceutical company, Saint-Gobain, the French construction firm, Reckitt Benckiser, this UK manufacturing firm, Mondelez, the food company that owns Nabisco and Cadbury, and countless others. We just knew that initial list that I just named because they were the ones who were public companies that had to declare their damages to shareholders. But we may never know the full extent of all of the companies that were hit by NotPetya.
JACK: Of course, if these companies either had MeDoc or were connected to networks of companies in Ukraine, or were sharing computers with infected companies, they were also getting infected with NotPetya, too.
ANDY: Counterintuitively, NotPetya also spread into Russia and did really serious damage there to the state oil company Rosneft, to the steel maker EVRAZ, to the medical technology firm In Vitro. Really, everyone who touched Ukraine in any way which of course includes Russia, suffered damages from this.
JACK: Companies all over were scrambling to figure out what happened. How do we fix this? Is there a way to recover or undo this? How do we get stuff working again?
ANDY: All across Ukraine, essentially, people were figuring out that it was better just to shut down your entire network, turn everything off, than watch it be devoured by NotPetya. Really, every government agency; the postal service, all of these companies, they were, in many cases, shutting down their own networks but usually it was too late. NotPetya had often infected the majority of their systems before they could even pull the plug.
JACK: With so many computers down all over the city and country, the feeling must have been surreal.
ANDY: The personal experience of being in the middle of this; I heard it best from this guy Pavlo Bondarenko who was an IT administrator at the Ukrainian Health Ministry. He had, very early in the day, figured out that they needed to pull the ministry’s network offline. That probably spared the Health Ministry from some terrible damage but nonetheless, he spent the whole day fighting off NotPetya and then at the end of the day, he left the office to go home, tried to get on the subway [BEEPING], found that NotPetya had actually destroyed the contactless payment system that he usually used to swipe in to get onto the Kiev metro. [MUSIC] He had to go out to find an ATM where he could get cash to buy a token. All of the ATMs that he tried were also paralyzed by NotPetya, one after another. [BEEPING] Until he found one ATM that was still working but had a very small cash limit and this long line of people trying to get cash. He waited in line, got the cash, bought the token, got onto the subway, went to his neighborhood, got out, and tried to go grocery shopping.
[BEEPING] Found that the payment system at the grocery store was down. He had to get more cash ‘cause he had run out, so he had to find another ATM among all of the paralyzed ATMs where he could take cash out again. Pavlo described that experience as not just being kind of annoying but being disorienting. He had found himself in a world where everything was suddenly broken. [BEEPING] He described it as a natural disaster except that it was entirely man-made and that things had gone very quickly from just seeing what was new on Facebook to asking questions like, did he have enough money to buy food for the next week? People were asking did they have the medicines that they needed? Would they be able to get to work and back? It was a kind of fundamental cyber-attack against the basic infrastructure of people’s lives that we had really never seen before.
JACK: This really scares me. This is a major disaster unlike anything any country has ever seen. For so much of the country’s infrastructure to be down like this? It’s chaos. I am not prepared for something like this to happen where I live; to suddenly and without notice to not be able to get gas, food, or money? To have hospitals turning away people because their network is down? In disasters, there isn’t enough emergency crews to help everyone. You’re on your own or you’re at the whim or someone else willing to help you. I just think of how connected our whole world is now and to see it so fragile like this where one well-crafted, well-timed, well-executed virus can do such an enormous amount of destruction? I’m shaken.
ANDY: I would say that the cyber-war began in Ukraine much earlier. [MUSIC] As soon as Ukraine came under repeated, sustained, disruptive cyber-attacks starting in the fall of 2015, culminating in two blackouts in late 2015 [00:25:00] and then late 2016, that was cyber-war but this was kind of a new stage of the cyber-war, a kind of carpet bombing of the whole country’s digital systems. In terms of what is cyber war, I would say that Richard Clarke got it right in his book in 2009, I think it was, his book Cyber War where he basically defined it as an act by a nation state’s hackers designed to disrupt an adversary’s systems. I think that that’s at least the most basic definition for cyber-war. I think other things that make something a cyber-war are that it affects critical infrastructure, that it is massive in scale, that it takes place in the midst of a physical war. All of those things are true of – in fact, the entire campaign of cyber-attacks carried out against Ukraine but especially NotPetya, this kind of climax of that whole series of attacks.
JACK: Okay, alright. I’m back now. I had to pause there for a second and go build my 72-hour kit because this is freaking me out. I don’t know what to think of this. I guess I’m just lucky this didn’t hit the US.
ANDY: Yeah, I mean, I think a lot of people see what happened to Ukraine and they think phew, that could have been us. That’s scary. But in fact, what I keep trying to emphasize is that NotPetya did hit us, too. It didn’t hit us at the same national scale as Ukraine but it hit American companies. It hit western companies; FedEx, I’m talking about FedEx and Merck in New Jersey, and Maersk’s Terminal also in New Jersey. Somehow New Jersey got a lot of damage here. But this was not a Ukrainian attack. This was an attack that spilled out from Ukraine to the entire world and immediately included us, too.
JACK: Okay, so let’s talk about Maersk. Maersk is not a Ukrainian company; it’s a Danish company. They’ve been the largest shipping company in the world for the last three decades. Picture those huge container ships at sea carrying tons of those big metal container boxes full of goods. They’re headquartered in Copenhagen in Denmark, but they were impacted by this, too.
ANDY: Maersk had one office in Odessa on the Black Sea coast on the south of Ukraine. In that office they had one computer, that I know about at least, that had MeDoc installed. That was all that it took for Maersk’s entire global network to be infected. [MUSIC] At Maersk’s global headquarters in Copenhagen, this beautiful, blue-windowed building on the Copenhagen harbor’s promenade, staff just noticed all of a sudden on the afternoon of June 27th, that screens around the whole building were just turning black. One staffer described seeing a wave of screens turning black all around him; black, black, black. Some staffers started to crowd around the Help Desk in the basement of the building but very soon it was clear that this was much larger than that, that every computer in the building was being infected.
IT administrators were soon running down hallways, unplugging computers, running into meeting rooms to unplug computers in the middle of meetings, jumping over turnstiles because even the turnstiles that control the physical security of the building had been paralyzed by this attack. They were rushing to really turn off all of the systems because they knew that every second meant hundreds or even thousands of more machines that would be compromised. But that was really just the digital part of the attack on Maersk. Maersk runs this massive global shipping machine with these container ships the size of the Empire State Building with another Empire State Building’s worth of cargo on top of them. All around the world, those ships were starting to arrive at Maersk-owned terminals everywhere in the world, and their systems had been shut down so that nobody even knew what was on these gargantuan ships. They couldn’t even figure out how to unload them.
Meanwhile, the real choke point’s at seventeen terminals that were shut down by this. Seventeen ports, essentially, all around the world were the gates outside where the trucks lined up at the Elizabeth New Jersey APM Terminal owned by Maersk. It’s a full square mile-size patch of land in the harbor. These massive ships pull up but so do thousands of trucks every day, and they come to this checkpoint outside the terminal where they’re told over this voiceover IP system where to go, what to pick up or drop off, and all of that on June 27th instantly shut down. [MUSIC] Trucks were arriving at that gate outside the terminal and nobody was talking to them. They were locked out.
They had no idea what was going on. Maersk couldn’t [00:30:00] even send them an e-mail to explain. The trucking companies were entirely in the dark, people were getting furious, the port police started to tell them you need to turn your truck around and leave, but they had stuff that they had to ship somehow for just-in-time manufacturing processes and perishable goods that had to be refrigerated. It was just a fiasco and soon, tens of thousands of trucks were lining up at seventeen of Maersk’s terminals all around the world from Los Angeles to New Jersey.
JACK: Tens of thousands?
ANDY: Tens of thousands of trucks in total, yeah, certainly. Each one of these terminals had lines of trucks that were miles long. From Los Angeles to New Jersey to Algeciras in Spain to the Rotterdam in the Netherlands to Mumbai in India; this was a significant chunk of the entire physical operation of the world’s largest shipping conglomerate just shut down in an instant.
JACK: That’s so frightening.
ANDY: Yeah, I mean, it’s hard to get your head around the scale of this in physical terms. It’s interesting in part because we’ve always been scared, or I’ve always been scared of these attacks that directly interact with physical infrastructure like Stuxnet. Some of the Ukraine attacks were like that too, the ones that turned off the power and utilities, causing the first-ever blackouts caused by hackers. But it turns out that if you just destroy tens and tens of thousands of computers, just the computers around the world, you can maybe do more physical disruption just by taking out all of that digital equipment. The data alone, just paralyzing the brains of a corporation like Maersk can do more physical disruption than directly attacking the physical equipment. I don’t know if that’s an idea you really care about, but it’s…
JACK: Yeah, it puts me in deep thought, this whole thing. Everything is on those shipping things, everywhere from diapers to food to medical supplies.
ANDY: Yeah, yeah. What did their ships contain? It was just absolutely everything that the modern economy runs on, from manufacturing components to food, consumer goods that are part of a just-in-time supply chain. I mean, Maersk is really at the heart of the global economy and its operations just kind of instantaneously winked out of existence.
JACK: Hearing this just reminds me about where we were in 2008. Certain banks were facing financial crisis in the US and they were deemed too big to fail because they were so integrated into our lives. The US government bailed them out, giving them billions of dollars to re-stabilize the nation. I’m starting to think that Maersk is also so interconnected into the US that they might also be too big to fail. Each ship has one million items on it, crucial items that we need in order to live, but as far as I know, the US government or any government did not help Maersk. Yeah, the FBI called them to investigate the case but that’s about it. Maersk could not solve this problem by themselves and the citizens of the US would suffer until Maersk could get back on their feet. Because not just the US; the whole world relies on deliveries from Maersk.
They have shipping yards all over the planet. NotPetya had a clear global impact. Maersk absolutely needed help. Something like 49,000 of their computers were down worldwide which was 100% of the Windows computers they had in their network. 100% of them. The only computers that weren’t encrypted were either Linux or Unix systems, or the ones that were down before this attack or were offline for this attack. Because their network would periodically sync to backups, all their backups and disaster recovery centers were wiped, too. Their e-mails were down, phones were down. You couldn’t even see your contact list on your mobile phone because that relied on exchange being up. Maersk was in trouble. [MUSIC] It wasn’t clear to them at first who was threatening them or what, or why. There was so much chaos everywhere, you just didn’t know who all the victims were yet. But they called up Microsoft right away and spoke to someone very high up there to discuss options.
Microsoft got busy trying to find solutions to this and they heard lots of complaints from other people, too. A few days later, they had some news; Microsoft called back Maersk and told them they cracked a decryption key to decrypt the ransomware but the bad news was is they only cracked the decryption key for one computer. The other problem is that it took them 22,000 compute hours to crack that single key for one computer. Maersk had 49,000 computers so this wouldn’t work. There was no choice; Maersk had lost everything, with no help in sight. They didn’t seem to have any way to recover. Everything was gone, all backups, too. Ransomware was holding it all hostage. Now, I heard from a few places that Maersk got in contact with the hackers who made this ransomware and there was discussion about prices on it and how much it would cost to unlock all of Maersk’s computers.
[00:35:00] This conversation went back and forth between the hackers and Maersk for a little while. The story goes is that the hackers said themselves that they didn’t expect this to spread so far, so quickly. It sounds like even the hacker was impressed by how effective it was. Ultimately, Maersk decided not to pay for a number of reasons. For one, it paints a target on Maersk’s back as someone who pays ransoms but also, security researchers were suggesting that this isn’t a ransomware; it’s a wiper and that even if you had the decryption keys, you’re not gonna get your data back. There was doubt that this could even be recovered this way. But more importantly, Maersk knew they needed to rebuild their network anyway.
Even with decryption keys, they still needed to go through every computer, unlock it, reconfigure it, secure it, check it for any tampering or misconfigurations, and get it back to working again. They opted just to ignore the ransom and start from scratch. But still, this meant a lot of work to do. Where do you even start to recover a network this big? Well, stay with us because after the break, we’ll hear how they got their cargo moving again. Maersk was screwed without a functioning network so the only option they had was to rebuild everything from scratch, their entire network infrastructure. They hired Deloitte, a consulting company, to come and help them do incident response.
ANDY: But they also set up their own emergency recovery center in this building outside of London in this town called Maidenhead. That building just was swarming with everyone who vaguely worked in IT for Maersk anywhere in the world who were all kind of shipped in within days to work 24/7, more or less, to rebuild Maersk’s global network.
JACK: Because everyone’s computers weren’t working and they wanted to get people stood up again quickly, they came up with a few different plans to get everyone back online. They decided to deploy USB sticks to employees with operating systems installed. With this, the IT team could stick a bootable operating system on a USB drive, then hand it to an employee, and they could just boot to the USB drive and have a working computer. Of course, it doesn’t have all their stuff, but at least it’s something. If that computer went down, they could just grab a new USB stick and boot up, and they’re online again. It’s a quick band-aid to get some systems back up.
It’s a good idea, so Maersk tried to buy three thousand USB drives. But this was a problem because even big-box stores like Staples or Best Buy, they only have a couple dozen in stock and they needed thousands. They quickly wiped the USB supply of anyone who was willing to sell it to them, and then they began buying directly from the manufacturer to get them in bulk. How long is that gonna take, right? Days? Weeks? This was slowly getting individual users back online but they still needed to rebuild the entire IT infrastructure, all the servers and stuff.
ANDY: As Maersk started that recovery process, really throwing everything they had into that Maidenhead building where people were trying to rebuild their network from scratch, the very first hurdle that they encountered was that they didn’t have a backup copy of their domain controllers which are a kind of core backbone of their network. [MUSIC] Maersk has more than a hundred domain controllers and each of them is designed to kind of backup to each other. If one goes down, it’s no big deal because it’s backed up to all the other ones. It’s this massive redundancy system but what they hadn’t planned for is a situation where every single domain controller is wiped at the same time. That is exactly what NotPetya did.
JACK: All of their domain controllers were ruined, wiped, destroyed. It was catastrophic. This is the heart of the network, the thing that knows everyone’s profile and logins, and passwords, and permissions, and so, so much more. [00:40:00] It was totally gone. Now, typically, you’re gonna have backups for this and they did have backups and redundancy, but this worm infected their backups and redundant domain controllers too, so they were gone. Maybe in a company this big, you might want to do some sort of weekly snapshot and then take that snapshot to some offsite location so in case something like this does happen, you can at least go back a week and get something from there. But it didn’t seem like they had any of this and they were stuck with pretty much no network.
ANDY: These frantic IT administrators are calling around to every Maersk facility everywhere in the world looking for any backup of the domain controllers. They finally found it in one place; it was in a datacenter in Ghana that had experienced electrical blackouts, just a normal loss of electricity, but the result was that that one domain controller had had its data preserved. It hadn’t been infected by NotPetya ‘cause it wasn’t online.
JACK: One domain controller in Ghana is still working. This could be the domain controller that could help stand up all of Maersk’s network. It became a critical mission to get this domain controller to the disaster recovery center.
ANDY: They had to get that data from Ghana to Maidenhead. They first tried to set up a secure remote connection but the bandwidth of the Ghanaian data center wasn’t fast enough so they tried to fly someone from Ghana to London, but the Ghanaians didn’t have the right VISAs, so they had to do this kind of crazy relay race thing where people flew from London to Nigeria. The Ghanaians flew to Nigeria too, and they handed off the data on some sort of physical medium and then carried it back to London, drove to Maidenhead, and that was the beginning of this weeks and ultimately months-long process of rebuilding Maersk’s network.
JACK: With this one domain controller, they were able to start restoring the network. Phew. Maersk needed even more help, though. They didn’t have a functioning network so they asked partners and clients if they could use their network. But of course, nobody wanted Maersk on their network since Maersk had a horrible virus. Maersk tried hiring more IT people but they couldn’t find anyone qualified or available, so they called up whatever companies that were partners and clients and friends of theirs and asked could they just hire their IT staff? These companies were like, no. But they did loan out a few of the IT staff to Maersk; forty engineers, analysts, and IT experts were loaned to Maersk and flown in to help recover the network. After about nine days of working on it 24/7, they were able to have a functioning network again. This ultimately cost Maersk 350 million dollars. That’s just the story of how Maersk handled this problem. There were over three hundred other organizations that were also hit.
ANDY: It would hit pretty much every Ukrainian government agency. The Minister of Infrastructure, Volodymyr Omelyan, told me that the government was dead and it spread to the postal service. The entire postal service of Ukraine shut down which includes all of their payment systems for sending money, their functions for handing out pensions to people in the country, newspaper delivery.
JACK: But there’s also 74,000 employees at the post office. How are those checks going to be issued when all the computers are down? Ukraine’s Ministry of Health thought they were going to be infected so they just unplugged their entire network, forcing themselves to go down which is unthinkable; to unplug yourself on purpose.
ANDY: Twenty-two banks were shut down by NotPetya, six power companies, two airports, four hospitals in Kiev alone, the card payment systems in the metro in Kiev and other cities, all of the ATMs across the country. This was the kind of, I don’t know what you would call it, a kind of full-spectrum cyber-war that had really never been seen anywhere else before and it hit Ukraine at a national scale.
JACK: This was a national disaster, an epidemic that caused panic and chaos everywhere. Yeah, this is an intentional man-made disaster, an attack that someone wanted to inflict on the country of Ukraine. Yeah, I think this is a cyber-war which is the first time I’ve ever admitted to saying that myself.
ANDY: [MUSIC] About a week after NotPetya hit, vans full of these militarized Ukrainian police pulled up to the Linkos Group headquarters and poured out into the building, up the stairs as if they were raiding the Bin Laden compound, pointing semi-automatic rifles at staff, kicking down a door. [00:45:00] It was all to grab this one server on the third floor of the building that had been, in some ways, the genesis of the NotPetya attack. But of course, what’s very ironic about that is that it was not the genesis of the attack; it was just an instrument of it. The real source of that attack was somewhere far away across the internet, ultimately, almost certainly in Moscow, hundreds of miles from Kiev.
JACK: Ah, yes, now we get into the who would do such a thing part of our story. Andy here thinks it’s Moscow but that’s no easy conclusion to get to. Just because Russia and Ukraine are enemies isn’t enough. You need more evidence than just that. I mean, it might have just been a criminal group of hackers. An investigation began on trying to find out what the evidence was behind who did this. Of course, that Linkos Group server and their network was analyzed to see what the intrusion there looked like. Were there any clues left behind with that? How did they get in? The virus was also analyzed to see if any notes were left on there. Maybe some comments or variable names or documentation might give us a clue. The virus was analyzed over and over and you can also look at compile times. At what time of day was the virus made? Like, 1:00 p.m. in Moscow is 5:00 a.m. in the US. All these things are worth investigation and writing down.
ANDY: [MUSIC] Within days of NotPetya hitting, the Slovakian cyber-security firm ESET had started to pull together forensic evidence that tied NotPetya to the earlier waves of attacks against Ukraine that included the data-destructive attacks against Ukrainian companies and government agencies and the blackout attacks that had hit in late 2015, late 2016. Those attacks, in turn, had been tied to this group Sandworm.
JACK: The security company ESET got ahold of a copy of NotPetya and studied it extensively. They published a report showing all of the evidence that ties this to Sandworm.
ANDY: Sandworm, this little company iSIGHT Partners had found in 2014, had a Russian language how-to manual for using their trojan on an open directory of their command and control server. If you follow that forensic line all the way back to 2014, it’s pretty clear, first of all, that who else is gonna be attacking Ukraine for years on end other than the country that has also launched a physical invasion into the east of the country and seized Crimea? That’s just common sense but also, we know that this group was Russian-speaking because of that file found on the open directory. Within days of NotPetya, it was pretty clear to me that this was part of the larger Russian cyber-war against Ukraine; that this was not a criminal act, that it was in fact the climax of a nation state-sponsored, escalating series of cyber-attacks against a military target.
For almost nine months I was kind of going crazy trying to understand why none of these victims were naming Russia; no government had actually named Russia, NATO had not said anything. It was weird enough that we had watched this Russian cyber-war unfold in Ukraine for years but now it had even hit these multinational companies, many of which were based in the west, and still nobody was calling out Russia for this worst-ever-in-history cyber-attack. Until finally, nine months after NotPetya hit, the White House put out a statement, a very, very short statement that just said yes, NotPetya was the worst cyber-attack in history and it was deployed by the Russian military against Ukraine and that there will be consequences.
That statement was in turn backed up with similar statements from all the four other Five Eyes, English-speaking nations’ intelligence agencies. The US, Canada, New Zealand, Australia, and the UK all simultaneously called out Russia as the perpetrator of NotPetya. There are still people, and in particular Russians, who question whether NotPetya was really a Russian state act but I don’t think we’ve ever had all five Five Eyes agree publically to call out someone like this before. I don’t think there’s really much room for doubt.
JACK: The FBI also did their own investigation working with some of these international companies and Ukrainian companies to learn more. But still today, we have no idea what the FBI found in their investigation but for Andy, he wanted to learn more about what happened there, so he packed his bags and flew to Ukraine to investigate.
ANDY: [MUSIC] When I was in Ukraine, I talked to the SBU, the Ukrainian equivalent of the NSA, and they had told me flat-out that Sandworm was Fancy Bear, APT28, this other Russian hacker group that had been named for years as linked to the GRU, Russia’s military intelligence agency. [00:50:00] I had suspected for a long time, and I’ve heard this from American sources too, but it was kind of unsubstantiated that Sandworm was likely the GRU and they were the most likely candidate because they’re part of Russia’s military, Russia’s military was invading Ukraine, the GRU had been very active in that invasion. But when the Five Eyes said that the Russian military had carried out NotPetya, that for me was ultimately the confirmation. I should give some credit here also to the Washington Post who, in a story before that announcement, said simply that NotPetya was carried out by the GRU.
JACK: The GRU is Russia’s military intelligence agency. Within the GRU are hackers. In fact, the FBI has indicted twelve GRU hackers from meddling with the 2016 US election for hacking into the DNC. Robert Mueller is who brought this indictment forward and I read through it; it’s twenty-six pages and it explains a lot of details about the GRU and how they hacked the 2016 election. It even lists the street address of where these hackers work out of. It’s a fascinating read but so far nobody has been indicted for NotPetya and there’s been no FBI report for that, either. The GRU hackers behind the 2016 election hacking, that hacking group has been called Fancy Bear but this group that did NotPetya, something was a little different here.
It didn’t have the same MO as Fancy Bear so a different name was given to them; Sandworm. It might be the same group as Fancy Bear. We don’t know. My guess is that it’s another hacker team just down the hall from Fancy Bear, or on another floor working in the same building as Fancy Bear. But what we believe is that both Sandworm and Fancy Bear are hacking groups both working for Russia’s GRU in Moscow. With the address in hand from the earlier indictment, Andy decided to take a trip to Moscow to learn more. He went right up to the tower that GRU works out of and looked at it.
ANDY: When I went to Moscow and stood there in the shadow of the tower, this glass building on the Moscow canal in northern Moscow that maybe I believed housed Sandworm, the hackers responsible for all of this destruction, I had a feeling of futility; that I was so close physically to the perpetrators of these attacks and yet I wasn’t gonna get any closer. Just as distance had not been a kind of defense against NotPetya, proximity wasn’t really enough to bring me any closer to these attackers. They were behind a locked gate with armed security guards. I knew that I couldn’t just ask for an interview. As close as I was to these hackers, that was kind of the end of the story for me and I don’t know if I will ever get any closer.
JACK: [MUSIC] The estimated damages from this attack totaled ten billion dollars. This is why this is the largest cyber-attack in history. No attack has come close to this amount of damage ever. Ten billion dollars; this was catastrophic, enormous. It set new records and was very scary. It’s scary that all this was done with hacking tools that anyone had access to. There was no super-secret hacking tool used here. Mimikatz is open-source for anyone to use and EternalBlue was dumped by the Shadow Brokers just six months before. You could slap any good ransomware on top of it and there you go. But wait a minute, this makes me think if Russia were the ones behind Shadow Brokers and Russia’s the one that did NotPetya, then why wouldn’t they just keep EternalBlue to themselves? I wondered this and asked Jake Williams from the last episode. Why would they give away EternalBlue and then use it to hack Ukraine, right? You would keep that.
JAKE: Oh, see, I disagree with that. I’ve thought a lot about this as well. You know, if you look at the NotPetya attack, I’m not sure that when – a couple things; first off, I’m positive that they got better return on investment if it wasn’t information operation releasing it and then using it than they would have just using it as an 0-day. I think as an 0-day it would have caused absolute panic and honestly the damage from it would have been so much more outside of Ukraine. I personally don’t believe that the Russians anticipated the level of damage outside of Ukraine that actually occurred. Honestly, I don’t think the InfoSec community did, either. I think that the why did they use it down the road was out there. Why give it up in the first place? I think a couple things; first off, I have no doubt that they have a similar capability or we said at the time, had a similar capability remotely exploited with SMB [00:55:00] vulnerability. I think that’s one.
JACK: Oh, that’s an interesting – I got your theory right away on that, ‘cause if they publically post it, then they don’t have to expose their zero-day but they can expose NSA’s zero-day.
JAKE: Exactly, exactly. Separately from that, they take out – suppose that in April when they go to release this, they don’t know that they’re gonna do NotPetya, right? I think that’s actually, I have to tell you, I think that that’s a reasonable assertion at that point. I think they know they’re gonna do something. I don’t think anybody’s got – I know, at that point, I think it’s clear they know they’re doing a destructive cyber-attack around MeDoc in Ukraine but I don’t think it’s clear they’re gonna worm anything. I don’t think that was ever part of the decision calculus for release. But taking NotPetya completely out of it for a minute; if you are a nation state operation, so roll back to the blog post that I was pushing where I was like hey, it is likely – basically, whoever this is, is operating in the interest of Russia where they are effectively shutting down or – I say shutting down; they’re effectively taking control of the InfoSec/technology news cycle with these releases.
JACK: Hm, besides that, it throws NSA into chaos, right? As soon as Shadow Brokers dumps their stuff, there has to be a mad scramble at NSA to try to look around at what got dumped and who did it and why and what. At the same time, it makes NSA look bad which gives the GRU some top cover to move into position and stage a massive attack while the world was dealing with EternalBlue. [MUSIC] Gosh, what a future we have set for ourselves, because I don’t think the world has learned from this lesson. There are still hundreds of thousands of Windows computers still vulnerable to EternalBlue out there right now. You can just update this any moment and protect yourself. But Microsoft, Microsoft still hasn’t patched Mimikatz. I mean, they have, okay? They’ve fixed it but more people just find more flaws in the authentication of Windows and Mimikatz works again. From what I’ve been told, this will never be fixed. Not that Microsoft isn’t working hard on it; they are.
They release fixes all the time. They’ve created this tool called the Microsoft Windows Credential Guard which protects against this. But if that’s the case, then why isn’t that enabled by default? Or why can’t the defaults just be secure and then a system admin is the one who has to click the button to make it insecure? Insecure by default is never a solution. The reason why Mimikatz isn’t just fixed once and for all is because there’s something inherently flawed with the way Windows authentication works just as a whole. It’s like every door or window in your house; these are the weak points by design because they’re literally holes in your house that things can go in and out of. Mimikatz just makes me really mad because it’s still a problem and it was used in this attack that brought down Ukraine and cost the world ten billion dollars.
I mean, is there a scenario that’s so devastating to the world that somebody finally does something about Windows authentication to make it secure? I don’t know, and this is what really makes me mad. Aah! I will not fear. Fear is the mind-killer. I will let this pass over me. Okay, so while this is the story of NotPetya, it’s just a small part of the story. Andy Greenberg, our guest in this episode, just published this book called Sandworm which goes into great detail about it. I mean, the guy flew to Ukraine and Moscow to get to the bottom of all this. This is not the only cyber-attack Russia has done to Ukraine; the book outlines so many more attacks that are equally as serious and scary you should be aware of. In fact, I want to say that this episode only covered like, a fifth of the book, so go get Sandworm in any bookstore right now, or get the audiobook and dive in and enjoy because it’s fantastic.
JACK (OUTRO): [OUTRO MUSIC] A big thank you to Andy Greenberg. Your book is amazing, the story is amazing, and I appreciate all the research you’ve done and coming on the show to tell us this story. To learn more about Andy, visit andygreenberg.net or find him on Twitter as @a_greenberg. I’ll also have affiliate links to the Sandworm book in the show notes. Thanks to Jake Williams once again. This show is made by me, harkonen, Jack Rhysider. Sound design was done by the dual-eared Andrew Meriwether, editing help this episode by the clip-happy Damienne, and our theme music is by the bouncing Breakmaster Cylinder. Even though people turn off their phone, yank the battery out, and go sit in that corner of their house that gets no WiFi every time I say it, this is Darknet Diaries.
[OUTRO MUSIC ENDS]
[END OF RECORDING]