Transcription performed by LeahTranscribes[START OF RECORDING]
JACK: Whenever we have a computer problem that we need to troubleshoot, we often want to know why that was a problem. How did it break? You know what? Sometimes you never get a good answer. One time when I was at work, a router suddenly crashed. The internet was down for that office and my teammate jumped on the problem to try to figure out what was going on. A few minutes later, the router was back up and online and was working fine all on its own. This router crashed and rebooted, but why? My teammate wanted to know, so he began a forensic analysis. [MUSIC] He looked at the environmental data before the crash. It was not showing high CPU or out of memory. It did not have a heavy amount of traffic going over it either, so this wasn’t an over-utilization issue. Next, he grabbed core dumps, memory snapshots of what was present at the time of the crash, and he sent that to the manufacturer of the router to see if they could figure it out. A few days later, the manufacturer told us they analyzed the core dumps and said the reason for the crash was spurious emissions from space. Spurious emissions from space. That’s what caused this router to crash. What the heck is that? Are they saying an asteroid hit this thing? We looked into this further and apparently there are cosmic rays that are constantly bombarding Earth, and sometimes they can come down, pass right through the roof, right on through the outer chassis of the router, and go right through the circuit board of the router which can cause a slight electromagnetic change in the circuitry, just enough to make a bit flip from a zero to a one or a one to a zero. If the wrong bit flips, it could cause the device to malfunction and crash. Cosmic rays can cause this, which is incredible that that’s even possible. But really, I thought this manufacturer was just using this as some kind of excuse, because they can’t prove that cosmic rays did this. So, in my opinion, it meant that we’ll never know what caused this router to crash. It’ll always be a mystery, and I wonder how many mysterious things happen to computers that are caused by cosmic rays.
(INTRO): [INTRO MUSIC] These are true stories from the dark side of the internet. I’m Jack Rhysider. This is Darknet Diaries. [INTRO MUSIC ENDS]
JACK: Nicole Beckwith started out with a strong interest in computers and IT. She studied and learned how to be a programmer, among other things. But somehow, at some point of her career, she decided she wanted to be a cop.
NICOLE: I am a former state police officer and federally sworn US marshal. I worked as a financial firm investigator and a digital forensic examiner for the state of Ohio.
JACK: Now, while she was serving as a police officer, she would see cases where hacking or digital harassment was involved. These were cases that interested her the most.
NICOLE: My background is in computers and computer programming. So, because of my background, I started taking all those cases.
JACK: Now, because the internet connects us all together, she’d often be investigating a case and find out that the suspect is in another state, so this would often mean that the case would turn into a federal investigation, where it landed in the hands of the FBI or Department of Homeland Security, or even the Secret Service. So, these cases that started out at her police department would sometimes get handed over to one of these other federal units.
NICOLE: So, the Secret Service kept seeing my name in all these reports. One day I got a call, sitting at my desk, from the Secret Service which I can tell you even as an officer is kind of daunting, right? To get a phone call and the agent on the other line’s like, hi from the Secret Service. You’re like oh gosh, what did I do, you know? They were like yeah, we keep seeing your name pop up on these cases and we’d really like to talk to you. So, I went in. It actually was just across the street from my office at the state. I went and met with them and told them my background and explained that I love computers and it’s a hobby of mine, and I like to work on all kinds of projects. So, they said that’s awesome. How would you like to work for us as a task force officer? We will send you to training, we’ll pay for everything; we just want you to help with any of the cases that we get. So, of course I jumped at the opportunity and they swore me in as a task force officer for their Financial and Electronic Crimes Division.
JACK: [MUSIC] [00:05:00] A task force officer for the Secret Service? That sounds pretty badass.
NICOLE: Thank you. Yeah, it was a lot of fun.
JACK: So, Secret Service; that’s who protects the president, right?
NICOLE: Correct, yeah. Yeah, so, most people don’t know in addition to their everyday duties in protecting the president and foreign dignitaries and other public servants and politicians, they actually are staffed with – or assigned to investigate financial and electronic crimes, including cyber-crime.
JACK: That’s where they wanted her to focus; investigating cyber-crime cases for the Secret Service. But before she could start investigating cases, they had to give her some training and teach her how to do digital forensics like the Secret Service knows how.
NICOLE: [MUSIC] I got, oh gosh, a whole host of different training. I started out with the basics, so you go through basic digital forensics, dead-box forensics, and then they work up to network investigations and then network intrusions and virtual currency investigations. So, all-in-all, I think I did seven different trainings, roughly eighteen months worth off and on, going back and forth from home to Hoover, Alabama, and then was able to investigate all these cases. It was like drinking from a fire hose. These training courses are – could vary from one week to five weeks in length. It was very intensive sunup to sundown. You’re doing extra work at night in your hotel room, and you still have to keep learning when you go back. Obviously, that’s not enough as we all know in this field, so you have to keep learning.
JACK: She worked a lot with the Secret Service investigating different cyber-crimes. Her training took her to another level, but then the experience of doing digital forensics gave her more insight and wisdom. Then one day, about seven years into doing digital forensics work, she saw some news that a police station in her jurisdiction was hit with ransomware. [MUSIC] Like, all the computers in the police department were no longer functioning.
NICOLE: It was ransomware across the entire network. It took down the patrol vehicles, it took down the entire police department, and I’m told also some of the city laptops because they ended up being connected in a few different places. It didn’t take the entire city down, but at least the entire police department.
JACK: She called them up as a courtesy to see if they needed any help.
NICOLE: So, for this story I’m gonna tell, I was in my role as a task force officer for the Secret Service. As such, like I said, I was called out to respond to cyber incidents. I do want to do a quick disclaimer of what I discuss in this episode is either publicly available information or I received prior approval to discuss this, so, I do want to get that out there. As a little bit of backstory and to set the stage a bit, this is a small-sized city, so approximately 28,000 residents, ten square miles. Not a huge city, but big enough that you – a ransomware incident would take them down. So, I’m already aware of this agency because it’s in my jurisdiction, so we had reached out when they were hit to offer any assistance. We were told that they had it handled.
JACK: [MUSIC] The IT team at this police department was doing daily backups of all their systems in the network, so they never even considered paying the ransom. They knew they could just restore from backup and everything would be fine again, because that’s a great way to mitigate the threat of ransomware.
NICOLE: As a lot of us know, you always have to make sure that your backups are good, and they did not test their backups prior to deploying them, so they simply restored the system from backup, checked the box, and said we’re good.
JACK: Something happened months earlier which meant their backups weren’t actually working. The latest backup they had was from ten months ago. That’s a really frustrating thing to realize, but by the time they had figured that out, they had already restored a bunch of their systems already, and the network was back up and online. So, they just went with it like that. It wasn’t the best restore, but it allowed people to get up and working fairly quickly. They just had to re-enter in all that stuff from the last ten months back into the systems again.
NICOLE: So, during the conversation when I’m asking if they need assistance, they’re explaining to me that IT has it. When I’m probing them for a little bit more details like hey, do you know what happened? [00:10:00] Did somebody click on a phishing e-mail? Do you understand the attack vector on this? They’re saying no; all we know is that morning our printers went down and then the next thing we know, all of our computers were down. So, that was pretty much all that they could tell me.
JACK: [MUSIC] So, time passes. How much time passes?
NICOLE: A week.
JACK: Okay. So…
NICOLE: Right?
JACK: Yeah, okay. So, a week later, what happens?
NICOLE: So, a week later, I’m actually – I just happened to be on the phone with the lieutenant on an unrelated matter. He paused and he said oh, crap, our printers are down again. [MUSIC] I said wait, isn’t that what happened the first time you guys were hit? It happened to be the same exact day, so Friday to Friday. He said yeah, actually, this is exactly what happened that morning. I’m like okay, stop everything. Don’t touch a thing. Let’s triage this. Let’s grab some evidence if we can. Can I please come help you? So, he’s like yes, please. We would love the assistance. When can you be here? I always have a go-bag in my car. I did happen to be at my office that morning but I always have a go-bag in my car, so I know that any given time if I need to jump in my car and respond, if at home or wherever, that I have all of my essentials in my car.
JACK: Well, hang on, now; when I hear go-bag, I think seventy-two hours of food and water and some Band-Aids. What’s in your go-bag, though?
NICOLE: [MUSIC] Yeah, so, in my go-bag I have a whole bunch of other – of things, including food and clothes and all of that that you just mentioned, but I have what we call a toaster. So, a toaster is a hard drive or a SATA dock that you can plug a hard drive into and do imaging or whatever. I have hoards of USB drives and CDs with all sorts of mobile triage and analysis software such as Paladin, Volatility, password cracking, mobile apps. I have several hard drives for evidence collection, both SATA and external. Then I always had a box of cables and adapters, tools just in case I needed to take the computer apart, so, you know, screwdrivers and stuff. A mouse and a keyboard obviously, because you never know what kind of system you’re gonna encounter.
Sometimes, like you mentioned, most folks forget that you might be at an incident for quite some time, so I always had non-perishable food items ready. I always had bottles of water and granola bars or energy bars, change of clothes, bath wipes, deodorant, other hygiene items, all of those things, of course. Then on top of that, for forensics, I would also include my WiebeTech Ditto machine for imaging. I also had two triage laptops, so, both a Mac and a PC. Then of course gloves after a really bad scare once where I thought I had gotten into something nasty on a computer. I learned to wear gloves no matter what type of case I was working.
JACK: Dang, that’s a pretty awesome-sounding go-bag, packed full of tools and items to help go onsite and quickly get to work. So, she grabs this thing and jumps in her car, and starts driving to the police department. But on the way, she starts making tons of phone calls.
NICOLE: Oh, yeah. So, the drive over, I’m immediately on the phone getting permission from all sorts of people to even be at this police department. So, I’m making sure the police department is okay with it, getting permission from the police chief, from the city manager, the mayor, my director and my chief at the state, as well as the resident agent in charge or my boss at the Secret Service, because there is a lot of red tape that you have to work through in order to even lay hands on a system to start an investigation. So, you have to have all those bases covered, so, I’m making a lot of phone calls. I’m also working to make sure that there is a systems administrator there to give me access to the servers, log-in details, making sure I have access to the room to even get to the server. It’s a police department, so, a badge to get in and out of rooms, or at least an escort to allow me to get in and out of places that I need to get to. I’m also calling a secondary agent and backup for me. I don’t ever want to be the only person there. You always want to have a second person with you for a number of reasons, but…
JACK: It’s funny though because you’re calling for backup to go to the police department. [00:15:00] Like, there’s enough officers ready to back you up, aren’t there?
NICOLE: Right, yeah. Not necessarily backup for physical security, although in this case maybe I wasn’t worried about it, but in other cases maybe I am, right? Maybe I’m responding to some place where the hostile actor is actually an internal person, and you don’t ever want to be with your back against a door or somewhere where you can be ambushed. Even in incident response you have to worry about your physical security. In this case, backup just for the forensics, but in some cases I am asking for backup for physical security as well.
JACK: She also keeps questioning herself; is all this even worth the fuss? So far the only problem reported were that printers were not working. You don’t deploy the Secret Service to go onsite just to fix printers. Maybe she’s just way overthinking this whole thing and she’ll get there and it’s just a false alarm. But it didn’t matter; she’s already invested and wants to check on it just in case. Okay, so, this is how I picture it; you’re arriving in your car, you’ve got your go-bag in your hand, you’ve got the curly earpiece that all the Secret Service agents use, your aviator sunglasses, and you’re just busting in the front door.
NICOLE: Exactly. Picture Lara Croft with cyber stuff, yeah. No.
JACK: Is it really, though?
NICOLE: Yeah, no, probably not. Yeah, I like to think that, but I’m sure that’s not how I actually looked. So, yeah, no, I’m arriving, I’m grabbing all this stuff out of my – the trunk of my car, meeting the lieutenant and the chief and kinda doing a data dump on hey, what’s happened since I talked to you last, letting all my other bosses know I have arrived on-scene and I’m going to start.
JACK: She knows she needs access to the computers in the building, and the best way to get into the computers is to have someone from IT help you with that. Well, since this was a small agency, the IT team was just one person. One guy was running all the computers in this place.
NICOLE: So, I’m on the phone with him when I first get there. I’m also trying to figure out where is the server actually located, which in this case was way back in the back of the building. When you walk in, it looks kinda like a garage or a storage place, I guess; dark, bicycles and boxes, and just everything that they didn’t want in the police department back in this room, cables, and just all sorts of things all over the place. The server’s kinda sitting not in the middle of the room but kinda away from the wall, so just picture wires and stuff all over the place. It’s a little bit messy, so a little bit concerned there.
JACK: She finds the server but then starts asking more questions. Is there anyone else who manages these computers? Yes, they outsource some of the computer management to another company. They had another company do updates to the computers and do security monitoring. But they were more reactive, not very proactive at handling security incidents. So, yeah, so you go into the back, you’re on the phone with the local IT admin, you’re trying to figure out what’s going on. What system do you try to get into first?
NICOLE: So, they had their main server which had multiple BMs on it. But I’m just getting into the main production server, what I thought was just a server for the police department. Turns out, it actually housed a couple other applications for the city, but at least everything for the police department. When I’m initially responding, I’m looking at the server, getting the log-in information from the lieutenant. You successfully log-in. Now, you – in this case, normally when you’re responding to a case like this, you’re trying as hard as possible not to leave a digital footprint. You’re being really careful about what you touch ‘cause you don’t want to alter the data.
This case was a little different because of the ransomware in the past and knowing that as soon as they lost their printers, it was within an hour that the ransomware was deployed. So, I didn’t know how much time I had before what I assumed was going to be ransomware was likely deployed again. So, I was trying to hurry and capture whatever I could for forensics right away, before something went down. I log into the server. I immediately start dumping the memory, so Volatility is one of my hands-down favorite tools to use. [00:20:00] I’m doing dumps of data on Volatility. I’m pulling reports, dumping that to a USB drive. I also – once that is running, I wanted to grab network traffic and so, I started Wireshark up and I’m dumping network traffic to a USB also.
JACK: Okay, so, Volatility and Wireshark; let’s jump into these tools for a second, because I think they’re really cool. [MUSIC] Volatility is an open-source free tool which is used in digital forensics. So, Step One is she’s gotta get into that domain controller which is like the central brain of the network, and take a snapshot of the memory which is what’s in RAM, because whatever data is in memory is what’s being ran right now, and it changes moment to moment. Now, this can take a while to complete. You’ve got to sit there waiting for all the memory to be copied over to the USB drive, but it’s more than just whatever memory is active in RAM. It’s also going to show what processes are running, what apps are open, the names of all the files on the systems, the registry, network connections, users logged in, and system logs.
Once she has this raw dump of everything on her USB drive, she’ll switch the USB drive over to her computer to begin analyzing everything. Are there any suspicious programs running? What connections are active, and what activity are the users doing right now? But depending on how big these snapshots are, each of these questions can take a while to get answers to. So, it’s a slow process to do all this. In the meantime, she fires up Wireshark which is a packet-capture tool. Any traffic coming in and out of this domain server is captured to be analyzed later. Basically, by capturing all traffic to and from this computer, she’ll be able to capture any malware that’s been sent to it, or malicious commands, or suspicious activity. As you can imagine though, capturing all network traffic is a lot of stuff to process.
You’re basically looking at a beach full of sand and trying to figure out that one grain of sand that shouldn’t be there. It’s hard to narrow down all the packets to find just what you need. It takes a long time, but it’s better to capture it now, because nothing else will, and it’s good to have something to go back to and look at just in case. Now, what really was fortunate for her was that she got there early enough and set up quickly enough that no ransomware had been activated yet. But she had all her listeners open and ready in case something did happen. While all that’s going on, she’s poking around in the server, looking for anything out of the ordinary, and she finds something.
NICOLE: After I run all of the quick stuff with Volatility, I’m analyzing that really quickly to see what accounts are active, who’s logged in, are there any accounts that are rogue? I immediately see another active logged-in account.
JACK: [MUSIC] Another system admin was logged into this server at the same time she was. She asked the IT guy, are you also logged into this server? He said no. She asks, do you think that company that manages the network is logged into this server? He checks with them and says nope, nobody is logged into our servers right now, either. She gets up and starts asking around the station.
NICOLE: So, I’m asking the police chief, I’m asking the police lieutenant, who else has access to this? They’re like, nobody should be logged in except for you. There’s only one access. So, my heart sinks at that point.
JACK: There wasn’t just one other active user, either; there were a few other people logged into this domain controller as admin right now. She’s baffled as to why, and starts to think maybe she’s just got there fast enough to actually catch this hacker mid-hack.
NICOLE: Yeah, so, for somebody that has complete admin access as a couple of these folks did, they potentially have access to everything that’s on this server. So, because this is a police department, you have case files and reports, you have access to public information or – and PII. So, social security numbers and birthdates, and drivers license, and sensitive information about cases as well as a whole host of other things that a police department has overseen, right? So, you’re looking at officers and officer security and their names and information, and e-mail addresses. There’s a whole lot of things that they have access to when you’re an admin on a police department server.
JACK: At this point, she knows for sure whoever is logged into this server should not be there. It’s crazy because even as a seasoned incident responder like Nicole, it can still affect you emotionally.
NICOLE: Because your heart sinks when you see that. You kinda get that adrenaline pumping and you [00:25:00] see that this isn’t a false positive, ‘cause going over there I’m wondering, right, like, okay, so their printers went down; is this another ransomware, potential ransomware incident? So, that was the moment when your heart starts beating a little bit faster and you know that there actually is something to this.
JACK: It’s clear to her that she needs to kick the admins out immediately, but another thought comes into her head.
NICOLE: So, right now, as I’m seeing the log-ins, I have to weigh in my head, do we leave them logged in and potentially allow them to do additional harm or do I immediately revoke them?
JACK: Because her tools are still trying to finish their snapshots. [MUSIC] If she kicked out the hacker, that might cause her tools to miss the information she needs to prove what’s going on. As a digital forensics investigator, it’s not often you’re in this situation. Usually you’re called in months after the fact to figure out what happened. Trying to both figure out what happened and fight off an active intruder is just on another level. She checks the status of her Volatility tool, and it’s almost done collecting what she needs. So, she just waits for it to finish, but the wait is killing her.
NICOLE: Right, yeah, so, of course I’m just letting Wireshark run, but then Volatility – yeah, there’s a whole host of scripts and data points that I want dumped. As soon as that finishes, then I’m immediately like alright, you’re done; out. Click, revoking access. So, as soon as you kick that person out of the system, you breathe a very faint sigh of relief, right, ‘cause you still don’t – you have a lot of unknowns, but at least you know that one big threat is eliminated for the moment. Because of the fact that we weren’t sure what the intrusion vector was at that point, like how they initially got in, I’m also changing the password of the supposed admin, the person who’s supposed to have access. So, I’m changing his password as well because I don’t know if that’s how they initially got in. So, I’m resetting that. Then I’m gonna go back in and grab all the other stuff that I need to grab, doing images and whatnot.
JACK: She swivels around in her chair, moving the USB stick from the domain controller to her laptop to start analyzing it, then swivels back to the domain controller to look for more stuff. She’s collecting data and analyzing it, but she knows she needs more data. That’s when she calls up the company that’s supposed to be monitoring the security for this network.
NICOLE: I wanted to make contact at that point. Now that I had what I needed, I didn’t want the IT contractor to immediately start restoring from backup or doing something that would just ruin my evidence. So, now I’m on the phone with them and I’m wanting to make sure that they had backups, that they’re currently running a backup just in case, asking them what data they had, like could they give me logs? Could they see the initial access point? Basically asking me to – asking them to send me anything that they could in the logs that could potentially help me with this case.
JACK: How did they respond to you? Were they friendly and nice?
NICOLE: No, they were a little upset that I was there and had not called them. They were upset with the police department. But then we had to explain like, look, we got permission from the mayor. We got permission from the police department, so they wanted us to come in. This is a law enforcement investigation at this point. So, I need your cooperation.
JACK: [MUSIC] They were upset because they were supposed to be the first contact if something happened. They were just learning now that all this happened, that the printers went down, that there were unauthorized admins accessing the network, and that the Secret Service is there onsite doing an investigation. I can see why they’re upset but professionally, there’s no time for that. If your job is to help your client be safe, oh well if you want the first to be called. Your help is needed now, so let’s get to work now. She kindly asked them, please send me the logs you’ve captured.
NICOLE: In addition to logs, I had asked them if – from the prior incident – they had saved a variant or a file of malware, if they were able to find a ransom letter, if what they had, that they could potentially hand over to me in addition to that so that we could kinda see what strain of malware it was, if we could do soft attribution on it based on that, if there were any other details that we could glean from prior evidence.
JACK: But they’re still upset on how this [00:30:00] incident is being handled.
NICOLE: Right, yeah, so, they didn’t want to hand over the logs and the data.
JACK: This is kind of infuriating to me. The police department is paying this company to monitor their network for security incidents and they didn’t want to cooperate with the Secret Service on this because they felt the incident wasn’t being handled the way they wanted it to be handled? I guess maybe they felt threatened or pressured, or maybe embarrassed that they didn’t catch this themselves or solve it themselves. By this point, they had internal investigators working on this, and I imagine they felt like their work was being undermined. But from my point of view, they completely failed the police department on that first incident. That was their chance to shine, and they missed it. I guess they didn’t want to fail again though, and wanted to show how they can fix it fast this time, and Nicole was just screwing up their plans. But she kept asking them to send her data on the previous incident.
NICOLE: They did end up saying that they had saved a file that was a paint.exe file for the original malware and had saved a text file for the ransomware that was the ransom note.
JACK: Well, that’s something for her at least to look at. Every little bit helps to build a complete picture of what happened and what could happen in this incident.
NICOLE: As I’m analyzing all of the data that I collected and the evidence, I ended up seeing that there was an external IP address that had been logged in at that time.
JACK: What she realized was this police station’s domain controller was accessible from the internet over Remote Desktop. The brains of the network was accessible from anywhere in the world without a VPN. You just needed the username and password to get into this thing or if you had an exploit for this version of Windows. But this, this is a bad design. This system should not be accessible from the internet. Ideally, you should be onsite at the police department to get into this system. But if you really need someone to get into this remotely, you should probably set up a VPN for admins to connect to first and then get into this. Having a system running Remote Desktop right on the internet just attracts a ton of people to try to abuse the system. So, she’s seeing all these external public IPs that just keep logging into this system, and she’s kicking them out one by one, but she’s realizing this has to stop.
NICOLE: So, with this, I politely asked them, I need you to turn off all external access, like who – how are these people getting in? Take down remote access from this server. There’s no reason for it. They refused to do it. [MUSIC] So, I made the request; they just basically said sure, whatever. I think it was a day later that I checked and it still was not taken care of. So, at that point I went right to their office, showed up to the office, knocked on the door, asked for the person that I was working with, and stood in front of his desk and just told him, you’re gonna lock this down right now. It wasn’t nice and I don’t have to do that very often, but I stood in front of his computer until he locked it down.
JACK: Whoa, it’s crazy to think that this IT company had to have the Secret Service explain the dangers of why this is a problem. Nicole is right; this should not be allowed. But I’ve personally tried to convince people to turn this off before myself, and what I’ve been told is it’s required because certain tools and systems need it to be open for things to work, and you’ll break things if you turn it off. Something about legacy equipment, too. Yeah, well, that might have been true even in this case. Certain vendors or apps might have no longer worked if you turned that off. I just think vendors that require this are dumb because the consequences of having your domain controller hacked is far greater than your app going down. In this case, the police department was hit with ransomware because this system was accessible from the internet which caused ten months of lost work. It would have been hit again if it wasn’t for Nicole’s quick reactions. So, she was happy that they finally turned off public access to this computer, and left.
NICOLE: So, after this conversation with the security contractor, I go back and do an analysis.
JACK: [MUSIC] She tries to figure out more about who was logged in as an admin at the same time as her. Looking through the logs and data she collected, she looks at the IP address of the user, which is sort of a digital address. Obviously they connected from a public IP, and she had that, but then from there she did a geo-IP lookup to see where this IP address may be located physically in the world. When she looked at that, the IP was in the exact same town as where this police department was. [00:35:00] That’s interesting. A local person did this?
NICOLE: So, I write a search warrant to that ISP asking for who this IP address comes back to.
JACK: So, what law enforcement can do is issue a search warrant to the ISP to figure out what user was assigned that public IP at the time. But this takes a while; a few days, maybe weeks. In that time, she starts thinking about why someone locally in this town might want to hack into the police department’s computers.
NICOLE: For me, I’m thinking that it’s somebody local that has a beef with the police department. Maybe a suspect or there’s a case or they got pulled over. A whole host of things are running through my head at this point.
JACK: Stay with us because after the break, things don’t go as planned. Okay, so at this point, she’s analyzed the system pretty well and found that this user did upload some malware and looks like they were staging it to infect the network with ransomware again, which means this was an actual and serious attack that she was able to intercept and neutralize before it had a chance to detonate. On top of that, she’s traced this hacker to come from a person who’s local to the city where this police department was, and issued a search warrant with the ISP to figure out exactly who was assigned that IP. She gets the documents back from the ISP and opens it to see.
NICOLE: [MUSIC] So, when I see the address and the person that is connected to this search warrant, I’m a little bit baffled. I’m shocked, I’m concerned, not really fully understanding what I’m looking at. Confusion comes into play there. A roller coaster of emotions are going through my head when I’m seeing who it’s tied back to.
JACK: Why?
NICOLE: Because it came back to the mayor of the city.
JACK: Whoa. The mayor of the city is who hacked into the computer and planted malware on it and was about to detonate it to take the police department’s network down again?
NICOLE: Correct.
JACK: What?
NICOLE: So, at this point, I’m running scenarios in my head as to why in the world a mayor would be connected to this server. Doing reconnaissance on this case and looking at some of the past cases and just knowing the city and wondering who could potentially have an issue with the police department, I did run across some information that suggested that the mayor of the city may have taken an issue with the police department because he was actually previously, prior to becoming mayor, arrested by this police department. So, having that in the back of my head, of course you’re wondering why is this person logged in and then, he does have motive to be upset with the police department. You’re running through a lot of things.
You’re told you shouldn’t make snap judgments. Obviously in police work, you never want to do that, right? But you’re still gonna think through the theories and the thought – you’re gonna have these thoughts and things are gonna pop into your head. So, you have to look at every possible scenario because you don’t want to be blindsided or put yourself into a potentially – a bad situation. So, armed with this information, obviously I have to make my leadership aware. I’m talking to the agent in charge, I’m talking to my bosses and just letting them know hey, this is what I’m seeing. We really need to go have a conversation with the mayor so it gets out, figure out why he’s logged into this computer at this time. So, we end up setting up a meeting with the mayor.
JACK: [MUSIC] So, on your way to meet with the mayor, how are you going – I mean, you’ve got a different couple ways of doing this. Are you going to get your backup to distract him while you grab his computer off his desk or are you going to do bad cop, good cop and sit him down and say we know what you’ve been up to, and we can make this easy or hard – like, what’s your strategy of confronting the mayor here?
NICOLE: Right, so, I am not the beat-around-the-bush type of person. I’m very direct typically, especially when I’m doing an interview or an interrogation. I tried good cop, bad cop; I’m not a very scary person, so that doesn’t work very well unless I’m the good cop. [00:40:00] We go meet with the mayor, and I start the conversation. He’s like oh, can you give me an update? I’m just walking through and I’m like yeah, so, you know, we did the search warrant. We see there’s a local IP address that’s on the network at this time. We really need to talk to you about this because it’s coming back to you.
JACK: She shows him the date and times when someone logged into the police department. He says no way; it couldn’t have been me because I was at work in the mayor’s office at the time. This alibi checks out, because people did see him in the office then.
NICOLE: Obviously we’re asking do you have kids, do you have somebody else staying at your house, is there additional people that have access to your computer or these credentials that would be able to access this server? He’s saying no, he should be the only one with access to this server.
JACK: Now, at this point, Nicole is doing more mental gymnastics to try to figure out how and why. How did the mayor’s home computer connect to the police department’s server at that time? It’s possible he’s lying and was either home that day or had some kind of remote access connection to his home computer and then connected in, but if he’s going to do something bad against the police department, he’d probably want to hide his tracks and not do it from his home computer. I mean, if he’s savvy enough to do remote connections and hack into things, then he would know he needed to hide his tracks better, right? She believes him but is hesitant. She looks at her boss who’s also in the room and then back to the mayor, and asks him another question. Well, have you ever used your home computer to log into the police department’s server before? He says…
NICOLE: Yeah, I was probably logging in to check my mail, my e-mail. I’m, again, completely floored at this point, not quite understanding what just came out of his mouth, right? I reiterate; okay, you’re logging in from your house to the police department’s domain server to check your e-mail? He’s like oh yeah, we all do it, every one of us. I’m like, what do you mean, we all? Who is we all? ‘Cause then I’m really starting to get concerned, right? He says well, I do, the city council does. Yeah, whenever we’re working from home or we’re remote, we just – and we’re not in front of our computer, we just log into the server and check our e-mail. I’m thinking, okay. I said, do you – what are your credentials to log in? Do you have separate e-mail address, password? Like, it’s set up for every person? Am I gonna see multiple accounts logging in? [MUSIC] He’s like oh no, we all have the admin credentials; they’re all the same. All of us log in. We just check whatever e-mail we want.
JACK: Apparently what him and others were doing were logging into this server through Remote Desktop and then using this computer to log into their webmail to check e-mail?
NICOLE: Correct, yeah. Yeah, so, admin credentials to this server, to RDP in, and then they’re checking their e-mail. I have seen a lot of stuff in my life, but that’s the – takes – that takes the cake. So, I just look at my boss and shake my head ‘cause at that point, I don’t really know what to say. We’re just like alright, thank you for your time.
JACK: This threw a monkey wrench in all of her hunches and theories. The network was not set up right. For whatever reason, someone decided that it was too much of a risk to have the webmail server exposed to the internet for people to log into, but thought it was perfectly fine to have the domain controller exposed to the internet for people to log into instead? Not only that, but to have them log in as admins, which means they have full permission to change anything they want or do whatever they want in the network? For instance, with domain admin access, the mayor could easily read anyone’s e-mail, not just his. He could sabotage users like change their passwords or delete records. Admins have full control of everything. The thing is, the domain server is not something the users should ever log into. [00:45:00] There’s just nothing there to help them be productive. It’s not where files are stored or even e-mails. This server does behind-the-scenes work, authorizing and authenticating connections among other stuff. Again, in this case, the mayor wasn’t accessing e-mails that were on this server. He was getting on this server and then using a browser to access e-mails on another server. It’s just silly. So, Nicole packs up and leaves the mayor’s office with more questions now than before she arrived. She calls up the security monitoring company to ask them for more information.
NICOLE: I have a conversation with the security vendor and say look, can you give me a list of all of the admins that have access to this computer? A) They’re with you or with the city, or anybody you know. Pull up on your computer who has access to this computer, this server. So, they give me a list and there are actually several people on this list, the mayor being one of them, and all of the city council, a secretary. So, there’s a whole host of people that have access to this server.
JACK: What’s more is that some of these people are sharing their admin log-ins with others. So like, if the city council member has a secretary, sure, go ahead, give the secretary this admin log-in so they can check their e-mail, too. This is a personal pet peeve of mine; I hate it when admin log-ins are shared, because when you have multiple people logged into one account, you have no idea which person is doing stuff. Is it the secretary that just logged in? The city council member? The mayor? Nobody knows, which is horrible when you’re trying to account for what’s going on in your network. She then told the IT company what to do.
NICOLE: Again, immediately it’s obviously you shut that down. I want you to delete those credentials and reset all the credentials for this server.
JACK: Of course, the IT company did not like this idea since it meant that city council members and everyone couldn’t check their e-mail remotely anymore. But the network obviously needed to be redesigned badly. But Nicole still had this mystery; who the hell logged into the police station from the mayor’s home? Well, they asked the mayor if they could investigate his home PC and he said yes. But it was around this time when Nicole moved on to another case and someone else took over that investigation. But she did follow up to see what happened.
NICOLE: Yeah, I did hear after the fact that they were able to find a phishing e-mail. There was credentials stolen. There was somebody in the mayor’s computer that ended up gaining access to the server through the mayor’s home computer.
JACK: Someone sent the mayor a phishing e-mail. He clicked it; this gave the attacker remote access to his computer. The attacker put a keystroke logger on the computer and watched what the mayor did. The mayor went and logged into the police department’s computer to check his e-mail, and the attacker saw all this, including his password he typed. From there, the attacker logged into the police station, and that’s how the police station got infected with ransomware the first time and almost a second time. The investigators were able to see whoever hacked into the mayor’s computer was coming from somewhere in Europe. But they didn’t track this down any further. That would just cost more time and money and probably wouldn’t result in anything.
So, there’s this practice in IT security of giving your users least privilege. Just give them the minimum necessary rights to do what they need to do, and maybe only give them the rights for a short duration, because this severely limits what a potential attacker can do. When you give someone full admin rights, it really opens up the attack surface. People can make mistakes, too. Maybe they accidentally shut down the domain server because they can as admin. Another thing to watch out for is when actual admins use their admin log-ins for non-admin things. Admins should only use their admin accounts to do admin-type things. They shouldn’t be logging in from home as admin just to check their e-mail. What did the police department do after this as far as changing their posture on the network or anything at all?
NICOLE: Yeah, so, they did a lot. They ended up firing the security vendor that they were using. They hired a new security vendor which has been fabulous. They ended up choosing a new virus protection software. They completely wiped all of the computers one by one, especially those in the patrol vehicles, upgraded those to new operating systems, they started being more vigilant about restricting the permissions that were given to staff for certain things, [00:50:00] reinstalled their VPN, thankfully, and had no network lag there. They changed and updated all the passwords. So, there was a lot that they did after the fact. My understanding is they’re – that’s a process because it costs so much money and obviously it’s a government agency – budgets only allow for certain things at certain times. But this was a process over time. I’m sure that they’re continuing to work on that, but they did quite a bit right away.
JACK: Yeah, a redesign like this does cost a lot, but they had their hand forced because the attorney general found out about these security incidents and was not happy. The attorney general revoked the police department’s access to the gateway network.
NICOLE: The gateway network is how this police department gets access to new suspect information, how we run suspects, how we run for doing traffic stuff, how we run plates. There’s a lot of information that’s coming back from this system. For a police department to be shut off from that system, which they were denied access to that, they had to use another agency to pull data. Obviously it’s both good and bad, right? It’s good because the attorney general is taking a very hard and fast stance with that in saying if you can’t control your networks and your systems, then we’re not allowing you access to ours because you’re a security risk. But in – at the same time, this is then also hindering the operations of the police department and could potentially put officers’ lives in risk for not being able to run a suspect for warrants or if they’re on a call. So, it – I see both sides of that coin. But they did eventually get granted access back after they could prove that they had done all of these upgrades.
JACK: With their network secure and redesigned and their access to the gateway network reinstated, things returned to normal. But it was certainly disruptive and costly for the police department to handle this incident. Nicole has since moved on from working with the Secret Service and is currently a security engineer where she plans, designs, and builds network security architectures.
(OUTRO): [OUTRO MUSIC] A big thank you to Nicole Beckwith for sharing this story with us. I have a link to her Twitter account in the show notes and you should totally follow her. Hey, I just released the ninth bonus episode of Darknet Diaries. Currently, it’s only available for Patreon users, but I am in the process of getting bonus content over to Apple Podcasts for paying subscribers there, too. The latest bonus episode is about a lady named Mary who got a job as a web developer, but things went crazy there which resulted in her getting interrogated by the FBI and facing prison time. To hear her story, head on over to patron.com/darknetdiaries. Thank you. This show is made by me, running at 7200 RPM, Jack Rhysider. Editing help this episode by the decompiled Damienne. Our theme music is by the beat-weaver Breakmaster Cylinder. You know what? I don’t like calling it a War Room. I’d rather call it a Peace Room since peace is our actual goal. This is Darknet Diaries.
[OUTRO MUSIC ENDS]
[END OF RECORDING]