Transcription performed by LeahTranscribes[START OF RECORDING]
JACK: Hey, it’s Jack, host of the show. Did I ever tell you about the time I tried to sneak into the Pentagon? Yeah, after college I took a trip to Washington, DC all by myself. I like traveling alone, I guess. There’s a certain kind of freedom I like about it which allows me to reinvent myself on trips. Anyway, there’s this metro, a subway, that goes underground through Washington, DC. I jumped on it just to see where it would go and one of the stations it took me to was the Pentagon. I’m like alright, this sounds cool. So, I jumped off and somehow ended up at the employee entrance to the Pentagon. There were no visitors allowed in this area for sure, so I stood and watched how people were getting in and out. Out were one-way turnstiles. In; everyone was scanning their badges and went through a metal detector. I decided to try to do a fake badge scan and see if I could just walk on in. I saw some guy walking up, so I followed him and did exactly what he did. He leaned over, scanned his badge on the reader, and then walked through. I leaned over, waved my hand over the reader and walked through, too. Immediately two security officers stopped me and didn’t even ask what I was doing. They simply turned me around and sent me right back out. They knew exactly what I was up to and must have spotted me like, a mile away. I’ve never been shut down so fast or kicked out of some place that quickly. No words were even spoken; they just blocked me, wouldn’t let me go any further, and pointed me straight to the exit. It’s funny what we remember on our trips, isn’t it? Anyway, this episode I interview two different NSA agents. I really like both of these guys and I think you will, too. What’s common between them is that they both started something at the NSA which still goes on today.
(INTRO): [INTRO MUSIC] These are true stories from the dark side of the internet. I’m Jack Rhysider. This is Darknet Diaries. [INTRO MUSIC ENDS]
JACK: Alright, so our first guest is quickly becoming a legend in the IT security space. He’s written four or five books on security now. I can’t even keep track. I started out by asking him where he grew up and he said in a small little country town in Texas and had horses and stuff, which sort of shocked me because I thought – actually, I don’t even know what I thought.
MARCUS: Hey man, hey, ask me any question you want to because this is one of the – this is quite funny to me. I can explain the country/hood paradigm.
JACK: Yeah, I mean – alright, I’m gonna hit Record again.
MARCUS: We’re gonna have some fun, man.
JACK: Okay, I don’t want to be embarrassed here with my wrong questions, so…
MARCUS: No, I get – no, you can’t embarrass me, so you shouldn’t feel – I feel totally comfortable with any question you can ask.
JACK: Well, I mean, I’m picturing Lil Nas X at this point, the black country singer.
MARCUS: Basically, yeah. [MUSIC] I mean, I’ll explain this to you, man. This is a crazy story. My dad’s actually originally from LA. Before I was even conceived, my dad moved from LA to my small little country town because his uncle was there – my uncle too, right? So, he was in LA and his vision of what Texas was was cowboy boots and all that stuff, so before he came to Texas he had bought all this cowboy gear to wear. This was in the 70s, right? He came to Texas. He was dressed up like a cowboy. But when he went out to meet other people, they were dressed like they were from LA, like Shaft or something. You know, like Black Panther Party. [00:05:00] But people thought he was cool because he was from LA. Now, many people where I from, my little small Texas town, they started dressing like cowboys, all the black people. The world is crazy, man, how we all fit in and stuff. Yeah, so that whole – I really feel that whole – that Lil Nas X thing, yeah, for sure. I mean heck, I got some cowboy boots still to this day. I got some nice cowboy boots.
JACK: But even though he was outside wrestling pigs, chasing chickens, he was still drawn to computers.
MARCUS: [MUSIC] Yeah, computers has been my love since I saw WarGames. I saw WarGames when I was young and I’ve been absolutely fascinated with computers and playing with them, so that was my introduction to – wow, I need to get one of those things.
JACK: But his family wasn’t able to get a computer for their home, so the only way he could really learn on it was at school or a library or other people’s computers.
MARCUS: In high school, I took Pascal. I also took BASIC in elementary, so it was something that I was always interested in. I just didn’t have the financial means. One day, a Navy guy was walking on campus. I didn’t have money ‘cause I grew up poor. I didn’t have money to go to college and I didn’t know about any grants or anything, and I ended up scoring high on my military entrance exam. I said, I want to work with computers. He’s like alright, we have this thing called cryptographic communications. We don’t know what that is because it’s classified. Do you want to do that? I was like yeah, sure, I’ll do that.
JACK: So, he joined the Navy but no matter what you want to do in the Navy, everyone first has to go through boot camp, [SINGING] where you get fit, learn combat techniques, and learn how to follow protocols. Marcus graduated and became a sailor and after that, he went on to study cryptography.
MARCUS: Yeah, so basically you go to Corry Station and you become – they teach you about signals intelligence and cryptography and all kind of crazy stuff. I was born in a small town. I graduated from Waco High and then I immediately go into – get a top-secret clearance and all this other crazy stuff. I’m poor, homeless pretty much in high school, and moving around on a lot of places until I moved to a – I’m on this military base and I get this top-secret clearance and I start learning all this crazy stuff. It was absolute night and day experience. It was the craziest thing ever.
JACK: He was taken care of in the Navy; always had food, medical checkups, clothes, a place to sleep.
MARCUS: My family though were still struggling and all that stuff. Psychologically that was tough because I was doing fine myself personally, but I had a family still back in the hood, struggling.
JACK: Back in – is it really – do you really call it the hood in the country?
MARCUS: So, let me explain that to you. 100% the black side of town, the poor side of town, definitely down south, is the hood. It doesn’t matter how big the town is. Absolutely, certainly hood.
JACK: Okay. Alright, so can you tell me some of the training you did in cryptographic communications? Was that what it was?
MARCUS: Yeah, yeah. Basically, when I went – the Navy taught me cryptosystems. Basically, the Navy has these ridiculous cryptosystems that secure communications. I had to learn how to operate those and I had to learn communications techniques that were specific to the Navy. Some of that stuff’s still classified like a mug, but you learn particular protocols and things that you – how to communicate from ship to ship, from ship to the White House, even. You learn how to do these communications protocols. Yeah, so it was pretty cool. I went from wanting to work computers to be fully immersed with computers in a couple weeks. It was crazy.
JACK: Did you do much time on a ship?
MARCUS: Yeah, I did three years on a ship.
JACK: While you were on that ship, were you handling the communication aspect of it?
MARCUS: Yeah, on the ship – my whole job in the military was you’re pretty much an attache or an asset for NSA. The whole time I was in, I was kinda like a spy. It was the craziest thing, man. Yeah, you do serious collection work and all that stuff, so you’re tasked by NSA to do what you do. It was the craziest experience ever, man.
JACK: [MUSIC] Hm, secret [00:10:00] missions, huh? This is fascinating to me because I thought the NSA was their own separate group but yeah, if there’s a Navy ship positioned in a place the NSA has no eyes or ears on, then sure, utilizing the cryptographic capabilities of the ship and crew makes sense.
MARCUS: Yeah, so the people don’t realize the NSA is the Department of Defense asset. The whole NSA supports the military. In each service; Navy, Army, Air Force, all of us had our own – and the Marines – had their own little signals group, you know what I’m saying? Like, that supports the mother ship. The mother ship is NSA. Pretty much, even though you’re a Navy sailor, you belong to – you’re an intel asset. I don’t know if that makes sense but that whole time – and you can get stationed at the mother ship, too.
JACK: Interesting. So, the DOD or Department of Defense is the department that the military falls under, and the NSA. I guess it does make sense now that they share resources sometimes. Marcus himself was that shared resource. Sometimes he would do missions for the Navy and sometimes he would do assignments for the NSA.
MARCUS: Definitely, geographically-specific stuff that you could do to be helpful. There was stuff that we did; we helped find people. If there was a ship that was lost, we could help find that. If there was a pilot shot down, we could help find them, or if there was some kinda incident. The tools that we had could be used for a lot of different things. It was cool doing it and it was great being out there on the front lines doing that work. It was dope.
JACK: Back in 1969, a Navy patrol plane was shot down in the Sea of Japan and there wasn’t a good method for handling all the real-time communications that were needed to help rescue them. After that, an operations center was constructed in order to get real-time updates from any ship, plane, or base nearby. Now, Marcus really took his job seriously and really sunk his teeth into the computers that were on this ship, learning about servers and networking and wireless technologies, cryptography, security, programming, and command line tools.
MARCUS: There was also HF communications, UHF, all the different stuff; satcom. I learned all kind of communication in the military. That made me pretty thorough in understanding how radio frequencies work and all that stuff. Definitely, you did a lot of communications training. That included air networking.
JACK: Now, while you’re on the ship, you’re – are you – you’re doing more training, right? You’re learning more about programming or cryptanalysis or something?
MARCUS: I mean, the Navy is full-time education. You’re always learning. You’re always doing OJT as well. We called it on-the-job training. You’d never stop learning in the military. I think the military, just like college, it teaches you how to learn. Since being in the military, that helped me be able to put my hand to anything. Yeah, I learned coding at the military, I learned internetworking. I was a CCNP when I was in the Navy, so mass notifications, and definitely being affiliated with NSA, I got any training I wanted to. When I was there, I got well over $100,000 worth of training and I did a Master’s degree, too. It’s like, I tell people it’s like being the Jason Bourne of IT or technology.
JACK: You got your Bachelor’s and Master’s in the Navy?
MARCUS: I got my Bachelor’s in the Navy and I got my Master’s as soon as I got out. I was still working around that stuff, but I got a free Master’s degree. I didn’t have no college debt or anything.
JACK: Because the Navy paid for it.
MARCUS: Yep, the military paid for it. I went in with no college or nothing like that. But after eight years I had a Bachelor’s degree and three years after that, I did my Master’s degree. That’s the good of the military.
JACK: He spent four years in the Navy and during that time he somehow met his wife and got married, and they had two kids. Because of this, he decided to spend another four years in the Navy. It was good job security. After eight years of being in the Navy, he then went to Fort Meade.
MARCUS: [MUSIC] I didn’t want to go to Fort Meade but I ended up – pretty much I had two options; I had Washington State and I was like, I don’t want to go to Washington State. They said or Fort Meade. We got a couple of places – a couple of jobs at Fort Meade that you can do. I was on a ship, so – and you had to be like, okay. You had to pick it right then and there. The military has these people called detailers that send you places.
JACK: So, if you haven’t guessed, Fort Meade is where the NSA headquarters are. Marcus went to work for the NSA. But he was still in the Navy and sort of on loan to the NSA. It’s called augmented staff.
MARCUS: Initially there I was doing [00:15:00] communications. It was proprietary communications systems that the military and DOD used. But what’s cool about that – I kinda worked at a NOC, and the NOC also had all kind of other cool stuff. Like, they had a heavy Cisco – they had heavy Cisco stuff back then and I learned – that’s how I started getting to the CCNA. I became a beast at Cisco stuff, so I ended up getting promoted to an engineering team of network engineers and I got to manage the whole NSA’s network. I started off doing a crappy job at Fort Meade and then the certifications allowed me to ascend to top teams there. That was during my day job. My day job, I was Navy. At night time, I took – I had a part-time job with a DOD contractor so I was like, doing a night job since I had the clearance. I ended up helping build out the NSA’s SOC. I ended up helping build that out. I wrote stuff like SIMs and all that stuff, so I started coding heavy as well. I made as much on my night time job as I did with my Navy salary. I was like man, I gotta get out. I gotta get out of the Navy and make this money.
JACK: Alright, so two words in there you may not know; NOC and SOC. This stands for Network Operation Center and Security Operation Center. This is a place where people watch the network for any kinds of problems, so there are typically multiple monitors on everyone’s desks and even a big screen in front of the room which monitors all the networks. The NOC typically looks for network-related faults; a router that went down, a switch went down, some office lost internet connectivity, and that sort of thing. A SOC watches out for security incidents and responds to threats. But both the NOC and the SOC are monitoring NSA’s network itself, looking for any threats that have targeted the NSA.
MARCUS: Oh, 100%, man. NSA is probably the – probably one of the most attacked organizations in the world. Absolutely crazy amount of attacks that go on there. That SOC has to manage all the networks. What they’re doing is they – there’s all kind of different levels, like high side networks and medium-tier networks, and then there’s unclassified networks. You have to defend all three of those or however many there are. There’s like, all kind of – and then there’s inner agency networks. Nobody trust nobody. It’s crazy.
JACK: Now, of course, I’m super-interested to hear what goes on in the NSA. What kind of detection capabilities do they have and what offensive tools are they using? But I can’t ask any of that to Marcus because they’re not allowed to share means and methods of how the NSA operates.
MARCUS: You don’t want people knowing how you collect information. I’ll give you a good example; a couple years ago – I don’t know if you remember this, but supposedly Bin Laden was using satellite phones to communicate, so – and basically, Orrin Hatch was – he was a Republican senator from Utah. He came out of a intel brief and he’s like oh, don’t worry about Bin Laden. We’re tracking him on a satellite phone, right? Orrin Hatch said that, out of the intel community. So, that’s a means and a method, right? The method we were tracking this number-one terrorist in the world was satellite phone. What happened is that burnt all – that burnt that method, you feel me? Now all the criminals that were using satellite phones and stuff, they now knew that – something that was being tracked.
Bin Laden went silent and that’s why it took so long to find him. When it comes to intelligence, the – how we collect the data is what matters. Now, as far as securing data, securing data is just like any other thing. A matter of fact, NSA and NIST, they work hand-in-hand to try to help all American businesses stay secure as well. As far as what NSA’s doing on the defensive side, you just look at what NSA tells people to do and most of that stuff’s public. Good defense is public. It’s the offensive means and the way that people – how they collect information that’s really secretive. I would never talk about how we collect information.
JACK: Yeah, it’s odd to me because there are publications that the NSA puts out, like Best Practices for Keeping Your Home Network Secure or Securing the Teleworker. They even have configuration recommendations for securing certain systems. But at the same time, the NSA loves the ability to collect data on their targets. If you follow the guidelines, then it makes it harder if the NSA were to target you.
MARCUS: That’s not true, though. NSA’s core mission is to protect US communications and assets. That’s like, the core mission. [00:20:00] People don’t understand that. A lot of the crypto research and breaking crypto and all of that stuff, and even exploitation; that stuff is – the core mission is to protect US assets and interests because what happens is all these American companies that go overseas to do business, they’re being spied on by foreign intelligence. Heck, foreign intelligence hired people to work in these big companies, by the way. All these big companies that you can think of, they have IP – I can guarantee you that they’re either paying an employee or they have moles inside of them stealing information and sending it over to their countries. I mean friendly countries, too. Think of a friendly country. They’re spying on us, too. What the agency’s core mission is – to do all this crypto research and all that stuff, is to protect our interests. That’s part of the mission and people don’t think about that piece. It’s a serious mission and people take that seriously. The other side of it – go ahead.
JACK: Well, my counterargument there is if there – if they want the US companies to stay secure, how come when they find zero-days on things like Microsoft or Google products, they don’t just tell Microsoft and Google hey, there’s a bug in your code?
MARCUS: You know, it’s funny though because you know that people say this all the time. There is nothing new under the sun. I can guarantee you if we find an exploit, somebody – some other country, some other person on the market somewhere found that exploit. I think that they should disclose everything. That’s not my decision to make, but the reason why though is so – is to help out our country. I totally believe – I 100% believe that the folly on their part is the thinking nobody else has the zero-day ‘cause I think that if you have it, somebody else has it, too, right? That’s the folly. But the reason why they try to protect that method is so they can help out the country.
JACK: This is obviously a complicated topic that we’re not gonna solve here. NSA has made many mistakes but at the same time, they’ve saved many lives. That’s a hard line to walk for anyone. But because it has some of the most advanced technologies, Marcus was having a blast working there.
MARCUS: [MUSIC] I loved it. It was so much fun, so much like, leading edge technology. Like I said, their – NSA put like hundreds of thousands of dollars into my education. NSA has its own training environment as well, so they train you their stuff and then you get to pick a list of what class I want to go to. Do I want to go to SANS? Do I want to go to the Cisco course? Do I want to do this? You get to take whatever you want to. Oh, do I want to take this course at the community college here? It was like, I worked two or three weeks and then I was off for training for a week. It was like, every month I was training. It really was so crazy as far as the educational benefits. That’s what you’re dealing with there. I would say that our foreign adversaries, those people are well-trained too, right? Basically it’s a lot of really smart people fighting this little behind-the-scenes battle. It’s crazy.
JACK: For a while, he was working for the NSA while still in the Navy, building out their network. At night he was a contractor and helped build NSA’s SOC which is pretty cool because the SOC is still up and operational today and Marcus is the one who built it. Once it was built, he was using it to defend NSA’s networks. But after a few years of that, he got out of the Navy and went to work for the Department of Defense Cyber Crime Center.
MARCUS: The Defense Cyber Crime Center does all the forensic investigations and things of that nature for the DOD. They’re like Cyber Command. Basically, the DC3 was the lead on all of the investigations until the – ‘til they created Cyber Command, essentially. The DC3 still exists. It has this thing called DCFL. They do forensics. It’s a forensic laboratory. They do a lot of forensic investigations. They do a lot of high-profile investigations that you’ve probably heard of on the news before. I work for CSC and at the time CSC had the contract to help train federal agents and all the DOD agents on how to do forensics. It was a pretty cool curriculum. They started from log analysis, Windows forensics, Linux forensics, Macintosh forensics. We teach these federal agents all these different forensic techniques, working with the best of the best software back in the day, and they still [00:25:00] do it today. The agents would get to use EnCase and all of these different open-source tools and all these different forensic things. But what was cool is I got a chance to build up a cyber range. This cyber range had – it was a complete mock-up of a corporate network. I mean, this is before cyber range is even cool. This is over ten years ago.
JACK: Do you know what a cyber range is? Let me tell you a story. Before I was podcasting I was working in a SOC myself, watching the network for security incidents, and I saw one of my co-worker’s computers lighting up my screen. It was triggering alerts telling me that this co-worker’s computer was actively trying to hack itself. To me, it looked like maybe someone took over his computer and was trying to get more access or something. I went over to his desk and I said hey, everything okay? He’s like yeah, what’s up? I said, I’m seeing some alerts that some pretty nasty PowerShell commands are being executed on your computer. He said oh yeah, I downloaded a PowerShell tool to see if it actually works. I said, this is not a safe environment to be running random hacking tools.
Please stop that. Delete it and run an antivirus scan right away. But see, the thing is, we didn’t have a safe environment to try hacking tools like that, so this is what a cyber range solves. It’s a separate network with all kinds of servers and computers to attack, as well as a whole range of nasty weapons to launch attacks with. A cyber range is great because you can really go nuts. You could try to exploit anything you want and you don’t have to worry about any vulnerability or virus or worm escaping and hitting production equipment. On top of that, defending teams can use the cyber range to see if they can detect and defend against such attacks. It’s a great place to practice network and system security.
MARCUS: Yeah, you can use them to detonate malware, all kind of different things of that nature. Usually some companies are trying to mock-up their corporate network almost like a development play, or I used to have dev environments and production environments. A cyber range is like that kind of thing but is used for cyber-security testing.
JACK: What does it look like? ‘Cause I’m trying to picture it. Do people go into a classroom and then there’s a server in the middle of the room and that’s where the range is and everyone tries to connect to it or is it all remote?
MARCUS: Our range was – the thing that we built was ridiculous. We had a complete – we had a nice-sized network room. Picture like, six or seven stacks of devices; Cisco devices, Windows servers, Linux servers. You’re talking about DNS, firewalls, IDSs, switches, routers, the whole complete corporate network, and we had physical gear back then. Everything was physical and real devices. In another room there was a classroom environment but it was networked to be into the – into this comp – into all this gear. Why this was important is because it allowed the investigators to actually interact with real stuff. They could come in and physically put a USB drive and collect information off of a server or they could go onto a CICSO switch and they could do a spam port on it. A lot of cyber ranges now are virtual but this was a physical representation of a real corporate network, and it was dope.
JACK: So, him and another guy named Johnny Long set up this cyber range to teach federal agents how to react to cyber-security incidents more effectively.
MARCUS: We had complete network – corporate network set up. Funny enough, I worked with Johnny Long at the time. Johnny Long was my buddy. Johnny Long would come up with scenarios where we had to attack, and the federal agents would have to find us on this network that we built. Since Johnny was always gone a lot, I ended up taking over all of the offensive scenarios, and so I became the bad guy. But these federal agents were like, forensics gurus. Some of these – this was like, the capstone course and they were beasts at forensics and stuff. Many of those people have gone to work for places like Mandiant and CrowdStrike and all that stuff. It was like a CTF but I was playing against professional people that catch people – bad guys on their network. It was pretty dope. What happens is people don’t know how to respond to an incident live on the fly. Usually when incidents are happening, they happen way before time, but if you were to drop people in, how do they respond live on the fly? That’s why we built this course. Basically, you – we taught them how to collect live information. We taught them how to set up a intrusion detection system on the fly, doing packet captures on the fly, setting up those SPAN ports. It was like, super-intense, man. It was dope. It was like, an [00:30:00] immersive course. They got a lot out of it.
JACK: Well, yeah, I mean, this is exactly what I picture when somebody goes into training at the NSA, right? ‘Cause if you say oh no, I just sat in a classroom and they taught me Pascal or something like that, that’s kind of boring. You’re sitting at a terminal with a bunch of other people and you just work on your own little thing and whatever, but going into an entire organization like a full campus network and getting access to all these things and you’re running around, plugging USB drives in or SPAN ports and figuring things and putting collectors in – like, physically in the network to get off the SPAN ports and stuff, that’s so much – yeah, like you said, it’s immersive and that sounds like the training I think everyone wants.
MARCUS: Yeah man, it was crazy, and they spent a lot of money on it. That’s the thing about working with the DOD, bro; money’s not a problem. We got to do pretty cool stuff like that man, and come up with different scenarios. Dude, we had like live Chinese malware on that network, bro. I was doing command and control for it. It was real attacks.
JACK: Like zero-day malware I bet, too, like stuff never – no one knows about.
MARCUS: We would grab stuff off – we would grab stuff and put it on there and they had to figure it out.
JACK: [MUSIC] Marcus took what he learned at this place and started his own company creating threat scenarios and doing more cyber range and tabletop exercises. He called it Threatcare. But Threatcare was acquired by a larger company called ReliaQuest which is where Marcus works now. But Marcus likes giving back to the community, so he wrote a book and it’s called Tribe of Hackers. The book interviews a bunch of notable people in IT security and tries to distill meaningful advice from them. When the book did well, he wrote another and then another, so now there’s Tribe of Hackers Red Team Edition, Blue Team Edition, and Tribe of Hackers Security Leaders. I’ll have links to all these books in the show notes. I own three of them so far and I find these books super-informative to me for finding interesting guests for this show.
MARCUS: Yeah, so basically the first edition of that book, we happened to be able to give out a lot of donations to a lot of different organizations. Rainforest Project did one of those things. We also donated thousands of dollars to Hackathons for kids. Just all kind of different organizations we’ve been able to give money to from those books. Also, we – I partnered with Wiley, we also raised additional money with the new books doing the Humble Bundle program. So, we just – I’m just continuously finding ways to try to give back to the community. What’s cool about the first book; we gave it away – we gave away that book for free, too. So, we had thousands and thousands of downloads from people that – they got that book, too. It was definitely well-received and helped a lot of people.
JACK: On top of all that, Marcus has even written a children’s book about security. There’s a lot of people out there that once they figure something out, they’re afraid to teach others because they fear someone else will learn it and get better than them. But Marcus is the opposite. He knows that the more he teaches, the better he becomes. I just love it when people out there are sharing all their skills and knowledge with anyone interested in learning.
MARCUS: Here’s the deal, man; work hard. I tell people, be so good that they can’t ignore you. Another thing that I find in life; if you self-train yourself, if you learn, the more you learn, the more people are gonna give you.
JACK: Oh, and of course, one of the people he teaches the most is his son who is also into technology.
MARCUS: Yeah, certainly. This is the story about my son; it’s a funny story. When he was eleven years old, he hated doing his homework. So what I did is I – my son is super-spoiled, man. My son had every iPhone that ever existed when he was growing up. He’s twenty-four now so he’s out of the house and he’s definitely doing well for himself. But he had the first iPhone, he had a MacBook, the whole nine – when he was like, eleven. He’s used to jailbreak iPhones all day. He was the jailbreak dude at school. He didn’t like doing homework so I was like hey, we can actually do code and your code can do your work. So, we used to write the math formulas in Pearl, all the algorithms, and it would give him the answers, and we programmed it to give him the work too, so he could work out the work.
He used to do – eleven years old, he was writing programs to do his math. Fast forward ten years, like fifteen, sixteen, he’s like, I want to be a video – I want to make video games. So, I was like alright, cool. So, I did the front end UI stuff for him and he wrote all the back end code and Objective-C, and he did six iPhone apps. He also wrote – this is crazy – [00:35:00] he wrote a Metasploit front end on the iPhone so you could connect to the Metasploit API and you control it from your iPhone. This is when he was like, sixteen or something like that. So, Rapid7 saw it. I worked at Rapid7 at the time but they was like holy crap, this sixteen-year-old kid – he was my kid but he wrote it all himself and he could pop shells from his iPhone.
JACK: Rapid7 is a security company that’s known for creating a vulnerability scanner and owns Metasploit which is a very popular hacking framework.
MARCUS: Rapid7, he started interning for them when he was sixteen and then he did another internship when he was seventeen. He graduated high school at seventeen and then he’s like hey, I don’t want to go to college. Then Rapid7 gave him a full-time offer. [MUSIC] Now he’s been working there for like, five years and he’s a software engineer. He’s got his own team. It’s nuts, bro. He’s a phenom and Rapid7’s lucky they got him.
JACK: The other guest we have on today is the wonderful Mr. Jeff Man.
JEFF: I’m happy to start. Do you want me to just ramble or do you want to just start with questions?
JACK: Yeah, I mean, I’ll just kind of lead you through where I want with the questions and then you can go from there. Yeah, start right where you wanted to start. How did you get into the NSA?
JEFF: Okay, so it all started when I was a small child. [MUSIC] I’ve essentially been a hacker my whole life. I’ve had the hacker mentality my whole life. I wouldn’t necessarily have called it that, but…
JACK: So, you say you call yourself a hacker mentality. Now, I think a lot of listeners might think oh, that’s the hoodie and causing chaos at school and getting on the dark web and doing stuff. Is that what you mean or it’s something else?
JEFF: Fair question. When I say hacker mentality, I sort of equate a lot of critical thinking skills and a lot of curiosity about how things work and wanting to learn but not – but wanting to do it probably at a faster pace than the general population, the general classroom. I remember being a child being bored in most of my classes because I either did the work really quickly or it was just boring to me. I had already read up on the topic and knew what I wanted to know, and so I was a bored student most of the time. Therefore, I looked for things to do to entertain myself. It was nothing necessarily extremely malicious. I was known as the class clown or the class cut-up.
JACK: His dad was a physicist; worked for the DOD.
JEFF: He was in the Pacific on a ship and got to witness the detonation of the first hydrogen bomb. I went through college, look – and I went through five majors when I went to college and I was basically looking for the major where I had to do the least amount of book work.
JACK: He ended up graduating with a business major.
JEFF: I was working for a naval organization, Naval Surface Warfare Center because my mom was in HR there and she was able to get me a job, so it was a low-level clerk typist-type position. It was a way to earn some money while I was looking for what I wanted to do with my life. She had a friend within HR whose daughter, I think, had gotten a job at NSA. This was in the mid-80s now. She thought well, you should apply. I was born and raised in Maryland which is where NSA is located, Fort Meade, Maryland. I had never heard of NSA. Back in the day, NSA was no such agency. It was a super-big ultra-secret that the organization even existed; unmarked, fenced buildings off the bottom where Washington [00:40:00] Parkway in central Maryland, a little bit between Baltimore and DC. Filled out the government application, sent it in, and heard back from them and they eventually invited me up to take a couple days worth of various aptitude and skills qualification tests. At the end of the day, I scored well enough that they offered me a job. What was weird at the time – and I still think it’s kind of weird – was they kinda hired me because I had potential. I didn’t actually have a job when I went to work for NSA.
JACK: Right, so Jeff started working at the NSA in the fall of 1986. Oh, how different the world of technology was in 1986. To actually start working at the NSA, he had to pass a fairly rigorous background check.
JEFF: They focus on several different areas. They want to know where you lived in the past ten or fifteen years, you had to list neighbors of all the different places you lived, friends, people you had contact with, social and beyond, which when you’re a young kid is pretty much your whole life. They asked all sorts of questions about your political affiliations, your political leanings. They were trying to find out things about you that had been used against people in terms of blackmail or were motivations of people that they had encountered that had basically – basically had committed espionage and become traitors. Lifestyle questions back in the day; if you happened to be a homosexual or have some sort of alternative lifestyle. It wasn’t so much an issue they wouldn’t hire you if you were like that; they just wanted to know about it so that somebody couldn’t blackmail you into giving away secrets because somebody had, and that had happened. They wanted to know about your financial records so that you weren’t – they wanted to know if you had a gambling problem and had huge debts which would be, again, a way that somebody could motivate you to steal secrets and they could pay things off, or they’re the people that you became indebted to, and that was your way of repaying.
JACK: This process took weeks since they needed to visit all his neighbors and friends to see if all this information he gave them checked out. So, while he’s waiting for his clearance, he tries to figure out what he’s gonna do at the NSA, so he starts shopping around, looking for a job there, meeting people at the NSA to learn what they did.
JEFF: Going out on these essentially job interviews that were really more like – as it turned out, it wasn’t so much a job interview as a sales pitch of oh, we want you to come work for us. One of the first interviews that I went on happened to be on the defensive side of the house. Back in those days, NSA was operations which is what most people know NSA for; intercepting communications, stealing all the secrets of the rest of the world, our enemies. Then there was also the defensive side which was called information security or InfoSec. I happened to – my first interview was on the InfoSec side. It was an office that was responsible for manual or paper crypto-systems. They were looking for someone to do a cryptologic, cryptographic review of all the manual paper crypto-systems that were currently deployed.
JACK: They needed a cryptographer and nobody at the NSA seemed to want to step into that role and help that particular office out.
JEFF: They thought well, the next best thing is let’s grow one, so let’s hire somebody off the street. We can go to the pool of people that are out there and train somebody up and train them to be a cryptographer and then have them do the review.
JACK: So, he took the job as a cryptographer.
JEFF: [MUSIC] So, I ended up going to work in InfoSec and working for what was known as the Manual Cryptosystems Branch. My job was to do cryptographic reviews of the systems that were being used at the time.
JACK: At this point, he had to learn what cryptography was and get good at it.
JEFF: NSA ran its own cryptologic school. They had 100, 200, 300, 400 courses, dozens of courses that you would take on various aspects of cryptography. I basically went back to school and took a lot of these training courses. What was interesting was I was learning a lot of classic manual cryptography on – back to the Ancient Greeks and Romans and how all the cryptosystems were used over time and how they evolved and all that type of thing.
JACK: Which was perfect because this was the same stuff he was tasked with reviewing as a cryptologist. One of the things he learned about is what a one-time pad is.
JEFF: A one-time pad is – the name [00:45:00] implies it’s a pad of paper usually forty, fifty pages, and on each page there is printed-out random characters. But it was essentially a key and the very basic form of encryption is you take your message, what we would call plain text; you would write that message out one letter at a time over or above the letters that were printed on the pad. So, you have plain text and you have the key pre-printed. You would go through some sort of cryptographic algorithm to produce a third letter which was the message, the encrypted message, or what we called the cipher. Quite simply, the way a one-time pad works is it’s – you use some sort of substitution algorithm to produce that third character that’s reversible. The beauty of it is that if – there’s obviously two copies of the pad but if those copy – if those pads are kept secret and nobody can see them and nobody steals them, there is no cryptographic solution if you intercept the cipher. You can collect it all day. There’s no way to break the underlying key because it’s random and therefore, there’s no way to break back what the actual message is. It’s called a one-time pad because you use the key one time. You write a message down on a page or two pages if it takes two pages, and as soon as you do the ciphering and transmit it, you destroy the pages.
JACK: These one-time pads were often used by spies like when a handler needed to talk to an asset, they would encrypt messages by hand using this method. Then the receiver would have to spend a while decrypting the message. But a lot of the time, spies didn’t want to lug around big pads of paper, so they shrunk the one-time pad down to like, an inch or two wide so it can be transported in a shoe or rolled up into a pen. In fact, some of them were printed on rice paper because you needed to destroy the pad after you used it and some spies would destroy it by chewing it up and swallowing it.
JEFF: But there was a run one time, a production run made one time of a set of one-time pads where they printed them on rice paper but for some reason, they didn’t take into account what type of ink that they were using, and they used some – I don’t know if they got a deal on some different kind of ink, but the ink that they ended up using was toxic, so our loyal spies that were sharing secrets with us in the field were getting sick.
JACK: Now that he was getting up-to-date on all this cryptography, it was time for him to start using it on the job.
JEFF: [MUSIC] My first assignment was – we were approached by I’ll just say a customer generically that was working with people in the field, and they were having these people report back to him on a rather regular basis; reporting information, data, secrets, things that they were observing. They were lamenting that it sometimes took them hours to do the decryption of these messages that were being sent to them and they asked the question; we’ve got this new-fangled IBM PC on our desk. Is there any way that we could do this encryption and decryption on the computer and just speed things up? My naivety, because I wasn’t experienced in the workings of NSA and in particular the InfoSec organization, I thought yeah, that seems reasonable. Why not? So, I set out to try to figure out how to get a computer program written that could be performed on the PC and then how to get the one-time paper pad into the PC which the option available at the time was on a floppy disc.
JACK: Now, remember, this is 1987 at this point. Windows wasn’t a thing yet, so everything he was doing was on the command line. Did you understand software or programming or anything at the time?
JEFF: No, not really, and what I set out to do was I went and started asking questions about all the different groups within InfoSec of how do you do this? ‘Cause InfoSec had a production facility that would actually print these one-time pads. We had an office that would generate the key and that they had a way of generating a random key that could be used to put on the one-time pads, and there was other uses for keys. There was a lot of machine-based cryptographic systems available that was much more prevalent at the time. I kinda set out to – well, how do you do this? How do we design something and produce something? That certainly wasn’t something that was ordinarily done within the manual cryptosystems shop. But we had to design something and [00:50:00] how’s it gonna work? We need to have a computer program and the program needs to be written securely. If we’re gonna put the key on the floppy disc, how do we emulate pages of key? Especially, how do we emulate the destruction of a page of key?
JACK: At some point Jeff starts to think surely there’s gotta be a standard in the NSA for how to secure software like this. Like, if you’re transporting keys on a floppy, what encryption do you use and so on? So, he looked around for such a standard and he found one. It described how to secure a cryptographic device.
JEFF: This is where it started getting interesting because it was written for hardware. There was no concept of doing anything cryptographic in terms of software back in the late 80s. I say this, I’m in contact with a fellow alumni from the InfoSec organization and people that were there years before I was, and I’ve asked. To the best that I have been able to figure out, what we ended up producing which was half paper pad, half key on a floppy, and a computer program that would do the encryption and decryption. That was the first foray into software-based cryptography that NSA produced.
JACK: [MUSIC] Now, hold on, it wasn’t that easy. Any cryptographic-based hardware that was made had a strict review process. So, he had to submit this software to a few departments to get it approved for use in the field. They had some push-back and questions on this software.
JEFF: So, I had to go through several iterations of presenting to senior management. They gave me the initial blessing, came back; here’s all the security concerns. Go address them. I came back, addressed them. They ultimately said alright, we’ll let you do it but don’t do this again.
JACK: Like most government agencies, the NSA was resistant to change. They weren’t entirely sure if this new-fangled computer thing was the direction they wanted to go in yet. That’s why they were hesitant with this whole thing.
JEFF: We produced what to my knowledge, the best I’ve been able to figure out, the first software-based system that NSA produced. Now, years later, I’m a hacker. I’ve done all the hacking pen testing things. I look back and say I hacked NSA. I did something that wasn’t supposed to be able to be done. I didn’t take no for an answer. I figured out a way to hack the system, hack the process, and I got through it. Not saying I would have been successful at any other time and certainly wouldn’t have recommended this solution in a networking world, but for the time, it worked and it was revolutionary in terms of being the first foray into software.
My career at NSA was roughly three different tours of duty as – or three different assignments; this initial assignment where I did the work with the manual cryptosystems, produced the first software-based manual cryptosystem. That ended and I became a cryptanalysis intern. The intern programs were special programs that were designed to get you the training, the diversity of experience, and higher education to advance you to higher levels of your career field. In this case, cryptanalysis. What it meant was essentially that I jumped from InfoSec over to the operation side. Then I was actually on Fort Meade and a couple of my six-month tours as an intern were in one or more of those – there’s actually two of those big, black buildings beyond the four-storey structure. I was all over that which most – if you’ve seen the aerial photos of NSA, I was in those buildings.
JACK: So, he went to work in another department and that department was in the middle of looking for how you can crack encryption or exploit systems if they weren’t using best practices and securing their stuff properly. For instance, remember that one-time pad and how each page of the pad was only good once and then you had to use a different piece of paper for the next message? Well, suppose someone doesn’t do that. Suppose they used the same pad over and over. Does that make it weaker?
JEFF: People were doing exactly that. They would take a one-time pad sheet of key and they would use it for thirty days. We could amass dozens or hundreds of messages that we knew had the same key on the bottom and you can cryptographically break that and figure out what the message is. Or if it was any kind of machine system of cryptographic radio or transmitting device, there’s things that you can do to bypass the security or shortcuts to get the message through that you think you’re doing it in a secure manner but sometimes [00:55:00] you’re not. So, somebody figured out – some group figured out, okay, if the bad guys or the adversaries, the rest of the world doesn’t use the crypto correctly and we’re producing all the crypto that the US uses, the DOD and everything US-related, and we produce the best crypto in the world and it’s tested and tires were kicked on it and everything and it has to be rated to be secure for however long it needs to be keeping the data secure, how do we know that our people that are in the field are using it the way that it’s been designed in the lab and the pristine condition?
JACK: It’s a great question to ask. If the NSA can crack codes because someone isn’t using best practices in security, is the NSA guilty of not following those best practices themselves? [MUSIC] So, his task was to go around and look at servers and computers in the NSA to make sure the NSA itself was secure.
JEFF: To make sure not so much that they were designed correctly. That was sort of assumed. That was a given, but that they were being used correctly, that they were being implemented correctly.
JACK: Jeff was there in the NSA in 1993 and what’s so important about ‘93? Well, that’s when the first web browser was created, called Mosaic. This is where the web and HTML sprang out from. Once they saw people were using this more commonly, this gave them a new idea.
JEFF: That’s when the focus within this office for a small group of us started to be hey, why don’t we start doing that hacking thing that we’ve seen in the movies and start doing that and start coming up with a methodology of looking into how do you break into networks and computers, this whole internet security thing? That’s what I and a small group of us gravitated to within this office.
JACK: When browsers reshaped how the internet looks and feels, it brought tremendous growth to the internet which caused a sea change at the NSA. In fact, around this time, the US created the fifth domain of warfare. Historically, they had land, sea, air, and space, but in 1995, they added cyber-space as a domain of warfare and Jeff was right there at the dawn of when the NSA really started the internet hacking that it does today.
JEFF: Yeah, all the really smart people, the suits, as I and many of us used to call them, they got together and decided to reorganize. So, they set up what became known as the System and Network Attack Center. It was built to be the – again, we didn’t use the term ‘cyber’ but it was supposed to be a center of excellence for everything computer and network security-related. Essentially, a lot of the InfoSec side of the world was a research organization because we were designing and building things. They set it up like that. There was a design and a research arm and a couple other variations of the theme. One group was supposed to look at networks. One group was supposed to look at operating systems. The intent was to do a lot of research, dig into them, and produce standards and guides on how to secure them.
JACK: Jeff and the people in his team really wanted to embody the hacker culture within the NSA and learn how to break into systems remotely over the internet and stuff.
JEFF: This small group of guys that had gotten together originally in this branch that was focused on fielded systems, we got swept into this reorganization and moved to a different building. We were moved off Fort Meade to one of the satellite locations and we were given our own office. We were given license to keep doing what we were doing. Everybody was happy with it although they didn’t necessarily understand it. [MUSIC] But we were testing the security mostly of NSA networks and domains within the NSA proper as well as other DOD customers, let’s just say. Things were bopping along nicely.
JACK: Okay, this sounds like a good place to start; learn how to hack stuff, then test your hacking ability on the NSA itself.
JEFF: We nicknamed our office The Pit. We referred to our little hacker hang-out, as it were although again, we didn’t call it that at the time but that’s essentially what it was. We called that The Pit. We decided we wanted to have our space and give it an identity and some – one of the members of the group said well, we should give it a name. A popular show at the time was a show called MASH. The irreverent doctors in the show MASH, their tent that they lived in, they called The Swamp. So, they said well, let’s do something along those lines like The Swamp. We didn’t want to call it The Swamp ‘cause that had been used. Somebody came up with Pit and it stuck. So, within our little office which we came to [01:00:00] call The Pit, we had to get special permission to get a little mini-fridge installed in it and we filled it up with Mountain Dew. That was the beverage of choice for the hacker culture in the early-to-mid 90s. Initially there was four of us in terms of our background. I was the business major so I sort of gravitated towards the business side of things; finding customers to do this. I think most if not all the other guys were computer scientists in terms of their academic training.
JACK: This Pit as they called it was part of SNAC, the System Network and Attack Center, and they were certainly participating in the attacking part of that. They were learning how hackers operated by reading hacker magazines and forums, and they would try these attacks out on some practice computers. They were also doing their own research and generating their own exploits.
JEFF: Back in those days, we weren’t relying as much on – in fact, we didn’t even have the term yet zero-days or 0-days. We were basically learning how the operating systems worked and learning about all the hidden or undocumented features of the operating systems at the time that could get you root privileges. We were learning a lot of the tricks and the trades. Now, there were some exploitation-types of things. Some of the – and again, this is where I like to caveat these are the types of things that were done at the time. I’m not saying that we necessarily did any of these. Use your imagination. But again, everything that we did against the classified system was labeled classified, so technically if I tell you that I was doing something to a classified system, I would be sharing secrets. With that caveat, what was commonly done back then to break into UNIX systems and UNIX networks was – there was password-guessing…
JACK: Ah, yeah; according to Rapid7, the oldest vulnerability discovered in 1970 were computers using the username admin and the password admin. I think it’s really embarrassing that forty years later we still battle with the same vulnerability.
JEFF: Everybody was getting on – getting a UNIX workstation and getting a network credential. Not everybody wanted to get on that new-fangled computer so very often everybody was set up with accounts but they hadn’t actually been used yet. The way that they were typically set up was everybody would get an account. But until you logged into it, you wouldn’t set a password, so there would be idle accounts that were just sitting out there. If you could identify the username, guess the user ID, you could get in without a password and set it and it would become your account. Back in those days, the password hashes were in ETSI password files or world readable, so you could just copy them and run crack programs like Crack or John the Ripper.
JACK: Jeff and the team in The Pit were doing internal penetration testing, red teaming. But at the time, these sort of terms just didn’t exist yet. It’s typical that when you’re in government and you’re helping other government offices, you call them your customer. They would find a target which was just another office or department, and that’s their customer.
JEFF: If we wanted to do an attack, we had a target, we had a customer and let’s say it was an internal customer, we had to get permission to do what we wanted to do. In order to get that permission, we had to get management sign-off. It had to go up our management chain, across the executive suite, and down the management chain of our potential target or customer. Because it was getting physical signatures or initials on a document from ten or twelve or fifteen people sometimes, that could take weeks or months. It was frustrating because being able to break into a network, being able to break into a computer, you kinda know what you want to do. You know what’s gonna work. You think you know what’s – you’re ready to do it and sort of develop the methodology which was required in this form without actually executing the methodology, so you sort of get right up to the edge and then get told to stand down for a month until you get permission to do it. That wasn’t cutting it with us.
JACK: Yeah, I’ve heard this from other hackers in the NSA too, that they have a target they want to hack but first they have to get approvals for what they’ll do. They don’t know exactly what exploits they’ll use until they get inside that network to see what they have. It’s still a problem today in the NSA, actually, and the only way they’ve been able to solve this is to do as much open-source intelligence-gathering on your target as you can to know what to expect once you get in there so that you can get approvals for your mission.
JEFF: We’d try to learn as much as we could about the target from a benign perspective; what kind of information’s out there? Who are the people involved? Can we identify what their user ID’s naming convention is so we can start to guess account names? What can we learn about the people; their interests, their hobbies, [01:05:00] their birthdays and anniversaries and pets’ names and kids’ names, because these are all very probably password possibilities when we were just trying to guess passwords. We would do that sort of open-reconnaissance which was very rudimentary back in – we didn’t have Google back then. We didn’t have LinkedIn. We didn’t have all this stuff that is so readily available like it is today. But there were ways.
JACK: It sounds to me that Jeff helped start the very first red team in the NSA which is quite remarkable seeing what the NSA has become. Now, this term ‘red team’, it actually comes from the military. The red team was someone who acts like an adversary to test your defenses. They think like the bad guys and the blue team is someone who defends against red team attacks.
JEFF: A couple years ago, a book was published called Dark Territory by a gentleman named Fred Kaplan. In that book, in the fourth chapter which is entitled Eligible Receiver, there is a paragraph that talks about NSA’s super-secret red team – was called The Pit. Now, none of us that were the original members of The Pit have any idea how the folklore grew to the point where it was included in this book but when one of us got a copy of the book and read it, we all got very excited. It’s like hey, we’re all in a book. So, apparently what we did in the early days in our office called The Pit came to be known as the NSA red team in The Pit.
JACK: They were doing pretty good, making a good name for themselves in The Pit and helping out a lot of customers. But everything they had worked on so far was attacking and securing classified networks. But that was about to change when the DOJ heard about them.
JEFF: ‘Cause the Department of Justice had heard that NSA had this crack team of hackers, pen testers that would test the security of networks and they wanted to have that, too. When I first found out about that, I had to go to the lawyers and say can we do that? We went to the lawyers, or the lawyers got wind of the fact that an unclassified network organization was asking us to do the work and we’re like sure, they’re a customer. Let’s do it. The lawyer’s like well, hold on a minute. The general council said let us educate you a little bit on how things work here.
JACK: The lawyers explained to him that while the NSA is responsible for protecting classified networks, another department, NIST, is responsible for protecting unclassified networks. It’s not the NSA’s jurisdiction to help the DOJ in this situation since they wanted Jeff to test their public-facing website.
JEFF: While NIST was responsible for unclassified networks, it was fairly well-acknowledged back in those days that they had no capability and this is all I’m learning from the general council. What effectively happened was sort of a nod and a wink, handshake agreement where NIST would be responsible but they would very quietly sort of pass it back to NSA to actually do the work. [MUSIC] When we were first approached by people in the DOJ saying we want you to come do your thing – went to the general council and they said well, there’s a way that you have to do this and you sort of have to just follow the rules, so we proceeded – I proceeded to start following the rules. The first thing they said was well, this is sort of – this sort of has to be a cabinet-level favor that’s being asked by one cabinet member to another, so the request to do this work has to come from the attorney general to the Secretary of Defense.
I worked with the DOJ people to generate a letter that was ultimately written by – signed by the attorney general who at the time was Janet Reno asking NSA to do that thing, that vulnerability threat assessment thing that you guys do, we’d like you to do it to this particular internet-facing public-facing aspect of the DOJ. That took a little while to get going. The director of NSA had to respond officially back saying yes, we would be happy to do that for you, that letter going back to the attorney general. It was like a three-month process and in all of this negotiation, we were down to the letter had been actually drafted, signed by the director of NSA who at the time was General Minihan – he was an Air Force general – who, his previous tour of duty had actually been down at AFWIC, so he came to us from AFWIC. Right before the letter could be delivered, I came in on a Monday morning and I had a call – I got a call from my point of contact at the Department of Justice [01:10:00] saying help, our website was hacked over the weekend.
JACK: Oh, wow, the very website that Jeff was supposed to run a security assessment on had been hacked. But Jeff didn’t have all the approvals yet to help out the DOJ, so he just couldn’t do much. But Jeff asked for more details and what happened became actually pretty big news. On August 16th, 1996, a hacker broke into one of the DOJ’s websites and replaced the picture of Janet Reno who was the attorney general with a picture of Adolf Hitler. They changed the name of the website to Department of Injustice and replaced the seal with a Nazi flag. Lucky for them it was a somewhat benign defacement attack and didn’t go much further than that.
JEFF: It was the first hack of any government installation facility website. It was the first time the government had been publicly hacked, compromised. Everybody was paranoid about it, everybody was very reactionary; crap, we gotta do something. It became very public very quickly.
JACK: I find this ironic because just before this, Janet Reno approved Jeff’s team to pen test that website, and now her picture is what got defaced. But the DOJ still wanted the team in The Pit to come help and take a look.
JEFF: I said well, let me see what I can do. [MUSIC] Knowing that ordinarily to engage them is a three-month process, I hung up with them, I got on the phone with the general council’s office and I said this is what’s happened. We’re this close to being legal, engaging with them anyway. The last letter has been signed. It just hasn’t been delivered yet. What do I have to do to get a team onsite tomorrow to help them out? They talked about it and they got back to me and said well, have them make the request in writing. Get it written down. So, I ended up having them do that and they said don’t go on your own. Make sure that it’s a group and have your management send you. Don’t go on your own authority. Have somebody tell you go ahead and go, somebody in the management chain. I was like okay, that’s easy. So, I followed all those steps and I assembled a team. Three or four of us went down to the DOJ office in Washington, DC and we were there Tuesday, we were there Wednesday. Thursday morning we’re down there for the third day and I get a phone call from somebody that was still back at The Pit.
They said dude, the shit’s hit the fan. You guys gotta drop what you’re doing and come back now. I was like, okay. So, we did. So, it took us a couple hours to get back to the office. When we got back to the office, we were escorted into the executive conference room for the deputy director of InfoSec and waiting for us there was the same general council, the same lawyer that I had been working with for the last several months. He’s an Irish guy and he was mad. He was red in the face and he was reading us the Riot Act about how what we had done was illegal. Didn’t we know that it was illegal? Didn’t we know that we could not only get the director fired but possibly go to jail? Don’t you know that you could go to jail? For the first time in my life, I was introduced to what was known as the Church Proceedings. He asked us haven’t you ever heard of the Church Proceedings? Of course, no, I hadn’t heard of the Church Proceeding.
JACK: So, he had to learn that in 1975, there was a senate sub-committee led by Idaho Senator Frank Church to review whether any of the intelligence agencies had abused their powers and what it would look like if they did overreach and abuse their power.
JEFF: The essence of the findings was these organizations have a lot of power and a lot of capability and a lot of potential. But they don’t have much oversight officially. How do we know that they’re benevolent and gonna do all the things that they do to the bad guys and not US citizens? One of the outcomes of the Church Proceedings was what came to be known as the NSA Charter which is a classified document, but it essentially says that NSA can only do what NSA does to foreign nationals. Anybody other than US citizens and NSA may explicitly not do what NSA does to US citizens. Well, you can imagine that in terms of ethical hacking, white hat hacking, breaking into US – what’s effectively US systems and networks sort of flies in the face of the NSA Charter. Now, we had never really confronted that explicitly in [01:15:00] all the negotiations with the lawyers for the months or years that we were working with them to do our vulnerability and threat assessments, but they certainly had it in mind. It just came to a head when, for whatever reason, somebody decided that we had not followed the right procedures to go down and help the DOJ out with their forensic exercise. It was a big deal. Well, at the time we didn’t think it was a big deal. We thought it was overblown but because I was sort of the project leader, I was the one that was thrown under the bus. I was put on probation. My clearance was pulled.
JACK: The NSA did an investigation on Jeff and they called him back into the office for a chat with the director. New rules were laid down which the people in The Pit had to follow from then on. But this whole incident just took the wind out of the sails for the people in The Pit. Their energy and passion was sapped, including Jeff’s. At this point, Jeff was with the NSA for twelve years and he had built up quite a lot of skills there, even getting his Bachelor’s degree in Computer Science. So, he looked at the private sector for jobs and sure enough, jobs for him were available and paying a lot more. So, he quit the NSA shortly after this incident and after that, three more people from The Pit quit, too.
JEFF: Then I started a week later that I think initially was a 50% pay increase. From a strictly economics perspective, it wasn’t a difficult decision to make. But if things hadn’t have gone south like that – a lot of people asked me why do people work at NSA? It’s like, because they really are patriots and they really are loyalists and they really believe in the mission. I probably would have been in that boat, too. I probably would have stuck it out and stayed there and enjoyed whatever notoriety which certainly I wasn’t seeking, whatever professional career success, I would have stayed there.
JACK: [MUSIC] The following year, in 1997, the NSA launched Operation Eligible Receiver. This was a no-notice training attack that the NSA would simulate on the US government and military. They were actively conducting DDoS attacks and using open-source intelligence to figure out ways to infiltrate different military bases and networks. The NSA had built a red team and were hacking into the US government networks. I found an old video of a Navy captain who worked at the NSA and was part of this exercise.
CAPTAIN1: Planning for Eligible Receiver at the National Security Agency began in 1996. A small handful of people who were appropriately cleared into the program at that time began laying the groundwork for the IW campaign in support of JCS objectives.
JACK: Jeff quit the NSA in ‘96 and believes that guy was taking notes from the team in The Pit.
JEFF: I’m like, I remember when they used to visit us all the time and a very congenial fellow – and he always had a clipboard and he always was asking lots of questions and taking lots of notes. Putting two and two together, looking back on it, I’m like damnit, he was asking us questions because he was working on putting together Eligible Receiver. But we were not ever planned to be part of Eligible Receiver ‘cause they didn’t want to put the A-team out on the job. They were recruiting people and training people up to be lower-level hackers, what they referred to as the B-team, to actually execute the exercise. Yeah, so yes, I was involved. I didn’t know it at the time.
JACK: Eligible Receiver, this exercise that the NSA was doing to hack into the US government, wanted to use the B-team ‘cause they didn’t want the best, most elite hackers trying this. Those people were busy, anyway. They wanted a little less-sophisticated team to try this and all with off-the-shelf tools, nothing super-advanced.
CAPTAIN1: We were faced with a very interesting situation. That is, there was a no-notice exercise that had not even been announced yet that it was coming. Yet, we were required to do reconnaissance of both the MILNET and the SIPRNet ahead of time to be able to characterize our attack for approval. This required us to actually conduct reconnaissance in such a way that we looked as if we were real to the outside world. This was done with commercial internet service providers and it was from those providers that we touched military sites in the Navy and in the Air Force and so on in order to gain our information, to do our open-source research, to do our web-surfing on the internet and move off from there.
CAPTAIN2: How we went about doing the reconnaissance was we looked for access points, ways to get into the DII or the .mil domain, better known as [01:20:00] MILNET or NIPRNet. We needed to get in…
JACK: Chunks of this video are just redacted but it became clear that the US military wasn’t securing their networks as good as they should have been.
JEFF: They were allowing fourteen days. They had to call it off after like, two or three days because somebody on a naval vessel noticed something weird going on with the network and they pulled the alarm which started kicking in the whole Defcon escalation thing. They wanted to stop it before real shots were fired.
CAPTAIN1: The most important lesson that we learned on the red team, given how we approached the US as a target; on open-source alone, no insider information, is that we know quite clearly how to take the DII down and how to attack the United States in an information warfare campaign.
JACK: Wow, that is scary stuff. To think that a B-team of hackers with off-the-shelf tools using commercial gear and conducting open-source reconnaissance was able to successfully access so much stuff. Well, I’m glad this exercise was conducted to help secure the whole network but again, I feel like Jeff and his team in The Pit was who created the original red team at the NSA, a rag-tag group of six hackers all hopped up on Mountain Dew. It seems like if that team didn’t exist, then Operation Eligible Receiver may not have happened or would have happened years later. This also speaks to the importance of conducting red team assessments. If you need to protect important data or valuable assets in your network, it’s probably a good idea to hire an ethical hacker to see if they can get into your stuff. Hey, if it’s what the NSA has been doing since the 90s, it’s probably good enough for your company to do, too. It’s not impossible to defend against cyber-attacks. Often it’s just a couple misconfigurations that can easily be fixed. It’s good to run a self-check sometimes. I have one more conspiracy question at the end, here.
JEFF: I’ll try.
JACK: Bitcoin uses SHA-256 as its private public key mechanism thing. SHA-256 was made by the NSA. Does this mean the NSA has a backdoor into all Bitcoin wallets?
JEFF: Well, as a cryptographer, I happened to be NSA-trained. If I knew the answer, I couldn’t tell you. My opinion is all the descriptions of backdoors or all the conspiracy theories that I’ve heard about backdoors are essentially – depending on how you define backdoor, I don’t see having a master key is a backdoor but call it that if it’s what you will. I don’t know how you would do that with a hashing algorithm. I suppose it’s possible so I’m gonna say no, I don’t think so, is my – that’s my final answer.
(OUTRO): [OUTRO MUSIC] A big thank you to Marcus J. Carey and Jeff Man, two excellent people who work hard at giving back to the community and making us all better. I’ll have links to both of their stuff in the show notes, but you can check out Marcus’ book. The title is Tribe of Hackers. Again, that book has brought value to me by helping me find guests for this show, so thank you Marcus. You’ve helped make this show better in some ways. If you want to hear more stories from Jeff, tune into the podcast Paul’s Security Weekly which is a podcast that goes into security news every week. It’s a great show that has lots of really cool, amazing guests too, and I’ve enjoyed many episodes of it. If you want to hear more about the NSA, I’ve made quite a few other episodes about this, interviewing people from there, even.
Check out Episode 53 called Shadowbrokers, Episode 50 called Operation Glowing Symphony, or Episode 29, Stuxnet. Not many of you stick around this far into the episode. I watch the stats. I know how many of you have dropped off by now. But if you’re the type of person who’s still here with me, I can tell you really like this show and want more of it. The best way to help support the show is to donate to it through Patreon. This helps keep the mic powered up and the .wav files flowing. Please consider donating at patreon.com/darknetdiaries. Thank you. This show is made by me, the irate monk, Jack Rhysider. Editing help this episode by cottonmouth Damienne, and our theme music is by the howler monkey Breakmaster Cylinder. Even though I don’t back up my data because I know the NSA does it for me, this is Darknet Diaries.
[OUTRO MUSIC ENDS]
[END OF RECORDING]