Transcription performed by LeahTranscribesJACK: Hey, it’s Jack, host of the show. For a while, I worked at a big company doing security engineering, and every year, someone would come in and do an audit on us, and they would ask us the same question; do you have a security policy? Yes, of course we do. Is it available for all your employees to find? Yep. It’s right there on SharePoint. But this got me thinking — yeah, sure, it was right there in SharePoint, but it was called something ridiculous like isp_overview or something like that. ISP stood for information security policy. It made me wonder; if this document was so important that we would be audited to check to see if we had it and make sure all our employees had access to it, could any of them actually find it if they needed it? This policy said stuff like what are our security objectives? Who are the people that we escalate things to?
What’s acceptable in our network and not? Who should be able to access what, as well as what we should do when there’s an incident, how often our security training should be, and what our security standards are. So, one day when I was feeling feisty, I decided to do something to make a point. I asked everyone on shift in our network operations center — hey, you have fifteen minutes to find the company’s security policy. Winner gets a free item in the vending machine. Go. Everyone started looking. First they typed ‘security policy’ in our department’s portal. That actually brought up security policies for some of our customers, which I thought was really cool that our customers were taking their security policy so seriously that they wanted to make sure that their partners had copies of it.
But that wasn’t our policy. Then people started looking through their e-mails. Nope, nothing in our e-mail about security policy. Then they looked at shared drives. They couldn’t find anything there, and eventually a few of them thought to look through SharePoint. Of course, not a single one of them could find it because it had the worst name and it was in the worst place. I don’t know if you’ve ever used SharePoint, but it’s a place to store documentation and files, and it’s an awful mess to navigate and find stuff. None of their searches came close to finding it. So, I just said, alright everyone, time’s up. Thanks for trying. Then I sent an e-mail to our CISO, our Chief Information Security Officer.
Security policy test queue one: Ten out of ten of our NOC technicians could not find our company’s security policy after spending fifteen minutes trying. He responded, um, sounds like your NOC technicians have a hard time finding things? I waited another four months. We got a whole new batch of technicians, and I try it again. One guy actually found it. I was really impressed. I also re-tested all the people that I tested four months ago. One in five remembered where I told them it was. So, I sent another e-mail. Nine out of ten of our new hires could not find our security policy. Four out of five of our senior technicians could not find it. He was like, why do you keep telling me this? Just show them where it is. I wanted him to understand the problem wasn’t my technicians.
It was that the security policy was buried way too deep. It was named poorly, and nobody knew where it was. Nobody could find it if they tried, which meant nobody knew what was in it. In my opinion, when there’s a document that’s so important that auditors ask if you have it and if it’s available for employees to find, then it should be way more front and center. Heck, I even suggested that we should print out a summary of it and tape it above the urinals and sinks in the bathroom so that everyone sees it every time they go to the bathroom. That way the whole company would be familiar with our security policy and know exactly what to do when there’s an incident and what’s allowed and not allowed. But of course, our security leadership didn’t see it that way and never did change the name of it or the location, and we kept passing our audits somehow. Yet, nobody in the company ever read it or knew where it was. Ah, the politics of office life and compliance.
(INTRO): [INTRO MUSIC] These are true stories from the dark side of the internet. I’m Jack Rhysider. This is Darknet Diaries. [INTRO MUSIC ENDS]
JACK: Today I have the pleasure of sitting down and hearing stories from Tanya Janca.
TANYA: Thank you for having me.
JACK: I’ve been going to a lot of conferences, and time and time again I see Tanya at almost all of them. But not only is she there, but she’s almost always giving talks when she’s there. She’s on a mission and is very driven.
TANYA: I hope software developers write more secure code.
JACK: Boiled down to a word, that’s called appsec, application security. She’s laser-focused on how applications become insecure and how to make them secure.
TANYA: I was a software developer forever, and then someone exploited one of my apps and showed me, and it created this fascination. It was an SQL injection and it was on the log-in screen of one of my team’s apps, and I was in charge of the team. So, if it’s not secure, it’s my fault. I remember he was giving a demonstration to us, and he showed me — he’s like, this is one of your apps. I’m gonna get past this log-in screen without a password, and the only reason it’s gonna take so long is because I’m talking, and it’s gonna be a minute.
JACK: [Music] He demonstrated how he can easily get past her log-in screen, and showed her how it was done. She was stunned.
TANYA: Like, oh my god.
JACK: This was the moment that she saw the whole world differently. Just because there’s a right way to use a website by putting in a username where it says ‘username’ and a password where it says ‘password’ doesn’t mean people actually play by those rules and follow the website’s logic. If you’re clever enough to think outside the box, you can manipulate the website to do things that the developer didn’t intend.
TANYA: We ended up becoming very close friends, and he became my first professional mentor for hacking.
JACK: This was a career pivot. Instead of building things, she wanted to know how to break things. All this is happening in Canada, by the way. Tanya was living and working in the capital city of Ottawa. So, her new mentor was like, okay, so you want to be a hacker. I’ve got some work for you. You can help me do some penetration test. She’s like, okay, but I’m not exactly sure how.
TANYA: So, he told me on the Friday, okay, so, go learn Burp Suite this weekend. There’s videos on YouTube. Just go watch them. It’s not hard.
JACK: [Music] So, she starts watching them. Burp Suite is a tool used to monitor the packets that go between your computer and an application or a network or a website so you can redirect all your computer’s traffic to it, and then Burp Suite will show you, hey, you went to this website and it responded with this code, and then your computer sent this information back, and then this website sent back a cookie which has this data in it. It’s kinda like getting under the hood of a car but for network traffic. Burp Suite is really cool; you can capture that data and replay it if you want. Like, maybe you look in the cookie that a website sent you, and it says that your user ID is 5000.
So, what if you change that user ID to 5001 and then reconnect to the site and present this cookie which has a different user ID? Will it think that you are a different user? It’s kind of like a way to do surgery on the packets that your computer is sending to an application or website, and it’s possible to manipulate your packets enough to make the site do some very strange things. So, she comes back in on Monday with the basics of Burp Suite understood, and he tells her, okay, great. Spend a few hours a day trying to hack this website, and tell me what you find. He says…
TANYA: I’m gonna be observing you silently while you’re working, and I just need you to report anything that you find. I was like, great. So, the first night I just found really tiny things, the second night I found really tiny things, and the third night he kinda gave me a lecture and he’s like, listen, Tanya, you’ve gotta find something. You can’t do a pen test and not find something big. I need you to really think outside the box. Take off your developer hat. Put on your black hat. So, I just tried everything I could think of, and I found server-side request forgery. Now, I didn’t understand server-side request…
JACK: How’d you find it?
TANYA: Basically, there was an e-mail field and I just started entering in code. I entered in an e-mail but then after, I just put in everything I could think of, like all sorts of code, all sorts of stuff. Inadvertently, I started copying and deleting files on the web server. [Music] Yeah, and I ended up crashing the production web server, and it turned out I had polluted the database as well. They had to restore both from backup. So, I called my boss. I’m like, I found something and I have crashed everything. He’s like, what? This is production. You can’t crash production. I’m like, well, I found an exploit and you told me to prove that I had exploited it, so here’s — you know, I took the whole thing down. He’s just — he yelled at me and was really angry. He’s like, how did you do it? I’m like, these are all the commands I did. He’s like, well, this is garbage.
This shouldn’t do anything. I’m like, well, I guess it did, right? I was like, well, weren’t you watching me the whole time? He’s like, no, that’s just something I said to make you feel better. I was like, well, what is the client gonna say? He’s like — I’m like, I could talk to them. He’s like, no, they don’t know that you’re testing. I’m like, but you said I’m a subcontractor, right? So, the contract I signed with him that I was subcontracting that they knew. They had no idea I was on their network. So, he had given me the keys to production to some random client. They had no idea I was on there. I destroy everything, and he’s like, great, now I’m gonna get yelled at tomorrow and it’s all your fault. I’m like, really? This is all — he was really pissed at me.
JACK: I think they both learned a lesson that day, but Tanya was hooked more than before. To copy and delete files on a web server all by putting in some code through a form field? Wow, such power. But wow, such weakness. These apps she was seeing are surprisingly weak, and she was drawn to that. What are other tricks and techniques for making an app give you data that it’s not supposed to do or do things it shouldn’t let you do? She got more and more into security, wanting to get more hands-on with everything related to it.
TANYA: I kept annoying the security team constantly. I would report security incidents. I would fix all the security bugs. I kept asking if I could use security tools. I volunteered to work on their projects. One day, they said that I could sit in on an incident and just watch and shut up. Literally, I did not have a seat at the table. They had only so many seats that would actually fit at the table, so I was actually against the wall at the back of the room just being — zipping it, being quiet like they told me to. Then I remember them putting all of this stuff on the screen and looking at it and being like, oh, that’s SQL.
Oh, that’s pretty bad. So, I said to them, I am seeing code here. We need to look at this. Can we talk? They’re like, you can read that? Just because someone’s trying to SQL-inject you doesn’t mean they’re successful, right? But I’m like, someone’s attacking us. Every organization’s getting attacked all day, all the time, but the fact that I could sit there and read code — they were like, oh, she’s an asset. Then a few weeks later they said, oh, we’ve opened a job on the security team. I was like, oh my gosh, I’m gonna apply. They’re like, obviously it’s for you, silly. So, they let me transfer onto the team, and I was just — I was so excited to be a part of their team.
JACK: With this new position she proved herself again and again and rose up the ladder, eventually landing in a security leadership position at an organization which was within the Canadian government. She was in charge of making sure that agency and all its apps were secure. One day she came to work and had an e-mail waiting for her.
TANYA: [Music] I receive an e-mail from Vice Magazine and it says, dear Tanya, we know you work at this place and that you’re the leader of this team. We would like a quote for you for our magazine about how your data is for sale on the dark web and how you feel about — that your data is worth only $48 Canadian dollars. Here’s a link if you want to see more. It’s a link to Pastebin, and on it it says, here’s a sample of the data from the name of my organization, and to get more, go here. Then I go there, and they’re auctioning my data for the Bitcoin equivalent of approximately $48 Canadian dollars, which is not a lot of money.
JACK: What a way to be notified that your agency has suffered a data breach, by getting an e-mail asking you for a quote on how you feel about your data being for sale.
TANYA: So, I talked to my team. I’m like, aah! All of us are just flabbergasted. We were like, first of all, what is this data? Is this actually our data? So, we’re looking through all of our apps, and this is when I realized my app inventory was not complete. We were missing lots and lots of apps that I did not know about that I am supposed to be securing.
JACK: Step one of a data breach like this is to verify that it’s your data. Find out which app or database it’s from, and this will help you identify maybe which app is vulnerable. But it took them a while to figure out even where this data was in their network. Eventually they narrow it down and figure out which app this must have come from.
TANYA: So, I go — I find it — that data is in there. The Pastebin sample does look a little familiar. I’m like, oh no. So, I go and I talk to my boss, and my boss was so pissed. He was like — have you ever had someone say your name in a way where it sounds like a swear word? They were just like, Tanya! I am not really good when people are upset with me, and so, I was like, well, sir, I — you know, our data is for sale on the dark web; that’s true, but I think the bigger problem is that it’s only $48. Don’t you feel we’re worth more than that? He was not impressed. Somehow my name was at a higher pitch the next time he said it, and he’s like, you are gonna go fix this now!
JACK: Yikes, there’s a leak. Data from the Canadian government is getting leaked, and Tanya is head of security for this department that it got leaked out of. This is really bad. She pinpointed the application, though, and got the owners of that application together.
TANYA: I sat down with the team, and they’re like, we have no idea what you’re talking about. I showed them and they said, yep, that’s our data. I said, okay. One of my team members said, I think we should buy the data to make sure it’s an exact match and it’s not like — ‘cause they were just showing two or three records on Pastebin. They weren’t showing all of it. I was like, well, I don’t want to give them money ‘cause I feel like that’s encouraging them, and it feels pretty obvious that if they got three records, why couldn’t they get more records? So, then we look at the data, and it’s completely unclassified. It was actually data, it turns out, that we had been trying to promote to the Canadian public for quite a while and had been being mostly ignored. Some journalists would look at it for a media piece or something, but generally no one was paying attention to a thing we were hoping that they would. So, we’re like, maybe this will help. My boss also did not find that funny.
JACK: Okay, phew, this wasn’t as bad as it seemed. A lot of data within government is, in fact, unclassified and publicly available, and it seemed like the hacker stole some publicly-available data, and nothing sensitive was actually taken or sold. But what do you do here? If a hacker stole data that’s publicly available, is it actually stealing? Is there any action even due here? What’s the big deal, right?
TANYA: Yes, this data was public in general, but they had the record ID number, and that’s not public, right? So, someone clearly got a copy of our full data set as opposed to just what we wanted to show the public, and every single record except the ID identifier that we used to look it up was considered unclassified. So, none of it was sensitive in nature.
JACK: Hm, I see. Every line of the database has a unique ID, which aren’t important or even sensitive information. However, it’s not public information. It’s only used for how the database sees the data. So, whoever got this had full, readable access to their database. So, it was time to drudge through the logs to try to find out what happened.
TANYA: [Music] We had only database logs. We had no web-app logs. The app itself did not log at all. It was really old. So, we looked through all the database logs, and very quickly I figured out what the attacker was doing was, one, attacking us on every single statutory holiday. So, we in the government, we get paid time and a half if we work overtime. But if you work a statutory holiday, you get two and a half times your regular pay, and it’s policy that unless it’s an emergency, you are never booked for on-call or anything like that on those days. So, we would never work those days. So, this person, for a year every single statutory holiday, would start hacking us basically at midnight all the way until the next day. So, I started looking through the logs. Well, basically, first I looked at the most recent logs, and then I was like, have I ever seen these commands before?
JACK: She did recognize what was happening, at least kind of. She recognized that the commands in the database were trying to do SQL injection. SQL is the type of database used, and injection is where you try to put your own database commands in through the web form field. So like, when you go to log in on a website, you put your username and password in, right? Well, the website will grab that information and then go check the SQL database to see if your username exists. Which, an SQL statement might look like ‘select from table where user equals Jack’, like, if that’s my username, right? Since ‘Jack’ is what the user typed in, then that’s actually what gets queried in the database. Well, what an SQL injection does is it messes with that.
Since you’re going to take whatever username I type in and search the database for that, what if I type in the username Jack but then also write something else, like Jack or select from table all the passwords? So, now if it’s vulnerable to this, it’ll take that input and go do this database command, select from table where user equals Jack or select from table all passwords. If it’s vulnerable, it might return all the passwords that you asked for. You see how adding extra commands into a form field can trick it to return extra stuff the developers didn’t want you to do? To fix this, developers of the apps need to sanitize their apps, not let users put in extra stuff like that, and really restrict what’s allowed to be typed in those form fields. So, Tanya recognized this was SQL injection by looking at the database logs, but it didn’t quite make sense. It wasn’t your classic SQL injection.
TANYA: But this one was doing truth and falses, and I was very confused. So, they were looking for this person, and they’re always ‘true’. Then it would say ‘and’ instead of ‘or’, which I was not used to. It would say, ‘and the name of this table, the first letter is A’. I was like, what? And? Why do you need ‘and’? I’m very confused here. So, we — I started running them, and almost all of them were false. It would just return an error. There was no one named Jack. I was like, but there is a person named Jack. Then I’m like, but that’s because they did the ‘and’. You have to have both of them be true. I’m like, this is so weird. So, I kept going through, and finally one was true. I’m like, well, what the heck is this? I look and it’s a letter from one of the names of the fields of that table. I’m confused. I’m like, why would you look up an A? It was like, is this A, is this B, is this C. I was super confused.
JACK: Well, they fixed that app so that it wasn’t vulnerable to SQL injection anymore, but they were still perplexed on how those commands worked, how they got data with those commands.
TANYA: We spent two or three months looking at it, and no matter what we did, we couldn’t figure out how they got the data. There were just errors or returning this one record.
JACK: She never did figure it out. She ended up leaving that organization still not understanding how that data got out with those commands.
TANYA: So, then I went to Defcon and I did a workshop called Blind SQL Injection. I was super excited to finally make it into a workshop, ‘cause I don’t know if you know, Jack, but there are long lines, and there’s a lot of competition to get those seats, and I made it. So, here I am at the back of the class, and the teacher is explaining, oh, well, what you’re doing when you do blind SQL injection is you are asking questions, and the questions you are asking is either the names of fields and the database, the names of tables, what’s inside a field. So, it’s like, oh, this record exists? Great. Is there a field called this? No. Is there a field called that? Oh, there is? Is the first letter…? ‘Cause you can’t say ‘return that record’. It won’t do it with blind SQL injection.
JACK: Yeah, so, the only option you get back is yes or no.
TANYA: Yeah.
JACK: So, you can ask the database any question, but they’re not gonna give you data. They’re just gonna tell you yes or no.
TANYA: Exactly. So, if it’s an error, it’s a no, and if you receive the record that you have searched for every time, it’s true. So, I went to this workshop, and it’s like this giant light bulb went up for me, and I was like, oh my gosh. So, I call my old boss and I’m like, I know what happened. He’s like, have you been poking around since you left? I’m like, no, I went and I took a workshop and I learned and I know exactly what happened. So, I went back and we had a meeting in our special, secret room and — that wasn’t very secret. Anyway, and we had a meeting and I — and it’s funny because I walked them through the logic of ‘this is how you ask the database questions and this is how you can know for sure that it’s true’. So, I explained this, and then all of them except the really, really big boss — the really, really big boss was like, I still don’t get it. But everyone else is nodding, so that’s fine. So, they did exfiltrate our data, and that is what happened, and — okay, so, now we know.
JACK: We’re gonna take a short ad break here, but stay with us because Tanya is gonna tell us more stories about the fires that she’s extinguished. Tanya had a lot of roles in different companies and organizations over time, and at one point she was leader of incident responders. You know, if there’s a severe security problem in the network, it would be her and her team that would manage the problem. She would identify the problem, engage with the right people, and get working on it and tell leadership what’s happening, and then stay on the incident in order to make sure it gets the resources it needs to get resolved.
TANYA: So, I was the lead of the incident responders. So, we had a guy that did malware analysis, all of those things. So, I was the appsec expert ‘cause — not surprising, right? So, I would always do the software incidents. I came into work late one day because I had a dentist appointment. I had told my boss. I had told my team where I was. It was in my calendar. Anyone could see. I come in at maybe 10:00 a.m. Basically, there were two of us that managed incidents, me and this amazing person named Eric. [Music] I come in, and all my team’s sitting there, including the Eric that is the incident manager. I’m like, hey guys, what’s up?
They all look really tense, and they’re like, there’s a really big incident and everyone’s in the really big boardroom. I’m like, but Eric’s sitting there and I’m standing here. So, who’s managing the incident? They’re like, some guy named Dan from help desk. I’ve changed Dan’s name because that is what you do. I was like, what? They’re like, yeah, they wouldn’t let us in the room. I was like, what is happening? They’re like, we need you to go in there. They won’t listen to us. So, I go in, and I open the door and they’re like, Tanya, where have you been?
I’m like, at the dentist. No cavities. No one thought that was funny. They were like, we needed you and you weren’t there. Everyone stared at me. I’m looking, and there is the director of every department, a bunch of managers, and all of the executives from our organization in this room. So, this is an extremely expensive meeting. Everyone looks really stressed and upset. There’s — so, this was a while ago, so there was that big, huge thing in the middle of the table that was the phone with the giant buttons…
JACK: The UFO.
TANYA: …and it sounds terrible and — yeah, it’s one of those. There’s this guy on the phone named Dan from help desk. They’re like, we’re having this huge incident and you weren’t here, and we needed you. But Dan’s helping us, so we don’t need you, and you can go. I’m like, I’m not going anywhere. I’m the head of incident response. I’m the incident manager that is on duty now, and I’m doing the thing. I’m like, I’ve got this, Dan. He’s like, oh no, I have it. I’m handling it.
JACK: She’s like, who the heck is this Dan guy? Dan was from help desk, which is often the front line for office workers when they have problems, right? If your computer stops working or the internet is out or you’re locked out of your computer or your password doesn’t work, who are you gonna call? Help desk, and that’s where Dan was working, and he was answering a lot of phone calls that day. He just kept getting call after call from that office. People were saying nothing is working. Managers were in a panic. They can’t do their work. People were getting so upset in that office.
TANYA: I found out later we had people go home because they’d had at least one panic attack. Just, several people were just too nervous and upset that they actually went home for the day ‘cause they just felt very uncomfortable and unsafe.
JACK: Just, call after call was coming in to the help desk, and Dan was answering these calls, and he was doing his best to solve the issues.
TANYA: I’m like, okay, so, what is happening? So, I’m standing there in HQ, our headquarter’s office. We have a satellite office that’s maybe twenty kilometers away, and I am informed that our satellite office is infected with malware. I said, oh, someone has malware. No worry; we’ll go mop it up. We’ll be right there. They’re like, no, no, the building has malware. I’m like, the building’s dumb. It can’t have malware. I laugh, and then someone says, don’t call them dumb. They’re nice. No, no, the people aren’t dumb. The building’s dumb. They’re like, don’t call them dumb. Okay, the building’s not smart. That didn’t go well, either. I’m like, so, a smart refrigerator is internet connected. It’s not internet connected. It’s cement. Cement does not get malware. They’re like, Dan knows and you don’t. You weren’t even here. You were busy at the dentist. I got so much flack about the dentist you would not believe.
But anyway, so, everyone’s very upset. I try to calm them down. I’m like, listen, my team will look into this. Dan’s like, we should evacuate. They’re in danger. He’s ramping them up, so they are panicking. I’m like, Dan, that’s not true. Everything’s fine. Let my team look at this. Finally I get everyone — I wouldn’t say settled. I would say that they were less panicky. I’m like, everyone, go back to your desk. I am gonna update you in half an hour. I am gonna find out what is happening. Everything’s gonna be okay. They were like, someone needs to go to the dentist instead of helping us. Literally, people were so upset with me. They were furious. So, I dismiss everyone. I hang up on Dan, ‘cause Dan’s not helping. He said over and over again, the building has malware. We should evacuate. I was like, no one’s evacuating.
So, I go back to my desk and I’m like, someone flip on Wireshark. [Music] He’s claiming the entire building has malware. We all know that’s not true. They all respond, but the building’s dumb. I’m like, I know. I know, guys. We all know. Dan has whipped everyone into a frenzy. We need to do something about this now. So, we flip it on. So, there are some stereotypes about Canadians, and some of them are true. Like, they take our passport if we’re rude, we all eat poutine. There’s many, many stereotypes, and one of the stereotypes is that we love the Winter Olympics. We love watching hockey. We love watching the figure skating. As an entire nation, we tune in. We really like it. So, when we turned on Wireshark, we immediately saw every single person in the entire building was going to the exact same site, and the figure skating for the Olympics was on, and Canada was skating.
JACK: Huh. So, there is no malware. The reason why nothing was working is everyone is live-streaming the Olympics, and that takes up a ton of bandwidth. So, the work that those office workers were supposed to be doing, they couldn’t do it because the network was basically clogged up, bogged down. They essentially did a DDoS attack on themselves. The funny thing was, they had a policy in place that should prevent things like this from happening.
TANYA: We have a policy in the government, or we did at the time, where when the Olympics happened, we knew Canadians are gonna Canadian, and so, we would make a boardroom in one building, and that was where the Olympics are showing. So, if you need to go see your guy win his thing, you go and you watch the skating and the twirling and whatever it is you’re gonna do. No one’s allowed to stream it because if every single person is streaming, there is no internet. So, we block that and make many Canadians cry, and we found out later that some executive had decided, oh, you’re gonna take a vacation day if you want to watch the Olympics.
Like, you’re here to work, blah, blah, blah, and had gone against policy thinking they were super smart, and this is what had happened, right? So, I call a meeting on the next hour, and I’m already sending e-mails explaining to everyone there is no malware, there was never any malware, everything’s fine. So, I call everyone into the room. I’m like, hi, everyone. Everything’s fine. Everything’s cleaned up. There is no problem. There was no malware. They’re like, but when are we gonna clean up the malware? I’m like, there never was any. Everyone was just watching the Olympics.
The internet slowed down. Everything is fine. It was actually always safe. We do not need to panic. I need you to all go calm your staff, especially the satellite building staff. Tell everyone everything’s fine. They were always fine. We just were too busy streaming and not busy enough working. Everyone seemed not super satisfied with that answer, but enough, right? So, everyone left. But going forward, people talked about how that building had had malware for six months. Like, I couldn’t squash the rumor. It didn’t matter how many times I corrected people. They were like, yeah, she doesn’t believe it. She doesn’t know. I’m like, I’m the incident manager.
JACK: [Music] So, after it was all fixed and resolved, it was time to pay a visit to the help desk to help them identify and handle incidents better.
TANYA: So, help desk wants to help, right? People that are really good at help desk, they love literally helping and solving problems. So, they are the first line of everything, right? Like, you call help desk. First of all, you try — you fiddle around yourself, you try to fix it, and if not, you go to them. I go to them, right, if I can’t fix it myself, which — it happens. So, this person received this call and they’re like, I know what I’ll do; I will solve this problem for them. ‘Cause that person — ‘cause — I know ‘cause I was working at that org — had never had any training about what a security incident looks like. So, what my team did to solve this problem going forward is we had help desk in and we gave them a training on what security incidents look like, and we told them, we will never, ever, ever get angry if you call us and it’s a false alarm. I’d rather twenty false alarms than one where you didn’t call and we made a mess.
JACK: So, her and one of her incident managers named Eric gave some training to them, and Eric had a doozy of a story himself to share with the help desk team. So, at Eric’s last job, he was an incident handler. If there was a security incident, it would go across his desk. One day, someone from the IT help desk discovered a problem. They were given a computer to fix something on, and when they were looking through the computer for problems, the help desk technician discovered sexually-explicit images of children.
TANYA: [Music] He understandably was extraordinarily upset.
JACK: Yeah, I mean, of course. Seeing images like that, you can’t un-see it. It feels like you did something wrong just by taking a look. Well, this IT help desk technician was like, well, that’s wrong. The employee shouldn’t have this on their computer.
TANYA: He deleted the images, and then he was still upset, and he formatted the drive.
JACK: Which actually makes sense. When people who work in IT help desk see problems, it’s usually on them to fix it. Virus on computer? Clean it off. Apps installed that are against company policy? Delete them. Apps missing which should be there? Install them. Software out of date? Update it. Help desk people are action-oriented. They take control and fix things. All day, every day, they’re fixing things. So, for him to delete these photos seemed like the right thing for him to do.
TANYA: So, he calls incident response.
JACK: He’s like, man, I was just fixing a problem on some employee’s computer and I found sexually-explicit images of children. This feels like something I should report to you. Eric, the incident response manager, is like, okay, wow. Thanks for telling me. How bad is it? Real bad. Okay, well, let’s be careful here. Can you show me what you found?
TANYA: Essentially what happened is the entire chain of custody, the evidence, was ruined.
JACK: Because the help desk technician deleted all the evidence and didn’t take any screenshots — I mean, how could you take screenshots? — and then he reformatted the hard drive, there was zero proof that what he saw was actually there. So, there was nothing for the incident manager to evaluate. But they did report it to HR.
TANYA: They were able to fire that person for violating the acceptable use policy of the computer.
JACK: But HR was like, hold on, this is actually more than an acceptable use violation. This is illegal. We should report them to the police. So, they did, but then the police are like, okay, show us the evidence. But they had nothing to provide. There were traces of backups and archives that they could have dug into, but it didn’t matter because the chain of custody was broken. So, they had nothing admissible to give.
TANYA: So, they were unable to prosecute that person.
JACK: Man, what a blunder by help desk there, huh?
TANYA: The poor help desk guy, he feels incredible guilt, and the person from help desk ended up in therapy for a long time.
JACK: Why?
TANYA: Well, probably for two reasons. So, one is he felt incredible guilt because he did not know better. So, he did what help desk does, which is usually erase, reformat, reimage. So, he did what his training told him to do, right? But meanwhile, he saw things he can’t un-see, and he also unintentionally let a very bad criminal go free. So, when I give training on that topic and I talk to help desk, I’m like, I know you want to help. I know you want to help. That’s why you’re so good at what you do. But if you see anything that you think looks criminal, you need to call us right away.
If you see anything where you’re like, this just makes absolutely no sense, I need you to call us right away. Like, if all your normal steps to fix something don’t work, please call us and we will come in, because we have different tools than you have. So, we started this annual training that me and Eric would give where it was just like, these are the things that we need you to know. The training would just be like, twenty minutes. It was just very basic. Like, if you see this, call us. We will never, ever be angry.
JACK: Now, Tanya has been to a lot of conferences. It’s a great way to learn and meet amazing people. But one really cool thing you often see at conferences are CTFs, which stands for Capture the Flag. It’s a game where you can form teams and then try to hack into something. Like, there’s a computer that’s intentionally vulnerable, and if you can hack into it, you’ll see a flag, and if you can get that flag, you’ll get points. The team with the most points wins the CTF challenge.
TANYA: I did do a few CTFs. I went to a bunch the first year, year and a half when I was trying to become a pen tester because I heard they’re a great way to learn, and I did learn lots of things. I also learned that I was always the only female everywhere I went. Everywhere, I’m the only woman, and I was a little tired of that. So, I put a note on LinkedIn and said, hey, do any women want to form a CTF team with me? ‘Cause I don’t want to be the only woman everywhere I go.
JACK: Where was this gonna be?
TANYA: It was gonna be in Ottawa. I form — I ended up having so many women say yes, we had to form two teams, which was really exciting. [Music] I was pretty surprised. All of them said the same thing; I was curious to go, but I felt like I didn’t know enough, and I’m always the only woman and it’s weird. So, a bunch of us wore party dresses, which was really fun. So, I was showing them, okay, so, here’s this log-in screen, and we’re supposed to try to get past the log-in screen. I’m like, I know how to do this. I’m sure there’s gonna be some sort of SQL injection opportunity. So, I was walking them through it the way that my mentor had walked me through it, and I showed it to them and we got in.
JACK: Tanya was able to use SQL injection to bypass the log-in screen. Basically, when you type in the username and password, the website sends the data to the database, and if they are a match, the database returns true. If they aren’t, it returns false. Well, she put in the username field something that will always return as true. Like, is there a user named Tanya or does one equal one? Because there’s an ‘or’ statement there and one equals one is true, the database returns true no matter what the username is. So, since the database returned true, she logged in without providing a valid password. Her teammates were amazed at how she did it, and asked her to explain.
TANYA: Yep. Then two of us got up and did happy dances, and a third one got up and she’s like, hi, I have to go. We’re like, where are you going? She’s like, I have to go to work right now, because I am not sure that we are safe from this, and I need to go test every app I’ve ever built and make sure that it is okay, and I have to go right now. She literally went to work and spent — apparently she was there quite late, ‘cause she came to the CTF quite bleary-eyed the next morning. I was like, oh, how did it go? She’s like, we’re fine now. I’m like, now? She said she had fixed a whole bunch of things. She’s like, what’s the next thing I’m gonna learn to fix? Let’s do this.
JACK: So, in the middle of the CTF she learned she was vulnerable and ran out of there.
TANYA: I think she suspected — I don’t know if she knew for sure, but she was like, I am shocked. I am — she just ran out of there.
JACK: So, professionally, Tanya has two passions; application development, which is coding, and cybersecurity, hacking. So, over time, she simply found her favorite place to be was at the intersection of these two things. She’s given talks and written frameworks on how app developers can write secure apps, which is known as secure coding or application security. Okay, so, application security is yelling at devs.
TANYA: [Laughs]
JACK: Why do you laugh? Why do you laugh?
TANYA: It should be helping devs. It should be helping devs make more secure code and being nice to them ideally most of the time or all the time, in my opinion. So, I was in charge of pen testing and doing — running and launching their first appsec program. So, there was five developer teams. I was asking to be able to pen test their apps before they went to prod. I was hoping that they would scan their apps with ZAP for me first.
JACK: ZAP?
TANYA: Yeah. So, ZAP is a dynamic scanning tool that used to be part of OWASP, and it’s the most-used dynamic scanner on the planet. Basically, I wanted the developers to scan the app first. I had made a grid. So, I’m like, if you find this, fix it. If you find that, just ignore it.
JACK: But the manager of that development team did not want his developers to do any of this.
TANYA: One of the teams — their manager told me, leave my devs alone. We don’t have time for your crap. [Music] I was pretty new to appsec. It was only my second job in appsec, and he felt I was inexperienced and that, in his words, I was a pain in his ass. I was like, I’m here to help. He’s like, then go away. That would help. I was like, listen, I need to take a look at your apps for security. He’s like, they’re fine. Just trust me. I was like, well, I’d like to talk. He’s like, I don’t have time. Each time I kept trying to approach him, he was more aggressive until the last time I talked to him, he literally said, go fuck yourself. Get the fuck out of my face, and was pointing at my face and pointing away, and then he just started yelling at me. So, I left.
JACK: Rude. Rude. But this is why I don’t want to be a manager. Managers take on too much stress; directives from higher-ups and deadlines with not enough resources to get it done, and their team always having problems, too. They can’t always be transparent about things, either, like how much their budget is or plans for upcoming layoffs. If their manager has a bad day, then that rubs off on them, and that means that manager’s team has a bad day, too, or someone like Tanya gets yelled at for no reason.
TANYA: My boss was like, I know what we’re gonna do. We’re gonna hold a meeting and we’re gonna tell them about a whole bunch of security incidents, and we’re gonna deputize them and tell them not to tell anyone. So, don’t worry. I’m like, I’m very worried. He’s like, and it’s gonna be fine. Then they’ll listen. I was like, this is a terrible idea. [Music] So, he invited them in and he explained what it’s like when a computer gets malware. He’s like, and then this guy on the team, he does the malware analysis and he does this, and you lose all your local files that you should have had. He’s like, so, this is why we don’t stick USB keys in our computers. They were like, okay.
Then our worst incident recently — there was this app and there was an SQL injection in it, and they managed to exfiltrate a whole bunch of our sensitive data. We had to report ourselves to the privacy commissioner. We ended up having to — ‘cause they attacked the SQL server itself, we ended up having to send that server away for analysis. He’s like, we had to do this, this, and that, and we ended up spending all these weeks of overtime on it. He’s like, it ended up costing over half a million dollars. They were like, oh my gosh. He’s like, yeah, we could hire five engineers for that. They were like, oh my gosh. Wow, what a giant screw-up. He’s like, that was your app. That was an app that Tanya asked in writing, and she came up and asked you personally, if she could test it, and you said no.
She has been bugging you for six months and you have not let her test a single one of your apps. Tanya can’t do this job by herself. She needs you. She needs your help so bad. She keeps asking for it, and you keep — you told her to F off. Dude, that’s rude. We need you guys. We can’t do it without you. Please, please, please help us. Let us test stuff. Let us tell you when things are wrong. Work with us, please. The manager was like, oh my god, I’m so sorry. I had no idea. He’s like, dude, we spend so much on appsec. We have her full-time; that costs money. But he’s like, there’s the tools she has to buy. There’s the time it takes. There’s when there’s an incident app and it’s a mess. We can’t do this without you. You guys are so much more important than you realize as this piece of the puzzle.
He’s like, I need you to let her test, and I need you to fix things if she says they’re serious. Please? The guy said, yes. Then everyone chatted a lot, and then when everyone walked out, the manager that had been so unfriendly with me, he came up to me and he put his hand on my shoulder and he’s like, Tanya, I had no idea how serious this was. I am sorry. This will never happen again on my watch. We are gonna be number one. You tell us everything. We’re gonna fix everything. Our apps are gonna be bulletproof. This is over. And he did it. He would fix all the things. He had them open up their old apps that weren’t even on my list and he had them scanning it with ZAP and fixing things.
His team — the next lunch and learn I had, they were all sitting there right at the front eating the bagels, ‘cause I bribe people with carbs. All of them were there, the whole team, right at the front. We’re ready, Tanya. I was just like, oh my gosh, this is so amazing. I thought by hiding — it sounds dumb in retrospect, but I was like, if we show them we’ve made mistakes, they’re not gonna trust us anymore. They’re gonna think we’re stupid and we’re bad at our jobs. We can’t let them know we’re having lots of incidents all the time. They’ll think we’re failures. But in fact, that made sympathy and empathy, and then it was like a completely different workplace then.
(Outro): [Outro music] Thank you to Tanya Janca for coming on the show and sharing these stories with us. She’s written two books, Alice and Bob Learn Application Security, and Alice and Bob Learn Secure Coding. She also has a newsletter and would love it if you joined. You can find the newsletter at newsletter.shehackspurple.ca. It’s totally free, but it’s crammed full of great, helpful information on how to make your apps more secure. It’s holiday time, and you know what your loved ones would love most? A Darknet Diaries t-shirt.
If they don’t want something like that, then you tell them to get you one. By the way, these shirts don’t all say ‘Darknet Diaries’ on them. Most of them are just really cool designs that I came up with. You have to check it out. Go to shop.darknetdiaries.com. This show is created by me, the spaghetti coder, Jack Rhysider. Our editor is the copy pasta coder, Tristan Ledger. Mixing done by Proximity Sound, and our intro music is by the mysterious Breakmaster Cylinder. One day, I hope to change the world, but I don’t have access to the source code. This is Darknet Diaries.
[END OF RECORDING]