Transcription performed by LeahTranscribes[FULL TRANSCRIPT]
TOM: I was having trouble sleeping one evening. I had gone to bed and then I woke up, so I went downstairs to just futz around for a little bit. I turned on my computer and I was looking at my e-mail and here was a message from my bank, B of A, saying that my account had dropped below $25. [MUSIC] At the time, it didn’t trigger anything in me because I only – I knew I had about thirty-something dollars in it and it was a account that I used to keep money for equipment materials, that type of thing, for my business. But anyway, so I thought of that and just eh, okay, fine. Then I went back to sleep and when I finally got up in the morning, all of a sudden I’m sitting there making my coffee and I’m going well, why did my account go down below $25? I haven’t used that account in a couple of weeks.
JACK: This is Tom. He’s just found out that somebody’s used his credit card without his approval.
TOM: Obviously things are shot so I immediately called the bank and I said I don’t know what’s going on but I got a notice from B of A that I was overdrawn. Their fraud department said okay, fine, and they started a deal and immediately notified me that my accounts were frozen and that I couldn’t do anything. This was kind of a frustrating thing. I’m sitting there saying to myself now, now how in the hell could that happen?
JACK (INTRO): [INTRO MUSIC] These are true stories from the dark side of the internet. I’m Jack Rhysider. This is Darknet Diaries. [INTRO MUSIC ENDS]
JACK: This is the story about a time when a major retail outlet got hacked. I’m not going to give the name of the store or even when this occurred because those details aren’t important. This story is fascinating enough without it. This company is huge so they have thousands of stores in the US and many more all across Europe and Asia. They do business both online and with physical stores all over the world. Of course, each of their physical stores have computers that are connected to the network. This story starts out with an e-mail.
One of their credit card brands had found some cards on the black market being sold and they were linked to this store. The card brand e-mailed the store, letting them know some cards that are on the black market have a common purchase point of these retail stores. Specifically they found ten credit cards on the black market whose last purchase was this store. Now, ten credit cards found on the black market is not really that big of a deal, especially when credit card dumps have tens of thousands of cards in them. But this store wanted to investigate anyways so they called a consulting firm called Kroll.
KROLL: Hello?
COMPANY: Hey.
JACK: And they asked Kroll to investigate their network and try to find if there was any traces of malware on it. They got a team to help them out. Let’s meet two of the members of that team.
COURTNEY: My name’s Courtney Dayter. I’m a Senior Managing Consultant.
MATT: My name is Matt Bromiley.
JACK: Both Courtney and Matt are incident responders. Their job is to go into a network that is breached or may have been breached and find, isolate, and fix the problems. Both of them are used to working on larger cases and specifically cases involving finance and retail. Doing an incident response for a global retail company is what they’re good at. These two got right to work looking for any signs of hackers in the network.
MATT: We usually start with [00:05:00] two different approaches. Number one, we’ll start to understand what’s already been out there. What type of data has been leaked and is it something that could actually be resulting from a breach? That’s kind of number one. Number two is we’ll also come in and drop in whatever tech we use. The tool that we use primarily is Carbon Black which is an end-point monitoring tool, end-point monitoring and analysis tool.
We use Carbon Black when we deployed that throughout the environment. It’s usually that two-step approach; is to quantify the data that’s been exposed, drop in our tech and we start to get deployment throughout the entire environment. This particular customer has over ten thousand systems globally. They’re acting in every major country that’s out there. They’ve got outlets in every single one, if you will. We were deployed on a global scale to almost every system that they owned that was running Windows and that could handle the agent’s software. Basically everything Windows XP and above.
JACK: The computers at each of these stores had antivirus running but it wasn’t picking anything up. The end-point monitoring software wasn’t finding a lot, either. But the store’s IT team had noticed one system was acting funny. So they took it offline and asked Matt and Courtney to look into it.
COURTNEY: Originally the company had identified that first system for us. It was actually already identified; they had pulled it offline so that was somewhere where we were able to look.
MATT: This particular system, it looked like a bloody murder scene. When you just take a look at this system, it’s just stuff everywhere. There’s malware all over the place. We also found several directories full of nonsensical binary data files that just didn’t have much of anything inside of them but there was size to them. On Windows environments, very rarely do you come across just blobs of data that don’t have any structure or purpose or are in the wrong place like these files were. We came across thousands of them. Every single file, believe it or not, also had the same naming convention. It was a six-digit date, so month, day, year, followed by a host name, followed by a seemingly random string of digits.
JACK: These weird and unusual files were encrypted so neither the company or the team could see what was in them.
COURTNEY: Based off of some of the information that we found in different logs, we were able to basically track the different IP addresses where that account that was owned was also reaching out to. From doing that we were able to see where it was reaching out and where they were pushing malware to from that machine, as well. Then in doing so, we also found additional malware as we kept going out. It was just more and more specific variations of the same malware. From there, we were able to get most of it and then we got to a second major host machine and we were able to then spawn out from that one as well. Then we’re back to the third one.
MATT: We identified upwards of about twelve hundred compromised systems. [MUSIC] The malware sits on these systems, scrapes all this data out of memory, and then it pushes that data over the wire to the central repository system.
COURTNEY: This particular malware was writing to an output file so we knew that this malware was writing to a particular output file with a particular file extension. We were going, looking across the network for all of the files with the same similar naming patterns and similar file extensions. We also looked at the different malwares ‘cause some of them have a different naming convention in each of the different output files. We made sure we were able to determine all the output files. We pulled them all together.
JACK: At this point the team had found over one thousand computers all in the network with the same malware. But here’s the worst part; want to guess what these computers were used for?
MATT: They’re the cash registers where you go buy something and you swipe your card, or you insert your card. These days you insert pin and chip, sometimes you still swipe depending, but the systems that are compromised are those particular systems.
JACK: Malware that exists on twelve hundred computers that are cash registers is terrifying. The team began studying this malware to try to understand what it’s doing. The output files were encrypted using XOR and the company couldn’t decipher it. But Matt and Courtney spent some time and eventually cracked the encryption. They were able to see what was in those files. What they saw confirmed their fears; the malware was grabbing every credit card that was swiped on that register and putting it in this file.
COURTNEY: [00:10:00] The way that they were scraping cards were – they were scraping from memory. But they did it using fifteen and sixteen-digit characters, so they were pulling anything that was fifteen to sixteen digits and usually it’s followed by something along the lines of an equal sign or some sort of delimiter and then it gives you the four digit expiration number. Based off of that, that’s where they were able to pull card data down from memory, and using that basic algorithm they were able to get mostly positives but occasionally you do get a few false positives and some of the things that are scraped.
JACK: This company is now in a nightmare scenario. So many of their cash registers have malware on it that’s scraping every credit card that’s being processed and then sending that data out of the network. As the credit card data leaks out of the company and into the wrong hands, the criminals who get these cards can use them for themselves. They do things like write the card data to a blank credit card and then withdraw cash from ATMs or use these cards to buy gift cards and sort of launder the money. The thieves found a spring of seemingly endless money and they were making some serious cash off this retail company.
MATT: Our mutual feeling was there’s gonna be a lot of data here. I don’t know, the initial thing that you think about – and it’s like an investigator’s curse. You never want this number to be high but one of the first things you think about his how much data has been stolen? How much data has actually been accessed? ‘Cause when you’ve got that much malware in an environment that big, the first thing you have to think is I’ve just found thousands of output files; what are the chances that I’m also looking at millions of credit card numbers? You never want to be in that target world. Right, those tens of millions of numbers and that kind of stuff.
You never want to be there but unfortunately when you uncover a breach this big, one of the things that creeps into your mind is please, please, please, don’t be that large. But wait, that was North America. We found the exact same type of infrastructure in Europe as well as in Asia Pacific. We had these central pivot points that the attackers were using basically as clearing houses. Malware was directly writing over the wire right to these systems as they were going thousands of systems at a time, simultaneously as well.
JACK: Like I said before, the registers all had antivirus installed on it.
COURTNEY: But it wasn’t picking up this malware in particular.
MATT: This is actually a new variant that we’ve uncovered. It was not previously known. There’s a couple of little surprises about this malware. First off, it was an unknown variant but it’s a derivative of Tiny POS. There’s a legitimate Tiny POS point-of-sales software, then there’s a Tiny POS piece of malware. Tiny POS is piece of malware that is pretty – I don’t want to say sophisticated but for what it’s able to do, it’s pretty well-developed. I say that knowing I just complimented a malware family but it’s real good put together but the kicker here is every single piece of card-scraping malware that we came across in this case was less than 6 KB in size.
COURTNEY: As we kept adding more and more stores it was definitely a point where we’re like uh, are we going to have millions of cards here? I think especially as we saw the output files go higher and higher in number I was like oh, well let’s hope some of these overlap. Yeah, definitely an oh, shit moment.
MATT: But wait, but wait, hold on, it gets worse than that. About eighty percent of the systems was the point of sales, the point of sale registers. The other twenty percent was the back of house systems as well, the back of house systems that sit in the back of the store that no one has access to aside from the store themselves. But those systems had malware on them as well. Those systems were also – also had data that was being scraped in them, too.
JACK: Delivering this kind of news to the client is never easy. The team had to call the client to tell them what they found.
MATT: Yeah, I would say it was us, them, and lawyers. That was a fun chat. I guess to give you a very brief perspective, there’s always that moment of like, when you receive that external alert or when your client receives that external alert, there’s always that moment of is this bullshit? Or how real is this? When we had that call we said hey, FYI, we have uncovered this malware. We found what it’s able to do. We’ve uncovered all these output files. The very first reaction because the output files [00:15:00] were encoded, they actually had a custom encoding that the malware was using. Because they were encoded there was a little bit of well, that data is just garbage, right? There’s nothing in there. Or like eh, well, we actually decoded it.
There was definitely a little bit of oh shit on their part as well because it becomes real once you find out that your greatest fear has come true, which is someone’s been hanging out in your house for a while. [MUSIC] The lawyers, they do a very good job of letting us get everything out and then they process it and then they come back and ask questions. But there’s always a little bit of hesitancy. They want to make sure you’re correct. If you come out of the gate and you say yeah, we have evidence of a credit card breach, there’s a lot of wheels that start to turn. You’re on a clock, depending on the state. Some states you’re on the clock then, about disclosure and that kind of stuff.
You’ve got to file for protection with the credit card brands and everything like that. They ask as many questions as possible to make sure you’re one hundred percent positive. What you’ve seen backs up – technically you can back up what it is you’re saying because they know that there’s a lot of money about to be spent based on that finding, based on that opinion. So next, this is where Courtney and I then fall into the quantification mode where now, okay, we’ve got a breach. Now we need to understand how wide, how big, and how much data is at risk here. That’s kind of the next step that you want to answer, is how much data is actually at risk when you have one of these breaches here, and just how far back does it go?
JACK: Doing some digital forensics, the team was able to see when the malware was originally installed and where it came from. From this investigation they found the hackers were in the network for…
MATT: About seven to eight months they were in there.
COURTNEY: They were in the network probably at least a solid month of just reconnaissance before they built their perfect piece of malware. Then even throughout time we saw them make slight modifications to it as they kept going forward. They knew exactly where to go get the credit cards on every system.
MATT: Unfortunately the system that they first compromised, that they first came into was no longer available. I think personally, they came in with a phish only because there was very little exploitation of systems even though there was a very vulnerable environment. There was very little exploitation of those vulnerabilities and that may be again, because they didn’t need to do that. I don’t want to rule that out immediately but I don’t know, most of these cases I always see start with some sort of a phish. Especially to set up the infrastructure that these guys had – that these attackers had set up. Here, it makes me want to think it was likely a targeted approach but you never know. You never know until you find it.
JACK: Phishing is when a hacker targets an employee to try to get them to click something they shouldn’t click on. It could be an e-mail with a malicious link, it could be a Word document with macros enabled. Once the person clicks the malicious link, that computer can become infected and then under control of the hacker. When the hacker is in the network, they can move over to another machine to start setting up their malware.
MATT: With simple [inaudible] math you’re scraping six hundred stores, five to six hundred stores for a period of eight months. That period of time includes the summer, includes multiple sale weekends, it includes the build-up to Christmas and that kind of stuff. Including all those various time periods. You could very easily get into the hundreds of thousands or millions of cards easily with those considerations and those factors. [MUSIC] The next step is let’s get these output files parsed, let’s start to incabinate this data together and let’s start to de-dupe it and see just how much data we’ve got exposed here.
COURTNEY: Yeah, I want to say it took us at least two weeks to get through all of the credit cards and really de-dupe them and make sure that everything we had were actual card numbers. Something particular in this case that we ran into was credit cards that look like credit card numbers but weren’t actually valid card numbers. That was something that we had to do a lot of de-duplication and verification with, both on our end and with a little bit of help of the card brands, determine if what we were seeing were all actually card numbers.
MATT: We started uncovering card data that was expired. [MUSIC] [00:20:00] We’re like, how are we finding so much credit card data that’s expired? It’s one thing if you find – let’s say you have twenty numbers from the day and you find one is expired. You’re like oh, okay, someone accidentally swiped an old card or something, right? But then you start to wonder, you’re like why am I seeing this significant percentage of cards that have already expired? What was happening in this case is, if you remember earlier, I mentioned that the malware was on the back of house systems.
The back of house systems were running SQL servers. The SQL servers had historical unencrypted tracked data that was being loaded into memory and the malware was picking that up. They were picking up transactions from as long as four years ago. The attackers were effectively peeking back in time. They were looking at transactions from three to four years ago that they had no visibility to, which is another unique angle because most of this malware exists at the swipe, or it exists to steal at the swipe.
JACK: Now the team is ready to begin removing the malware from the network. They needed to understand every hole that was in the network and patch every one so that the hackers could not get back in.
COURTNEY: We didn’t actually have to take any of the servers offline. Once we were able to really find those pivot points, taking those offline or at least making sure that we had process blocking in place, was able to stop it for the most part across the network.
MATT: Yeah, we ended up shutting down the clearing houses, the central points that they were using. We ended up taking those off first and then waiting to see what would happen next. The first time we kicked them out, we actually saw them re-enter through Asia and within about three seconds of re-entering they had re-compromised forty different systems. [MUSIC] I never want a company to actually be breached but part of my job is to find that stuff and part of Courtney’s job is to find that stuff. Our initial reaction was first off like, ah shit, we knew this was gonna happen. We didn’t know it was gonna happen this fast. That’s reaction one.
Reaction two; usually what happens if you successfully kick an actor out one hundred percent, usually you’ll see a phishing campaign or something. They’re trying to get back in. To see them come through the network with that speed, the next thing is ah, shit, there’s another back door out there which we got to go track down. There ended up being a system we didn’t have visibility to. Number three, there’s that moment where you see how quickly the attacker’s doing what they’re doing. That speaks volumes to how long they’ve been in the network. If you’re watching an attacker, if you’re seeing artifacts from an attacker very recently and it looks like they’re fumbling around in the dark looking for a light switch, then you’re like, there’s a very slim, good chance this person hasn’t been here that long.
To watch someone re-compromise four dozen machines in ten seconds, okay, this person has come back home, they’ve put their feet up on the couch, they know exactly where the remote is and it’s a very easy thing for them to slide right back into.
COURTNEY: I think it was interesting to see them come back in so quick but it was also interesting to see which tools they were immediately using. Because at that point we had the live response there. We were able, essentially, to sit there and track what they were doing and see exactly how they were moving. We had some of these records before but it was just nice to actually sit there and be able to confirm oh, they came in through here, okay, well they ran an IP scanner and then they hard-coded and were able to log into all these IPs in the matter of three minutes.
JACK: The team had discovered that besides the malware, there were also back doors installed on many systems which is how the hackers kept getting back in.
COURTNEY: They had added 350 back doors. However, they weren’t pushing the malware to each, like all 350 systems didn’t have all the malware on it. It was almost as if they were just allowing themselves access back in incase they ever got closed off.
JACK: But the team was able to find [00:25:00] each and every back door and take every pivot point offline and stop any more credit cards from leaving the network. It was a good feeling to finally get this malware under control.
MATT: There was about a month where every call got worse and worse. There was definitely a moment of really, is this ever going to end? You can definitely get bad news [inaudible] after a while but eventually once we started taking things offline, it then turned into positivity and we were able to actually deliver good news calls. Which is hey, we’ve actually remediated things. The attitude started to shift when – once we had figured out the way the global – the attacker’s global infrastructure was set up.
But it’s only when you get to that point where you’ve mapped out the whole world that you can start to actually breathe a little bit. Luckily this malware was not ingrained to the point where it became symbiotic with the environment like some malware families do. But this one was pretty easy to delete. I don’t say that as a challenge; I say it as – it was definitely a pain in the rear but it was simple enough to delete and then disable the services to prevent it from running again.
JACK: As the team was cleaning the malware off the network they gave some suggestions to the company to improve their security.
COURTNEY: Yeah, some password changes were necessary, maybe something down the line of changing their network infrastructure so it’s not so flat. That was definitely one of the things that enabled this malware to get as far as it did because every system could essentially reach every other system. There were similar passwords shared, administrator accounts, privileged accounts that were accessible on many, many machines across the network. I think this was a big learning point of how to properly secure your network and make sure your encryption’s in place to prevent this from happening again.
JACK: To try to trace this hack down to the person who was responsible is sometimes impossible. You can look for clues in the malware like the language that was used in writing it, or the time zone that it’s set to, but these things are just small clues that aren’t very strong. Trying to figure out who did a hack is called attribution.
MATT: I’m a firm believer that attribution doesn’t really get you anywhere unless you’re sitting in a political or an executive role and you’ve got to make decisions off of who may be behind this keyboard, and that kind of stuff. However it’s always interesting to know. The only thing I can say about this one is one thing we haven’t mentioned yet; North America, there was one server that was treated as a clearing house. Then there was two additional systems that had back doors on them. In Europe it was very much the same thing. Europe and [inaudible] and whatnot was very much the same way. There was one or two systems that served as central pivot points.
Asia on the other hand, Asia had somewhere in between 350 and 400 unique back doors installed on it. Almost every system got a back door. A lot of the compromise itself actually started in Asia. It actually started in mostly Southeast Asia and that kind of stuff. That doesn’t lend to any attribution whatsoever. It’s just when you’re going after credit card data, it’s a very interesting place to start, if you catch my drift. Even if you pinpoint it, that specifically, what are you going to do? You’re a company that’s headquartered in the United States. What recourse do you have? You’ve got to get your network back up. You’ve got to get to a point where you’re not having to fight fires every day. You don’t really have time to. What, are you gonna hire a team to gonna go after these guys or something like that? Good luck.
JACK: What was the final number of credit cards that were stolen?
MATT: That number, to the best of my knowledge, is still being sussed out but I think after everything we can had come across, we landed a little shy of 100,000. That was all. Which was a very surprising number and a very relieving number as well.
JACK: I don’t know a lot about the current carding black market conditions but it’s safe to say these are probably too many cards for these hackers to try to scrape money out of themselves. They’re probably selling these cards in bulk somewhere. The cards go anywhere from ten dollars to a hundred dollars each, so even if they got ten dollars per card, that means these hackers made a million dollars off this company. Now the company has to do it again to try to clean up the problem.
MATT: Primarily, then it falls onto the company to work with the banks and work with the credit card companies and get new cards out there.
JACK: This breach was publically announced and it hit the news, but the public’s reaction to it wasn’t a huge deal.
MATT: In short it was [00:30:00] not as crazy as you’d think. It was not that big of a deal. I say that because you’ve got predecessors like Home Depot and Target and some of those huge major breaches, you’ve got predecessors like that which received weeks if not months of news. This one was not as prolific as that. From that, and the world view.
COURTNEY: That and then on top of it, with the whole network infrastructure right now and how often we’re almost seeing these reported in the news. Some of the larger breaches that we’ve recently seen, including social security numbers, I guess credit cards kind of a little bit fall to the back. You’re always worried it gets stolen but in the back of a lot of people’s minds they’re like oh, I’ll just replace it, get a new one.
JACK: Besides this being a major headache for this company and even a bigger headache for the credit card companies and banks, this also can severely impact the people whose cards got stolen. At the beginning of this episode you started to hear from Tom. It’s possible that Tom’s card data was stolen and sold on the black market just like in the story you just heard. Someone used his card fraudulently and his bank was investigating to see what went wrong. Let’s hear how his story pans out.
TOM: Well, the morning that they did it was the 12th or 13th of December so this effectively wiped out Christmas. [MUSIC] I’m a licensed contractor and I receive some of my business through an outfit and with my accounts frozen and nothing able to go in or come out, the first thing I found was that they stopped working for me and they said well, your bill is overdue and your bill is overdrawn and we can’t get any money so you’re stopped until something happens.
But luckily I had a financial backup on this and so I was able to survive but I could not work until this was finally taken care of. It made a couple of months where things were very, very difficult. My main bank had gone through and they said okay, we have found the problem and they had now put everything back the way it was supposed to be. I was now able to do business with the account but even having done that, it still took a couple of months to get things squared away. It was a major interruption in my life.
JACK: Courtney and Matt gave a presentation at the Kaspersky SAS Summit earlier this year. In their talk they went into detail about this new strain of malware. They also reported this new strain of malware to the antivirus companies so it can be detected in the future. Matt has since moved from Kroll and is now working at Cylance and has most recently been accepted as a SANS instructor teaching digital forensics and incident response.
JACK (OUTRO): [OUTRO MUSIC] You’ve been listening to Darknet Diaries. For show notes and links, check out darknetdiaries.com. If you want more InfoSec podcasts, there’s one that does an episode almost every day. They do a daily wrap-up of the news and interview some really smart people. It’s called the Cyber Wire and I recommend it for your daily commute. A lot of you are asking how you can help with this show. Right now I’m just trying to grow the audience. It’s hard to get the word out, so you’d be a big help to me if you would tell others about this podcast. Think whose phone number you have of someone who might like this show and text them right now to tell them about it, or post about it on social media, or tell your co-workers. These kinds of things make me super excited to make more episodes. This show is made entirely by me, Jack Rhysider. Theme music for this show including this song is made by Breakmaster Cylinder. [OUTRO MUSIC ENDS]
Hey, one last thing. [MUSIC] I made something you might like. I made a random password generator. Yeah, it’s a website that creates some fresh, new, random passwords for you. Just in case you ever need to create a random password, I’ve got you covered. Oh, and there’s an extra feature, too. It has an API which allows you to use it in your own programs. Anyways, if you want to check out the site it’s called passwordwolf.com. That’s passwordwolf.com. See you there.
[MUSIC ENDS]
[END OF RECORDING]