Transcription performed by LeahTranscribes[START OF RECORDING]
JACK: You ever get fascinated with the cyber-crime supply chain? It’s never a solo hacker doing the whole thing; there’s a lot of layers to this onion. So, let’s say a hacker breaks into a place and steals a bunch of information from some company. Well, next he’ll typically want to sell that data to make some money and do it again, so now you’ve got to find a buyer. But before we even get to the buyer of stolen data, there’s sometimes brokers involved, people who have negotiated deals between hackers and buyers. So you might go to one of these brokers, offer a percentage for selling the database to someone. Now it’s on them to find someone. But when the broker finds a buyer, sometimes one side doesn’t trust the other so they bring in a trusted third party, an underground escrow agent if you will, who will wait for both the cash and the database and then make the trade. Okay, but then what does the buyer do with this database dump? Well, if it’s full of e-mail addresses, they might use it to send spam to people. But of course, the spammer isn’t selling anything themselves. They’re typically promoting someone else’s business; a porn website or a pharmacy. It’s just fascinating for me to think about that sometimes. It’s never about the data breach itself but what happens to that data after its stolen.
(INTRO): [INTRO MUSIC] These are true stories from the dark side of the internet. I’m Jack Rhysider. This is Darknet Diaries. [INTRO MUSIC ENDS]
JACK: I’m sure you all know what LinkedIn is, right? It’s the social network for professionals. You pretty much start your account by posting your resume of where you worked and what you did there. You can use the site to look for jobs and connect with other professionals in your field. It’s pretty popular in the US. In 2012, a person wanted to hack into LinkedIn and get as much user data as they could, but how are you going to get into the network of LinkedIn? This is a major Silicon Valley company made by some really skilled engineers and administrators. They would certainly be following all the latest best practices for securing a network by doing things like securing the front door to the network by putting a big firewall up to block all non-critical traffic from coming in and inspecting it for malicious activity. Then they’ll conduct security audits on all the internet-facing systems to make sure there’s no security holes.
Of course, they’ll be running state-of-the-art monitoring tools and antivirus tools to watch for any intrusions. They did all that; the front doors of LinkedIn’s network was airtight. So, the hacker would have to find another way in. [MUSIC] He knew that engineers at LinkedIn had access to the corporate network when they were remote. I mean, today it’s obvious that a lot of companies have remote employees but back in 2012 there were LinkedIn employees who had remote access into the network. That is, they didn’t have to be physically in the office in order to access the database or other critical systems. So, the hacker set out to figure how exactly do some engineers get remote access into the network? He concluded they must be getting in through a VPN. A VPN is a way to securely connect into a remote network. The traffic is encrypted from the edge of the corporate network all the way to the user’s computer, wherever they are in the world. That’s just it; if there’s a backdoor entrance for employees only, it would also mean that the hacker could try to get in through that.
So, the hacker starts looking on LinkedIn’s website for people who worked there; engineers, system administrators, anyone who might have access into that VPN. So, he looked around for a victim which by the way, this is the reason why I don’t like posting my information on LinkedIn, because you can easily search for all the people who work in a specific company and then figure out who the admins are there who are probably posting things like oh, I’m good at Cisco firewalls and Oracle databases, and they might even be posting what versions of Oracle they’re good at which is a clue to any hacker to know what to expect once they get in. But what this means is that it’s pretty easy to find a target and narrow your sights on them by just looking at who’s on LinkedIn. This hacker found a LinkedIn engineer who probably had remote VPN access as well as access to the database inside, and the hacker zeroed in on this guy. [MUSIC] The hacker saw this engineer’s LinkedIn profile and on his profile there was a URL to this engineer’s personal website. Basically, it was like this engineer’s name.com.
The hacker went to the website to check it out. It was just a basic About Me-type blog; it said hello, I’m a site reliability engineer at LinkedIn and here are my hobbies and things. The hacker poked around here for a bit but couldn’t find anything to exploit. But he looked at where the site was hosted and it was hosted on a residential IP address. Hm, it seemed like this LinkedIn site reliability engineer was [00:05:00] running a web server out of his house. This means there are open ports from the internet into his computers. The hacker thought well, hm, if I can get into this engineer’s home computer, this might give me a way into LinkedIn. So, he looked to see if there are any other websites also hosted at this IP address, and he found one called cockeyed.com. He browsed around here and this is a much bigger blog-type website. Cockeyed.com was a site ran by this engineer’s friend. He just hosted it for him. There’s videos of pranks and pictures and it’s basically a blog. But this site was built using PHP as the back-end technology.
The hacker started looking for ways to exploit this site. He found a way to upload files to the site and he uploaded a couple malicious PHP files. One was specifically called madnez.php. Now, if a hacker can upload their own PHP program to your website and get that program to execute, then the hacker can take over that computer that’s hosting the website, because when you go to view the file through a browser, it’s going to execute whatever code is in that PHP program, and you can configure it to give you remote access to that computer. That’s just what this file did. The hacker got shell access to the website cockeyed.com which was being hosted in the same place this LinkedIn engineer’s personal blog was hosted. Once he got into the web server, he started scanning for other IPs in the network and found one, an iMac computer. He found that this iMac had an open SSH port which allows people to connect to that iMac. So, he started trying to brute-force login to that iMac. For the username, he used a first initial and the last name of the LinkedIn engineer and he just started hammering away, trying thousands and thousands of passwords, looking for one that worked. This was all happening in February of 2012.
For days, this web server was attacking the iMac, all within this engineer’s house without the engineer knowing it. After a few days of trying thousands of passwords, one worked. [MUSIC] He got a hit for a valid username and password, so the hacker logged into that iMac with this and looked around. First he realized this is a person’s personal iMac. It’s not a LinkedIn computer or even an official work computer. Then he discovered that this web server that he got into was running on this iMac. Yeah, it was actually running on a virtual machine in the iMac and I find this fascinating because in essence, the virtual machine was the only thing exposed to the internet, but the hacker got into the virtual machine and then got into the host computer from there. It’s fascinating to me because this shouldn’t happen, but the way he did it was through the virtual IP interface. Because I always wonder how it’s possible to escape out of a virtual machine and onto the host computer, and here’s an example of when a hacker did that.
After looking around the iMac for a while, the hacker stumbled upon the keys to the kingdom. Literally; he found a private key to LinkedIn. This was the key that the engineer could use to log into LinkedIn with. See, when you log into certain systems, you could use a username and a password but another option is to use public and private keys, where the public key is put on the server you need to access and your private key is on your own computer so when you connect to the server, you authenticate using the keys. This is all done automatically and saves you from having to type passwords. But when the hacker saw the private key, he snagged it right away. But where does this private key connect to? Well, the hacker had to look around a little bit more to find out, and that’s when he found a set of VPN profiles that allowed this person’s iMac to connect to LinkedIn. The profile contained everything they needed to connect; the server name, the IP address, the username.
The only thing missing was the private key which the hacker just got. [MUSIC] The hacker then took this VPN profile and credentials and connected directly into LinkedIn’s VPN server. Now, here’s where the hacker made a major mistake; [00:10:00] he made this connection into LinkedIn from his home which was in Moscow, Russia. LinkedIn is in California. Well, there was nothing to stop this connection coming in from Moscow though, so he just got right in. From here, he was able to find his way around the network, looking for the user database, and he found it. He was able to log into that and grab the username, password hash, and e-mail addresses of as many LinkedIn users as he could. After that, he logged out and disappeared. LinkedIn had just been breached but they didn’t know it yet and they wouldn’t find out for another three months. The first moment when LinkedIn learned about this was through a forum called insidepro.com. This was an underground criminal forum where you could buy and sell stolen data from hacks. Someone was offering LinkedIn user data for sale there.
The team at LinkedIn saw this and immediately sprang into action. [MUSIC] First, they needed to verify that the data being sold online was real LinkedIn user data. They compared what was in that sample database with their own database and sure enough, the hashes matched. The horror and the fear you get when confirming that you’ve just been breached, it’s indescribable. Now, LinkedIn’s response for something like this is a four-step process. First you confirm, contain, remediate, and do post-mortem. They just confirmed that they were breached. Next is for them to contain the problem. Is the hacker still in the network? What did they steal? How did they get in? Can we block them from getting in again? All these questions needed immediate answers. LinkedIn engineers and security team took over a conference room and called it a War Room. Something like forty to sixty people from LinkedIn were all working on this incident. They were flying in from foreign countries to help. They had the security team involved hunting through events and logs, looking for evidence. They had SREs, or site reliability engineers, and they’re combing through their systems looking for traces of unauthorized activity. There were lawyers present.
Their Chief Internet Security Officer was present and active, doing all kinds of triage. Other executives were in the room too because this was the most important thing going on at LinkedIn at the time. The atmosphere was heavy and intense. The first clue they got was from the VPN logs. The LinkedIn security team saw that one of their California-based engineers had logged into the VPN from Moscow numerous times. So, they called him in for an interview. Hey, have you been to Russia lately? No. Did you use any kind of Russian proxy lately? No. Did you give anyone in Russia your login details? No. The security team was on the trail. This was a major clue. They found out that this engineer had been connecting to the corporate network from his home iMac and I’m gonna guess that probably wasn’t allowed. But the security team asked him to bring that iMac in for an examination. So, let’s back up a month, actually. So, a month before LinkedIn even knows this happened, the hacker was looking through the database that he stole. It contained e-mail addresses, usernames, and password hashes.
This isn’t the password itself; it’s a representation of the password after it goes through an algorithm. So, in other words, you couldn’t see anyone’s password in this dump. But the hacker wanted to find a way to crack these passwords, so he posted a few of the hashes to a forum asking for help on how to crack them. When LinkedIn was investigating this, they saw that old post which matched the hashes that were in their database. The situation was getting worse for LinkedIn. They’re now seeing that the hacker is actively trying to crack users’ hashes. Unfortunately for LinkedIn, they weren’t yet salting their password hashes. Salting password hashes is an extra step that you do to make cracking hashes even harder. They were in the process of doing this but when you have hundreds of millions of users, it’s not easy to get it done. [MUSIC] At LinkedIn, they have different designations for how serious an incident is. Code Yellow is something that is some kind of technical risk like a server running over capacity or they’re not sure how to scale it properly, or a degradation of service that isn’t causing a whole outage but could at any moment.
Code Yellows happen every few months or so there but as the LinkedIn team investigated this, they determined this incident was a Code Red, meaning it was business-impacting and because they were already seeing user data leaked to the internet, it meant this threat was certain and it could be at any moment that the LinkedIn database dump was revealed to the world. I believe at the time it was only available to whoever would buy it and it wasn’t freely available for anyone to look at. This creates a tense and scary moment for any security team; not knowing [00:15:00] what or how much got stolen and not knowing what the thieves plan on doing with it. There’s a level of anxiety here especially because this was already hitting the news media who was announcing this hack to the world. Lots of different teams were pulling logs and saving them for incident responders to comb through. It’s best to have logging turned on so you can sort of go back in time and see what happened where. One problem though is that there’s now a lot of logs to go through and if you think about the millions of users who are on the site every day, trying to find a needle in a haystack is tricky.
I think the day they discovered this or the day after is when LinkedIn called the FBI to inform them of this breach. The FBI was very responsive. They asked for logs and started interviewing people right away. [MUSIC] LinkedIn saw that IPs were connecting in from Moscow, and so they took that IP and started tracing it through the network. Where did it go? What did it access? They were looking through SSH logs, Wiki logs, server access logs, and they saw connections from that IP which told them what user agent the hacker’s computer had. The user agent can tell you things like operating system, browser type, and version. The hacker’s user agent was unique. It actually had the word Sputnik in the end which isn’t normal. Sputnik is the name of the first satellite to be put in orbit by the Russians and this user agent just wasn’t seen by anyone before that which makes me wonder if the hacker put it there as sort of a signature. With this extra information, engineers can now search logs for that particular user agent to see if that has any hits for it, because maybe the hacker wasn’t using the same IP every time.
Maybe they had come in from different IPs and different channels and different VPNs or something, but if there was a matching user agent, then you could know that that’s probably the same person. Next, they looked at the public website to LinkedIn to see if anyone with a matching IP or a matching user agent was logged into any user’s account on linkedin.com. Sure enough, there was some activity. The same IP and user agent was seen logging into thirty different LinkedIn accounts through the public website. This meant that the hacker had cracked some passwords that was in the data and was logging into LinkedIn with those users. This certainly elevated the concern for LinkedIn. Back in Moscow, the hacker did in fact have a rather beefy GPU farm. See, cracking password hashes is process-intensive. You have to cycle through millions of passwords, hash them, and see if those hashes match the hash in the database. If so, you’ve found a password. Graphics cards, GPUs, are particularly good at doing a lot of these little calculations like this. They can do a lot of simultaneous things at once. So, the hacker was running the database dump through this password-cracking station he had, and when he’d get a match on someone that he found interesting, he’d try logging into LinkedIn to verify it, and it did in fact work.
He was logging into linkedin.com as different users. Back at LinkedIn, engineers started checking the database servers. They took the information they discovered and searched the logs to see if anyone had logged into the database servers with these IPs, username, or user agent. The database LinkedIn used at the time was Oracle which was on a UNIX machine. So, they looked to see if anyone connected to it using SSH and sure enough, they did. There were logs of the hacker logging into the database server and then accessing the database and running queries in that database. To get this far into the investigation took LinkedIn six weeks. [MUSIC] I’m talking an active War Room, Code Red situation for six weeks solid with multiple teams, dozens of people looking through thousands of servers, combing through millions of logs. It just takes a lot of time. During this time, they forced LinkedIn employees to change their passwords. They created a whole new account for the engineer who initially got hacked with that iMac computer, and they rebuilt servers to make sure there were no traces left on the system. Also, it appears that LinkedIn announced this breach as soon as they could to the public to inform their users that something bad has happened.
HOST1: LinkedIn users beware; the business social network says some of its users’ passwords have been stolen and leaked onto the internet.
HOST2: A hacking group rocked online network LinkedIn this week by publishing almost six and a half million user passwords to the site.
JACK: 6.5 million LinkedIn user accounts were claimed to be on some hacker forums. However, not all 6.5 million passwords were visible and the dump seemed to only be in the hands of a few people at the time, with just a sample of it posted publicly. Now, the posting and the investigation all happened in June 2012, but my research shows the hacker got into the network back in March of that year, a whole three months earlier which means LinkedIn had no idea of this breach until it showed up on this public forum. [00:20:00] So, the hacker did this hack back in March 2012 but in May of 2012, before LinkedIn even knew they were breached, he hacked into another website too and I’m not exactly sure what steps he did here, but I’ve got high confidence of how it went. So, by May of 2012, the hacker had already cracked quite a few hashes in the database that he stole from LinkedIn and was testing some of these logins by logging into LinkedIn with those usernames, and it was working. My theory is that he went through the cracked passwords looking for anyone that worked for a big IT company as an engineer or admin, because this hacker was in the business of selling massive databases to make money. So, this was his thing.
Looking through his cracked passwords, he found a quality assurance engineer who worked at Dropbox. He tested that the password worked by logging into this Dropbox engineer’s LinkedIn account. Yep, he got in no problem. Now, this Dropbox engineer has stated on the record that yes, in fact, he was reusing passwords at the time. His Twitter, Facebook, LinkedIn, and Google accounts all had the same password, so this hacker actually gained access to a ton of this engineer’s accounts. [MUSIC] But did this engineer use the same password where he worked too, at Dropbox? I don’t know for sure but the hacker was able to login as the engineer at Dropbox. I mean, he was able to get into the corporate network with this login either through a VPN or an admin web portal; I’m not sure. Obviously I shouldn’t need to tell you this, but it’s not a good idea to reuse passwords for this exact reason. Now, this quality assurance engineer didn’t have access to files that users stored on their Dropbox accounts but he did have access to users’ metadata, information about their accounts and stuff, because he would sometimes need to look into issues that users were facing.
The thing is, if a hacker has this guy’s login information, then the hacker can access anything this engineer can. So, the hacker got in and grabbed all the user data he could that this engineer had access to which consisted of usernames, e-mail addresses, hashed and salted passwords. Then the hacker transferred this to himself and got out. A month later, the FBI was investigating this LinkedIn breach. One of the things they looked for was to see if that IP address or user agent was logging into any LinkedIn accounts as users of the site. Sure enough, there were logs indicating that the same person had logged into about thirty different LinkedIn user accounts, meaning the hacker had cracked the usernames and passwords for these LinkedIn users and was testing it to verify it worked. One of those accounts belonged to a Dropbox employee and I don’t know if it was the same quality assurance engineer but the FBI saw that connection and thought maybe Dropbox might be the next target. So, the FBI called Dropbox to tell them this information and even gave them IPs and user agents to look for.
Dropbox looked through their logs and confirmed that yeah, those IPs did connect in as a quality assurance engineer and got into the corporate network. [MUSIC] Once Dropbox did confirm there was unauthorized access into the network, they immediately set up a War Room to handle the incident. This is basically like Command Central, a place where all data can be combined and up-to-date information is relayed to. Now, at the time in 2012, Dropbox had just under 150 employees working there and just like at LinkedIn, this was the biggest thing going on at Dropbox at the time, so it was practically an all-hands-on-deck response. Dropbox had over twenty people working on this incident but they knew they needed even more help, so they started hiring more security incident responders to just come on in and help. They first discovered that someone had unauthorized access into the corporate network of Dropbox. This seemed contained though, as the connections didn’t seem to make it into their production portion where dropbox.com was ran out of.
This was in the corporate side. While this is still a big deal to have someone lurking around in your corporate network, they weren’t able to see the crown jewels of what data users were storing in their Dropbox accounts. Now, specifically they were seeing that a Dropbox engineer was connecting into the Dropbox network from Russia, then once they connected, they went to the internal Dropbox Wiki which has information on how to troubleshoot certain things and other technical details about Dropbox’s network. The Dropbox team kept examining the logs. They saw which Dropbox engineer had his username and password stolen and went through the logs to see if there was any other suspicious activity around that. This engineer had an account at dropbox.com, so they looked at his recent activity and it shows that he invited another Dropbox user to join his Dropbox team. The thing is, the Dropbox engineer did not invite [00:25:00] that user, so this means the hacker got into the engineer’s Dropbox account and then invited himself to see that engineer’s files. After they had their accounts linked up, the Dropbox engineer’s account transferred some large files to the hacker’s Dropbox.
When Dropbox looked into what these files were, it was a list of twenty million Dropbox user details; e-mail, username, and salted password hashes. This made Dropbox aware that not only were they breached but the hacker stole at least twenty million user details from their customers. But they still weren’t sure how the hacker got these or even if it was from Dropbox at all. The next victim here is a company called Formspring. This is a social networking site which is focused on asking questions. Think of it like a place that’s dedicated to ask-me-anything-type interviews. In 2012 they had 30 million registered users and in June of 2012, the hacker got one of the admin usernames and passwords from Formspring’s server and logged in using SSH. My guess is that he got this login from the LinkedIn data too, but I’m not sure on that. He also logged into one of the web admin panels using the same username. He was able to install a malicious program called madnez.php on the server so that he could get back in anytime he wanted. It’s the same madnez.php that was found on that iMac. He found their internal Wiki and did a search there for hashed passwords.
I guess what he was looking for was information about them or where they’re stored or something. Using the web admin panel, he was able to run SQL commands on the database and grabbed a large amount of user info, specifically e-mails, usernames, and salted password hashes, then he logged out. That was in June. [MUSIC] On July 9th, someone posted a database dump of Formspring users on some underground forum. It contained 420,000 accounts. Someone saw this and contacted a journalist. The e-mail addresses in the dump contained the word Formspring in them a lot, like user+formspring@gmail.com, that kind of stuff. So, the journalist called up Formspring to get some answers. Formspring had no idea they were breached and had no idea how this database got on the forum, but this turned into an all-hands-on-deck situation for them, too. They only had a few dozen people working there at the time but everyone who was technical got involved with this investigation. When one journalist posted about it, soon many more journalists were calling, so the marketing team had to get involved too, trying to handle PR.
First, the Formspring security team needed to confirm the data. They took the 420,000 users and compared it to their database and it was a match. It certainly was their data. Next, they started looking for anomalous activity. That’s when they found someone had SSH’d into the server from a Russian IP. They took the IP and looked at more logs which indicated that this user logged into the web admin portal and from there, they were able to see what the user agent was for the person who accessed it. They also saw this hacker accessed the Wiki and placed madnez.php on the web server and ran some SQL queries from the admin control panel. They discovered all this in about one day. I guess their environment was just a lot smaller than LinkedIn to be able to get through it all quicker. Once Formspring confirmed that they had an intrusion, they needed to contain it. They changed the username that was used to login, deleted madnez.php, put more rules in place to change passwords more frequently, and set up monitoring rules to look for logins that weren’t from where the admin lives, and then totally destroyed and rebuilt certain servers that they knew the hacker had been on.
On top of that, they notified all their users that a breach had occurred. They told users what data had been stolen and had them change their passwords immediately. The day after they discovered this breach, the FBI called them and said hey, heard you had a break-in. Can we see what happened? Formspring sent the FBI all the logs they could to assist in the investigation. A few weeks later, Formspring had everything back to normal and things were working just fine again. But this story isn’t over; one guy hacked into three major websites and the FBI is now on the trail. You’ve got to hear what happens next. Stay with us. [00:30:00] At this point, the FBI was aware of all three of these cases. Someone had breached LinkedIn, Dropbox, and Formspring all from the same IPs and same user agents with a trail connecting it all. Pretty much the day LinkedIn found out that they’d been breached, they called the FBI and the FBI began interviewing people at LinkedIn and collecting logs from them. They saw that the hacker had connected from multiple IPs in Russia. They also saw the user agent with the word Sputnik in it. LinkedIn was sending them hard drives full of logs to examine and feeding them all the information they were finding.
They even supplied an image of the engineer’s iMac that got hacked to the FBI. While reviewing all this data, the FBI found something crucial in the logs. [MUSIC] They knew what IP and user agent the hacker had, so they looked at all the people who logged into linkedin.com, the public website, in the last few years. They found a person named Jammiro Quatro. This person had registered for a LinkedIn account way before this hack and had the same IP and user agent as the hacker. This could be gold. Like I said, users of LinkedIn often post their full resume there, so if this user had information about himself posted on his account, it could wrap up this whole story really quick. But Jammiro had a blank LinkedIn account. He wasn’t associated to any company and he didn’t have a single connection or friend on LinkedIn. But to create a LinkedIn account you need an e-mail address, so the FBI looked to see what e-mail address registered this account and it was chinabig01@gmail.com. The FBI thought this might be the e-mail address of the hacker.
Now, the FBI had quite a few IP addresses that they were considering suspicious with this, but they narrowed down their interest to five that may be owned by the hacker, and all of them were in Russia. If this had been in the US, the FBI could issue a subpoena to an ISP and get information on who pays the bill for that connection and get answers almost immediately. But things work differently when the FBI wants information from an ISP in Russia. There is a thing called the Mutual Legal Assistance Treaty, or MLAT. MLAT was set up to allow foreign nations to cooperate in helping criminal investigations by supplying law enforcement with internet service subscriber info. So, the FBI requested from Russia subscriber records through MLAT to see whose IPs those were. But this is not a fast process; it takes eight months to five years to get subscriber info through MLAT, so the FBI had to wait for a while on this. [MUSIC] In the meantime, they started cross-referencing the LinkedIn data with Dropbox and Formspring data. In all three attacks they found similar IOCs, or indicators of compromise; the same IPs, the same user agents, the same browser and OS.
Once again, they looked for users on those sites from those IPs who registered for an account before this hack took place. Dropbox also had a user registered with chinabig01@gmail.com. That person was named Jammis Gurus, a bit different from the Jammiro Quatro from LinkedIn. Formspring also had a user registered as chinabig01@gmail.com, too. Because there were so many similar indicators on all three breaches, the FBI was starting to believe that this chinabig01 e-mail address might have been owned by the hacker. So, the next step is the FBI contacted Google, the owners of Gmail, and issued a search warrant to get any information on that user. See, Google is a US company, so it’s fairly easy for the FBI to get information from a US-based company. Actually, I think they have to comply with law enforcement in this kind of way, and Google loves collecting logs on its users, so they had plenty to share. [MUSIC] First, the FBI saw that whoever was connecting to the server had the same IPs and user agents as the intruder who got into the other companies. Next, the FBI agent was able to see what search terms this person Googled while logged into their account.
Here are some of those search terms; WordPress vulnerabilities, TrueCrypt hack, Oracle export utility, EMS data export for Oracle. The user’s Google activity also showed them visiting a few sites like insiderpro.com which was the forum that these database dumps were getting posted to. The user also visited articles which talked about the LinkedIn hack. Then the FBI took a look in their inbox and looked at what e-mails they had and saw a welcome e-mail to Vimeo, a video file-sharing website. So, he requested information from Vimeo which also came back with matching user agents and IP addresses. The FBI also saw evidence that this person was logging into some LinkedIn accounts which were employees of automattic.com. Now, Automattic is the parent company to WordPress. [00:35:00] The FBI contacted Automattic and requested to see any login activity from these Russian IPs and sure enough, there were some login activities. Someone from Russia was logging in with different Automattic employee usernames and passwords. It’s unclear what exactly the hacker stole out of Automattic’s site, if anything, but it is clear he got in multiple times with different Automattic engineers’ usernames. The FBI agent then saw a welcome e-mail to afraid.org.
This is a website which offers dynamic DNS services. It’s a US-based company, so the FBI agent issued a search warrant to get data on who owned the account related to chinabig01. The name on this account came back as “Zopaqwe1”, and afraid.org also showed that whoever was accessing the site had the user agent with Sputnik in it, too. The FBI did a Google search on this “Zopaqwe1” which is a strange and unique word, and found a user registered on an online gaming site called kongregate.com. The FBI requested user details here and confirmed the user agent and IPs matched, but also discovered the user had registered a credit card on that site and had purchased some game credits. The FBI tried to trace the bank details of that card but it led them to a bank in Russia which they could not get extra information on. However, they did look to see what e-mail address was registered with this “Zopaqwe1” Kongregate account, and the e-mail for this user was r00talka@mail.ru. Now, since mail.ru is hosted in Russia, the FBI couldn’t issue a subpoena for that, either. But the FBI Googled the beginning of that e-mail address, r00talka, and found a Gmail user with the same name, r00talka@gmail.com.
[MUSIC] So, the FBI issued a search warrant with Google to get information on that Google user, and Google responded with more information. The first thing they saw was what Google searches that user had searched for, and they were searching for things like LinkedIn hack, MySQL count fields, change Mac address WiFi Windows 7. There were also some Google Map searches that this user did. They saw the user searched for a dentist in Moscow and some other map searches in Russia. Next, the FBI agent was able to look at e-mails for this r00talka Gmail account. He saw this person had registered an account at VKontakte which is like the Russian version of Facebook. The site would e-mail you anytime someone messaged you there and there were people messaging him asking about hacking e-mail accounts and different relationship-type stuff that was going on. But here, everyone is referring to him as Zhenya and everyone only spoke Russian to him. By this point, the subscriber records for that IP address came back from the MLAT request to Russia and it showed IPs were owned by two people, and it gave their physical address.
Yevgeniy Nikulin had one IP and someone else had the other. Yevgeniy lived on the same street that this person was doing Google Map searches on, so the FBI started looking up information about Yevgeniy and found photos of what he looks like. They compared those photos with the person on VKontakte’s account and they looked like the same person. The VK account was for a person who went by the name Zhenya which is actually a common nickname for people named Yevgeniy. I’m not sure how but the FBI investigation led them to a Russian guy named Kislitsin. [MUSIC] Kislitsin has been known in the past to broker deals between hackers and people buying database dumps. The FBI found Kislitsin’s e-mail address which was a Hotmail address owned by Microsoft, so they issued a search warrant with Microsoft to see what was in his inbox. There, the FBI saw e-mails going back and forth with the buyer of the Formspring data. The buyer ultimately decided to buy the dump for an equivalent of 7,100 US dollars. I’m not sure how much data was in there though; somewhere between 400,000 and 30 million user records.
Supposedly, the person who bought this was half-Belgian and half-Turkish, and what’s really strange is they used a middle man for this cash deal and he was also involved with the E-Trade and Scottrade hacks which I talked about in Episode 76, Knaves Out. The FBI indicted Kislitsin as a co-conspirator to this but ultimately was unable to capture him. However, the FBI was able to arrange a meeting with him in the Russian embassy in Moscow. So, they visited with this Kislitsin guy and he gave them a lot of information, not only information about this case but information on a few other cases the FBI was investigating, too. While meeting with Kislitsin, he told them that Yevgeniy Nikulin was the person who broke into Dropbox and still had access to it and had the Formspring [00:40:00] data, all with the goal to sell the database dumps on the black market for money. This was enough information for the FBI to issue an indictment for Yevgeniy. Multiple trails led right to him, but would they be able to catch him? Stay with us through the break to find out.
So, who is Yevgeniy Nikulin? Well, he’s Russian, from Moscow, born in 1987, so that made him twenty-five years old in 2012. Yevgeniy loves cars so much that he started a business in Moscow buying luxury cars and renting them out to people, and it was from that where he was often seen driving Maseratis, Lamborghinis, and Bentleys around town in Russia. I don’t know where all this hacking started for him. Details of his past are foggy. I imagine he got into IT and computers like anyone else; probably started with playing video games and then wanting to hack the video games or cheat, or maybe wanting to change his grades in school or wanting to just mess around with his friends by hacking into them like it was a game. Who knows why he got started in hacking? But by 2012 he was pretty familiar with how computers worked and how to exploit them. From reading about him, I also feel like his online life overlapped with some of these pretty notorious Russian cyber-criminals. He knew some of the other big hackers. Maybe they taught him.
Maybe they were hanging out in the same forums or something and maybe Yevgeniy wanted to be part of that hacking world that they were part of, because after all, he liked expensive things and was seeing some of the Russian hackers making gravy from their digital exploits. The FBI issued an indictment for Yevgeniy Nikulin but the problem was, he was in Russia and Russia was not going to arrest Yevgeniy for the FBI. Even if Russia did arrest him, he’s not gonna get extradited to the US for trial. So, the FBI had to wait. They knew the exact address of where Yevgeniy lived but had no way to go into Russia to get him. Now, up until this point, the world had thought the LinkedIn data breach was for 6.5 million users because after all, that’s what was posted on insiderpro.com and what’s more is that LinkedIn never clarified how many accounts got stolen. But in May 2016, someone posted that they had even more LinkedIn credentials for sale. They claimed to have 117 million user details from LinkedIn and was selling it for just over $2,000 in Bitcoin. This triggered a whole new news cycle.
HOST3: Morning topic; America’s money. A security breach at LinkedIn turns out to be much bigger than first thought.
HOST4: That’s right. The social network for business now says a hacker stole 117 million user passwords in the 2012 breach, far more than the original estimate of about 6.5 million.
JACK: I think about all the users of LinkedIn. Yes, of course; professionals looking to network, but also many top executives have accounts there. I mean, after all, if your business is listed there, shouldn’t the leader of that business be on there too? But on top of that, you have government officials on there. Lawmakers are there, members of congress, FBI agents, NSA agents, senators, and yes, even the president of the United States. Barack Obama made his account in 2007 when he was running for president and was president in 2012 when this happened. [MUSIC] This news swept through lots of circles and impacted a lot of people. What’s more is this new dump contained a lot of cracked passwords that anyone can see in plain text. It wasn’t that LinkedIn stored passwords in plain text but the hackers were able to find ways to crack a lot of the passwords that were in there.
Oh, and in fact, we got to see what the most common passwords that got cracked were. I’ll read you the top six [00:45:00] most common passwords that LinkedIn users were using in 2012. The most common was simply ‘123456’. Over 700,000 users had used this password because yes, LinkedIn’s minimum password length was six characters at the time. The next most popular password was ‘LinkedIn’, then the password ‘password’, then ‘123456789’, then ‘12345678’, then ‘111111’. People use bad passwords and you’re telling me some of those users are using the same passwords on multiple sites? On top of that, they’re using the same password at work? Ugh, it’s outrageous. A few months after that, in October 2016, the FBI got the break they were waiting for. Yevgeniy Nikulin was spotted in Prague, in the Czech Republic. With the indictment all processed and things ready to go, the Czech police tracked him to a restaurant where he was eating with his girlfriend. There’s bodycam footage of the police arresting him. Here, have a listen.
POLICE: [CZECH]
JACK: There’s actually not much to listen to, so I’ll describe what’s going on. Yevgeniy and his girlfriend are sitting at a restaurant. From the video, I see about three police officers coming in and they tell him to stay calm and put his hands where they can see them. Yevgeniy puts his hands on the table and they ask him to stand up and walk backwards towards the officer. Yevgeniy does exactly that. Then they pat him down, take some things out of his pockets and handcuff him, all while the other officer is making sure the girlfriend isn’t getting up. It’s all done very quietly and without any fuss, and Yevgeniy was taken to a Prague jail. For the next two years, Yevgeniy’s lawyers fought to keep him from getting extradited to the US. I wasn’t able to get a second confirmation on this but his lawyer said the FBI was trying to pin Yevgeniy for hacking Hillary Clinton’s e-mails at the time and was trying to get him to confess to that. Eventually, two years after his arrest, in 2018 the Czech Republic did extradite him to the US for a trial. Yeah, I went through the court records and I never saw one reference to Hillary Clinton in there.
The US had nine charges on him; computer intrusion, aggravated identity theft, conspiracy, and an international transmission of information causing damage to a protected computer. The victims of this case were listed as LinkedIn, Dropbox, and Formspring. But here’s why I love this story so much; Yevgeniy pleaded innocent on all these charges. [MUSIC] He claimed he didn’t do any of this and why is that my favorite part? Because it means this case had to go to trial which means witnesses, evidence, FBI testimony, and so much more becomes public record. To research this story, I got the pleasure of reading hundreds of pages of court transcripts. It was glorious to hear all the details from victims and law enforcement. We rarely hear these things. Like, there were three people from LinkedIn who all testified, explaining how the hacker got in and what their incident response plan was. There were three people from Dropbox giving testimony, and the CEO from Formspring explained everything he saw. On top of that, there were three FBI agents and a Secret Service agent who all gave testimony on how they were able to link all these pieces together and track him down.
It’s only from all this that we know anything about this story other than what you’ve seen in the headlines. I mean, I reached out to LinkedIn multiple times and the FBI multiple times to get someone to tell me about this story but nobody wanted to talk because I get it; what company wants to come on this show and tell me about the worst thing that’s ever happened to their company? No one. So, it’s just really rare for us to see all the details of what happened written out so wonderfully. This trial started in early 2020 but then the pandemic hit and the trial was delayed like, three months. During that time, the Secret Service arrested a hacker suspected for breaking into the SEC, Oleksandr Yaremenko. When they arrested him, they got access to his laptop and on it was all kinds of evidence on Yevgeniy; pictures of him, videos of him, chat messages with him, e-mails to him, tons more evidence. But it’s weird though because by 2020, Yevgeniy had been in jail for four years which if you think about it, that’s 13% of his life that he’s been in prison without anyone deciding on whether he’s guilty or not. This took a mental toll on him for sure.
He barely spoke any English. He would sometimes shove guards or medical examiners or just try to walk out of the place sometimes. He made a mess in his cell by getting toilet paper wet and throwing it up on the ceiling. He kept asking the judge for permission to have a Game Boy or a PSP to play. Yevgeniy didn’t testify himself. He had an interpreter in the court room saying everything to him. The whole time, he kept [00:50:00] saying he had nothing to do with this but the trial started up and there was quite a lot of evidence connecting him to the incidents. Just to recap the trail here, IPs that accessed the victims’ networks were registered to his name specifically. The IP used to steal data from LinkedIn led to a certain browser user agent, and that was associated to a LinkedIn account with a Gmail address, and that e-mail was registered at afraid.org which had a username that was on kongregate.com, and that person bought things with a bank card, and that bank card matched to other things that Yevgeniy bought. On top of that, there was Kislitsin who said that Yevgeniy did this, and Yaremenko’s computer which had more evidence. On July 10th, 2020, the trial concluded.
The jury found him guilty on all nine counts. The judge then sentenced him to 88 months in prison which is just over seven years, and they also ordered him to pay 1.7 million dollars in restitution for the damage he caused to the companies he hacked. Oh, and after that, he has to do three years of supervised release which I’m not quite sure how that works if you’re not a citizen of the US. [MUSIC] So, what do we learn from this story? Well, it sounds like in 2012, these victim companies weren’t doing user behavior anomaly detection. Like, if a user VPNs in from California and then an hour later that same user VPNs in from Russia, that should trigger an alert, right? Yeah, well, it didn’t. The technology at the time didn’t really do that kind of correlation. Now there’s better tools for monitoring user behavior analytics and I think tools like that have a lot of potential. Next, it’s crazy to me that some people use the same password for their LinkedIn account as their work accounts. Don’t reuse passwords like that. Use a complex, unique password for every account you have. The best way to do that is to use a password manager. They aren’t hard to use, so go get one.
I have a affiliate link to one in the show notes if you just want a good recommendation. It’s also worth noting that these companies seemed to have exceptional logging turned on and when they learned about this breach, they were able to archive those logs and do system snapshots right away to preserve any data that can be used forensically. I’ve seen a lot of companies just not log properly and it just always really bothers me. Oh, and that LinkedIn engineer who was hosting those two websites on his home computer, he’s moved those websites to host them on Linode now which is one of our sponsors. I think one of the lessons he learned from this was that opening ports from the internet into your home network can be dangerous. It exposes your computer to a world full of chaos which can ultimately result in someone getting access to your home network. I do think about that a lot in this story; if he wasn’t hosting those little websites at home, he probably wouldn’t have been the way in for this hacker.
It’s also interesting to see that bad guys target employees at their house, because that network is often not as strong as the corporate network. But here’s the crazy part of all this; remember that LinkedIn dump of 117 million user details that showed up in 2016? Later on that year, it just hit the public for anyone to see, so anyone can go look in the LinkedIn database to see what is in there, and there are still many people who did not change their passwords or changed it to something and then just changed it right back to what it was before that. What about all those people who reused passwords on all the other sites? Like, yeah, I changed my LinkedIn password ‘cause I was told to, but I didn’t change all the other six things that use that same password. That’s where we pick up in the next episode. Someone finds a password in the LinkedIn database and has quite a story to tell about that.
(OUTRO): [OUTRO MUSIC] Hey, do you know about the Darknet Diaries shop? Listen, I love coming up with new shirt designs. Every month I throw a few more up in the shop. These shirts look great; one is of Medusa but she’s got Ethernet cables coming out of her head instead of snakes, and there’s one that looks like a bouquet of flowers but the flowers are actually made of computer cables. Another one is of an archer who’s shooting an arrow but the arrow looks like a USB symbol. You’ve got to see these shirts for yourself to understand what I’m saying, so visit shop.darknetdiaries.com and find some shirts. I’m an independent creator who loves bringing this show to you free of charge every two weeks but what really helps me keep on that schedule are my Patreon supporters. These are people who donate money to the show every month to help keep it going. If you want to show your support for this show, please visit patreon.com/darknetdiaries and consider donating. Thank you. This show is made by me, the chief biscuit-dunker, Jack Rhysider. Sound design by the dream alchemist, Andrew Meriwether. Editing help this episode by the wizard of light bulb moments, Damienne, and our theme music is by the phonic magician, Breakmaster Cylinder. Even though when you ASCII stupid question, you get a stupid ANSI. This is Darknet Diaries. [00:55:00]
[OUTRO MUSIC ENDS]
[END OF RECORDING]