Sonatype's crown jewel is its database of descriptions of over 1.2 million open source packages. “If that is lost, it could be an existential outcome,” said Wayne Jackson, CEO of the Fulton, Maryland-based software supply chain management company.
To shut down any such leak quickly, Sonatype monitors the web for any indications that its data has been stolen and is being shared on line. That monitoring includes the dark web.
The internet's dark side isn't actually all that big. Media accounts frequently overestimate the size of the dark web by lumping in everything that's not accessible by search engines, and that includes corporate intranets and password-protected sites like online forums, bank websites and email platforms.
According to the FBI, there are only about 800 criminal internet forums worldwide. While their impact might be large, the number of people using them often isn't.
A 2015 scan of the anonymity network Tor by the PunkSpider Web vulnerability scanner found around 7,000 Tor sites, only 2,000 of which were active. Not all of these sites are run by criminals. Dissidents who live under repressive regimes, security-conscious agencies and companies, and individuals very concerned about privacy also use TOR, Freenet and the Invisible Internet Project, or I2P.
When it comes to criminally oriented dark web sites, not all of them are of interest to enterprise InfoSec professionals. "There's a large part of the dark web that deals in human trafficking and drugs and that kind of thing," said Jonathan Couch, SVP of strategy at Reston, VA-based ThreatQuotient, Inc., which helps firms collect and organize data coming in from disparate internal and external intelligence sources. "I would say that that has become the majority of the illegal traffic on the web, and it doesn't affect corporate networks."
In 2015, a Trend Micro scan found approximately 8,000 suspicious sites on the dark web, of which about a third were connected to malware download pages on the public web. Just under a third were proxy avoidance sites that help users get around school, company or government filters, and a quarter were related to child pornography. Only 5 percent were related to hacking.
Some of those forums have since been taken down, said Ed Cabrera, chief cybersecurity officer at Irving, Tex.-based Trend Micro Inc. Others have become more fractionalized and specialized.
In the wake of high-profile take-downs by law enforcement authorities, many have also tightened up their security. "You had to be vetted to gain entry into the forums," Cabrera said.
According to Terbium Labs, the number of forums of interest to enterprise cybersecurity professionals has grown from a few dozen in 2015 to a few hundred today, many of them highly specialized. (For a sample of what's sold on the dark web, see the downloadable infographic, "How hackers make money," from Keeper Security at the end of this article.) While the dark web is typically illustrated by an iceberg — where the small tip that's showing is the public web — the part of it that's of interest to security researchers is growing but it still reasonably manageable in size.
A company can set up a dark web data mining operation and become productive in about a day, said Jason Polancich, founder and chief architect of SurfWatch Labs, Inc. “Most businesses already have all the tools on hand for starting a low-cost, high-return dark web intelligence operation, within their own existing IT and cybersecurity teams,” he said. “And most large enterprises are either starting this or already have it in place.”
According to ThreatQuotient's Couch, however, most companies would be better off letting someone else do the digging. "There are a lot of risks you run, from law enforcement and other perspectives, from interacting on the dark web," he said.
A safer, and more cost effective, approach is to use vendors like SurfWatch, Terbium and Recorded Future that offer monitoring, indexing or alerting services, helping companies react to, or stay ahead of, dark web threats. That could be someone posting sensitive company records, discussing a planned attack or selling a vulnerability in software a company uses.
These vendors develop specialized tools that help them gather this data and embed operatives deep within the criminal communities. Plus, the vendors get a broader picture of what's going on because they serve a large cross-section of customers.
According to Adam Meyer, chief security strategist at SurfWatch, companies like his have improved their ability to mine the dark web over the past two years — and to keep up with changes. "Shops go down, shops go up, sites change their URLs, law enforcement comes in and sites scatter," he said. "It's a fluid environment." Sometimes, sites shut down to rip off their customers.
Doing business with criminals is a risky proposition. Some dark web marketplaces position themselves as trusted brokers, offering escrow accounts to guarantee delivery and payment. "The more users they have, the more money they have in the escrow," said Meyer. "At some point, the operators look at that bank account and say, ‘We can just take the money and run.’ There's no honor among thieves."
Meanwhile, the operators set up new sites or competitors step in to fill the gap. "If a market with a 1,000 users shuts down suddenly, those users have to go somewhere," he said. "We will collect from the new marketplaces manually for a while and see if it gets traction. If it gets traction and starts growing again, we'll apply automation to it and start mining it automatically. Once you do this 10, 15, 100 times, you start to know what works from an automation perspective, traffic perspective, and you get a lot more efficient at it."
SurfWatch has also spent two years on developing, using tools like natural language processing to pull out the most interesting information and delivering it to customers. "For example, if there's some kind of credential dump, you can instantly see someone asking for a copy of it, and they're going to start attacking companies with that -- you can see that conversation happening right away," he said.
Another vendor in this space, Terbium Labs, offers a search service — Matchlight — that allows enterprise customers to search for proprietary information via a fingerprint. “It's a blind search technology,” said CEO Danny Rogers. “We give clients the ability to search this index in an automated way without revealing to us what they're searching for.”
The core feature of Matchlight allows enterprises to set up alerts for data that they want to monitor for, such as customer lists or trade secrets. “The faster they can find out that there's a data leak, the faster they can kick off their response and the less damage will occur,” he said.
For example, if the scan shows that the data is being distributed on a legitimate, law-abiding site, the enterprise can request that it be taken down. If the data is credit card numbers, they can be canceled quickly, before criminals can make fraudulent charges. If a company is aware that there's a leak, they can find it and shut it down before more damage is done.
One of the customers using Matchlight is Sonatype, which is using the service to keep an eye out for any sign of its open source software database. “The golden asset for us is our metadata which describes the attributes of open source code,” said Sonatype's Jackson. With Terbium, a breach can be discovered in just minutes, he said, in a private and totally automated way.
Another vendor, Somerville, Mass.-based Recorded Future, Inc., can create a fingerprint based on the hardware and software that an enterprise has deployed, then search the dark web for new vulnerabilities identified in those systems — as well as also looking for mentions of the company or its employees, IP addresses or email addresses.
Finding those mentions is getting more difficult, however, as criminals have gotten more clever at covering their tracks. "They are well aware that almost every large company is crawling the dark web in some way or working with different vendors or providers of such data," said Andrei Barysevich, director of advanced collection at Recorded Future, Inc. "We rarely see really valuable data openly advertised."
Instead, criminals are making deals on a one-on-one basis, with an established circle of trusted counterparties. That's when it helps to have people in place, he said.
"Our analysts and agents who are deeply embedded in these communities are getting direct messages from sellers notifying them that they have this information for sale, that information for sale," he said.
Watch for cyber criminals migrating to messaging apps
In the wake of recent successes that law enforcement has had in shutting down some dark web operators, some criminals are turning to messaging apps to reach their customers. These apps are seen as safer because they encrypt communications. That makes it harder for law enforcement to crack down on them.
"It's part of a global trend -- not only in the cyber-crime world -- of online activity shifting to mobile," says Alon Arvatz, co-founder and chief product officer of security service provider IntSights. "It seems that law enforcement agencies are targeting traditional underground communities, and cyber criminals are aware of it. Thus, this factor should keep pushing them towards mobile apps."
According to IntSights's recently released report, "Messaging Applications: The New Dark Web," mobile dark web activity has seen a 30-times increase in activity from July 2016 to July 2017. While the firm found dark web activity on nearly all the popular messaging apps, a relative newcomer, Discord, had the most dark-web activity, nine times as much as the app with the next highest activity level. “This report shows that the underground community is always adapting to new realities, so it's important to keep track and identify the next shift in their activity characteristics,” says Arvatz.
It is possible for companies to monitor the mobile dark web to see if their data has been compromised. Each criminal group will have an invite link where you can request access. IntSight has identified more than 11,000 of those links and will make them available upon request.
Arvatz notes that mobile apps groups are highly volatile and have a short life cycle, so you need a different approach to gaining access to new sources. “It's important to implement an automated means to identify new mobile apps invites and gain access instantly. Manual work won't necessarily be fast enough,” he says.
Another alarming aspect of the mobile dark web is that it makes cyber crime accessible to more people. “Essentially, it makes monetization of cyber-crime easier. In the long run, it will increase the profitability of cyber-crime, so companies should invest more resources in protecting themselves against cyber-crime,” says Arvatz.
Some countries such as Russia and China have moved to limit or ban the use of encrypting messaging apps, although not necessarily to stop the mobile dark web. As criminal use of these apps increase, it may prompt more government action. “It provides governments with more incentive to ban these types of apps. In addition, law enforcement is now targeting black markets on the dark web, so we should expect to see them targeting cyber-crime on mobile apps in the near future,” says Arvatz.