Bots now outnumber humans on the web

Bot traffic has surpassed humans this year, now accounting for 59 percent of all site visits, according to a report released today by Distil Networks.

By comparison, last year, bots accounted for 45 percent of all traffic to Distil's customers' websites.

The majority of the bot traffic was "good bots," said Distil CEO Rami Essaid, such as that from search engines and social media sites. These bots comply with "robot.txt" files that Website administrators use to tell bots what to do, and bring value to the websites they visit.

He attributed the growth in bot traffic to more aggressive indexing by Bing, and by new search engines coming online.

"The 'good bots' almost doubled as a category this year," he said.

"Good bots" accounted for 36 percent of traffic this year, up from 21 percent last year. "Bad bots" were responsible for 23 percent of traffic this year, down slightly from 24 percent last year -- not because volumes were down, Essaid repeated, but because the number of "good bots" rose dramatically. Human traffic was just 41 percent, down from 55 percent last year.

The company helps customers keep out bad bots, of which the company saw 23 billion last year.

The company defines "bad bots" as those that don't respect "robots.txt" files and don't provide value to the sites they visit.

The bad bots fall into three categories. The largest one is the unwanted bots that scrape data from websites, especially bots used by business competitors such as those looking for pricing information. Then there are the malicious bots that explore websites looking for vulnerabilities.

The smallest category, which accounts for 10 percent of the bad bots, are the ones engaging in click fraud, making brute-force login attempts or trying to post spam. These actively malicious bots can be identified by their use of "post" requests

Amazon is the bot leader

Amazon led the charts this year. Its cloud computing platform was popular with bot creators -- 78 percent of all traffic from the Amazon cloud was bots this year, accounting for 15 percent of all bots, up from 9 percent last year.

In addition, Amazon itself was a bot creator, with 70 percent of traffic from Amazon.com being bots, accounting for 3 percent of all bots.

"Some of our customers compete with Amazon," said Essaid. "They don't want Amazon bots indexing their pricing, inventory and content."

Traditional hosting companies, like Verizon Business, Level 3, Hosting Solutions International, and the Las Vegas Datacenter dropped in the rankings or fell off the list altogether as defenders learned to block traffic from those sources.

"So the bad guys are now leveraging more and more residential networks, relying more on computers that are part of botnets," he said.

It's harder to tell if traffic that originates from a cable company like Comcast is human or not, he said.

Last year, only 1.8 percent of Comcast's traffic was bots. This year, that share was up more than three-fold, to 6.4 percent.

Bots get mobile

Traffic from mobile devices was also up, he said. Last year, 6 percent of bad bot traffic looked like it came from mobile devices, up from less than 1 percent in 2013.

While most of the this mobile traffic is spoofed, so as to avoid anti-bot defenses, between 20 and 30 percent of it is running on actual mobile devices, Essaid said.

The bot makers either install botnets on infected Android devices, or set up bot farm using cheap Android phones, he said.

Not only is traffic from actual mobile devices more like human mobile traffic, but it also allows the bad guys to interact directly with native mobile applications or custom content that requires a mobile device.

In addition to pretending to originate on mobile devices, some bots also try to mimic human behavior.

In 2014, 41 percent of bad bots used such tools as browser automation software to replicate actual human web browsing, for example.

In addition, in 2013, many bot makers scheduled their bots for the late evening hours, when security staffers were less likely to be at work. Bat bot traffic percentage would jump from less than 2 percent during business hours to as high as 18 percent in the evening.

"This year, we've seen that level out completely," said Essaid. "They continue to try to adapt their behavior based on what's getting them caught."

Some bots even disguise themselves as other bots. Webmasters typically allow the Googlebot to index their pages so that their sites would rank well in search.

Join the CSO newsletter!

Error: Please check your email address.

Tags securitybinglegalmalwarecybercrime

More about Amazon.comVerizonVerizon Business

Show Comments

Featured Whitepapers

Editor's Recommendations

Solution Centres

Stories by Maria Korolov

Latest Videos

  • 150x50

    CSO Webinar: The Human Factor - Your people are your biggest security weakness

    ​Speakers: David Lacey, Researcher and former CISO Royal Mail David Turner - Global Risk Management Expert Mark Guntrip - Group Manager, Email Protection, Proofpoint

    Play Video

  • 150x50

    CSO Webinar: Current ransomware defences are failing – but machine learning can drive a more proactive solution

    Speakers • Ty Miller, Director, Threat Intelligence • Mark Gregory, Leader, Network Engineering Research Group, RMIT • Jeff Lanza, Retired FBI Agent (USA) • Andy Solterbeck, VP Asia Pacific, Cylance • David Braue, CSO MC/Moderator What to expect: ​Hear from industry experts on the local and global ransomware threat landscape. Explore a new approach to dealing with ransomware using machine-learning techniques and by thinking about the problem in a fundamentally different way. Apply techniques for gathering insight into ransomware behaviour and find out what elements must go into a truly effective ransomware defence. Get a first-hand look at how ransomware actually works in practice, and how machine-learning techniques can pick up on its activities long before your employees do.

    Play Video

  • 150x50

    CSO Webinar: Get real about metadata to avoid a false sense of security

    Speakers: • Anthony Caruana – CSO MC and moderator • Ian Farquhar, Worldwide Virtual Security Team Lead, Gigamon • John Lindsay, Former CTO, iiNet • Skeeve Stevens, Futurist, Future Sumo • David Vaile - Vice chair of APF, Co-Convenor of the Cyberspace Law And Policy Community, UNSW Law Faculty This webinar covers: - A 101 on metadata - what it is and how to use it - Insight into a typical attack, what happens and what we would find when looking into the metadata - How to collect metadata, use this to detect attacks and get greater insight into how you can use this to protect your organisation - Learn how much raw data and metadata to retain and how long for - Get a reality check on how you're using your metadata and if this is enough to secure your organisation

    Play Video

  • 150x50

    CSO Webinar: How banking trojans work and how you can stop them

    CSO Webinar: How banking trojans work and how you can stop them Featuring: • John Baird, Director of Global Technology Production, Deutsche Bank • Samantha Macleod, GM Cyber Security, ME Bank • Sherrod DeGrippo, Director of Emerging Threats, Proofpoint (USA)

    Play Video

  • 150x50

    IDG Live Webinar:The right collaboration strategy will help your business take flight

    Speakers - Mike Harris, Engineering Services Manager, Jetstar - Christopher Johnson, IT Director APAC, 20th Century Fox - Brent Maxwell, Director of Information Systems, THE ICONIC - IDG MC/Moderator Anthony Caruana

    Play Video

More videos

Blog Posts

Market Place