Seven Insights into the Continued Persistence of Bad Bots

Bad bots are evolving and more sophisticated than ever. While the goal of each bot operator might be different depending on their industry, bots are the tool of choice and vital to hackers’ and fraudsters’ success. No organization–large or small, public or private–is immune.

Increasingly, bots are mimicking real human workflows across web applications to “behave” like real users. Advanced attackers now show definitive behavior that they know about the technology they’re trying to defeat, and they’re continuously learning how to adapt their tactics.

In its “2019 Bad Bot Report: The Bot Arms Race Continues,” Distil Networks investigated hundreds of billions of bad bot requests from 2018 across thousands of domains to provide deeper insight into the daily automated attacks wreaking havoc on websites, mobile apps and APIs. Below are the top seven findings from the report.

Data Point No. 1: Bad bot traffic slightly less in 2018

In 2018, bad bots accounted for 20.4% of all website traffic—a 6.35% decrease over the prior year. This is the first time since 2015 that bad bot traffic has decreased from the previous year. More good news: the number of human users is up in comparison with bots for the first time since 2016. But it’s still surprising to see that human traffic comprises only 62% of all internet traffic. When the goal is to attract real humans to your website, these numbers show that the bot problem is still significant. 

Data Point No. 2: Bad bot sophistication levels remain consistent

Advanced persistent bots (APBs) are a combination of moderate and sophisticated bad bots and continue to plague websites, accounting for 73.6% of bad bots. APBs tend to cycle through random IP addresses, enter through anonymous proxies and peer-to-peer networks and are able to change their user agents. They use a mix of technologies and methods to evade detection while maintaining persistency on target sites. 

Data Point No. 3: The bot problem affects every industry

Some bad bot problems run across all industries while others are industry-specific. Websites with login screens are hit by bot-driven account takeover attacks two to three times per month. Content and price scraping is rampant and is undertaken by bots. Meanwhile, nefarious competitors use bots to undercut prices on ecommerce sites, hoard seats on airline flights, and scalp the best concert tickets. 

The industries hit most by bad bot traffic include financial (42.2%), ticketing (39.3%), education (37.9%), IT & services (34.4%), and marketing & advertising (33.3%).

Data Point No. 4: Half of bad bots claim to be Google Chrome 

Bad bots continue to follow the trends in browser popularity, impersonating the Chrome browser 49.9% of the time. The use of data centers reduced in 2018 with 73.6% of bad bot traffic emanating from them—down from 82.7% in 2017. 

Data Point No. 5: Amazon was the source of the most global bot traffic

Bad bots were launched from 1,935 ISPs during 2018, with Amazon the leading ISP for originating bad bot traffic. In 2018, 18.0% of bad bot traffic originated from it compared to 10.6% the previous year. This is no surprise; AWS is by far the world’s largest and most-utilized cloud service provider, owning about 33 percent of the global market.

Digital Ocean and Comcast Cable were the second and third largest sources of bad bot traffic. Last year’s number one, OVH Hosting, dropped to fourth place with 3.1% bad bot traffic in 2018 compared with 11.6% the prior year—a significant drop.

Data Point No. 6: Bad bots are all over the world

With most bad bot traffic originating from data centers, the United States remains the “bad bot superpower” with more than half of bad bot traffic coming from the country. The U.S. is followed by the Netherlands (5.7%), China (3.9%), Germany (3.9%), and Canada (3.2%)

Data Point No. 7: However, Russia and Ukraine are the most blocked

The two countries combined make up nearly half (48.2%) of country-specific IP block requests. A third of companies block Russia – the most blocked country for the second year running – while 15.5% block Ukraine. The other countries in the top five include India (15.2%), China (11.2%) and the U.S. (6.6%).

If you have a suggestion for an eWEEK Data Points article, email cpreimesberger@eweek.com.