Subscribe / Unsubscribe Enewsletters | Login | Register

Pencil Banner

Your strategy for dealing with web bots has to take into accout business context

Renny Shen, senior product marketing manager, Akamai’s Web Experience & Security Division | Sept. 5, 2016
Simply blocking all bot visits can end up hurting your business

This vendor-written tech primer has been edited by Network World to eliminate product promotion, but readers should note it will likely favor the submitter’s approach.

Between 30%-70% of traffic to most websites is from bots, meaning it is non-human traffic. And while many assume bot traffic should be blocked, that is a black and white approach to a problem that’s very much grey. The reality is that some bots are good, some bots are bad, but most will be somewhere in between. What you need is a bot management tool that lets you apply a range of management actions based on your website’s business model.

Consider the airline industry. A typical airline bookings site might see 50% of its traffic coming from human visitors, with the other 50% coming from bots. While the first thought is often to block the bot traffic, a more effective approach is to understand why the bots are here and what the impact is on the business.

The airline industry is highly competitive with airlines competing on schedules and prices across thousands of routes every day. Flight schedules and prices are highly variable from one day to the next and also fluctuate rapidly based on supply and demand – prices go up as any given plane fills, and vice-versa. Consumers trying to get the best price, competitors are trying to win your customers, and partners are trying to help customers get the best prices and all are dependent on continuous access to the latest data.

Providing this data access comes with a cost. Most airlines don’t actually store their flight schedules and prices in house. They use a global distribution system (GDS), like Amadeus or Sabre. Every time someone looks up flight data, that entails a call to the GDS. And every call to the GDS, whether it’s from a human or a bot, incurs a cost, which can be substantial over large volumes – on the order of hundreds of thousands of dollars every month. With consumers and partners, the GDS cost is built into the cost of sales. With competitors, however, it’s just a lost cost without any potential benefit.

All of this is business context that has to be considered in a conversation about bots, because it helps us determine the technical requirements of a bot management solution. First, we know that blocking bots won’t work. The airline industry is a $700 billion global industry. Both competing airlines and the competitive intelligence services that sell to them have a tremendous financial incentive to get the flight data and capture a larger percentage of that $700 billion in revenue. Blocking their bot doesn’t make them go away. Blocking just lets them know that you’ve detected their bot. And that just triggers them to evolve their bot to evade your detections.


1  2  3  Next Page 

Sign up for Computerworld eNewsletters.