Webserver Throttling Techniques

Introduction

Mostly ill-behaved spiders can cause quite a bit of load on a TWiki server. TWiki's cgi-scripts are quite CPU intensive. If a spider grabs pages in the range of a few seconds, a site can get slow for users working on the site.

A brute force method is to ban by IP address if there are too many requests in a certain timeframe. The BlackListPlugin does that effectively. However, TWiki scripts are still invoked for requests by banned IP addresses.

A more gentle method is to throttle spiders if they access too many pages in a certain timeframe. That way, ill-behaving spiders are slowed down, which reduces the load on the server.

There are several throttling and blocking methods:

Throttling with mod_perl handler Stonehenge::Throttle

RandalSchwartz wrote an article for Linux Magazine, Oct 2000, on "Throttling your web server", http://www.stonehenge.com/merlyn/LinuxMag/col17.html

Temporarily Blocking with mod_bwshare

From http://www.topology.org/src/bwshare/README.html:

The Apache module mod_bwshare throttles HTTP requests to Apache 1 and 2 servers for each client IP address independently. The mod_bwshare module accepts or rejects HTTP requests from each client IP address based on past downloads by that client IP address. If the HTTP client's download rate exceeds specified levels, the reponse to the HTTP client is an HTML warning message. A human browser will see a warning message indicating how long to wait.

Throttle with mod_throttle

From http://www.snert.com/Software/mod_throttle/

This Apache module is intended to reduce the load on your server & bandwidth generated by popular virtual hosts, directories, locations, or users according to supported polices (see below) that decide when to delay or refuse requests. Also mod_throttle can track and throttle incoming connections by IP address or by authenticated remote user.

External Links

• http://meta.wikimedia.org/wiki/Request_throttling - Request throttling on Wikimedia
• http://en.tldp.org/HOWTO/Apache-Overview-HOWTO-2.html#ss2.4 - How-to on Apache performance and scalability

-- Contributors: PeterThoeny

Discussion

Please help compile more throttling techniques, and please share your experience here!

-- PeterThoeny - 28 Mar 2006