E-Dribble

If you own a site that has a contact form, forum or other method of communication or contact and your site has been in existence longer than a month, you’ve probably witnessed the effects of a bot. Whether it’s bogus referrers in your access logs, Viagra spam in your forum, or sales pitches being sent through your contact form, it’s a pain to the webmaster. The chances are slim that you’ll ever find a way to completely rid yourself of it, but I’m going to give you the top three fastest ways to drop the number of bots to knock on your door. These Will Not get rid of the problem, but will simply lessen the load. The lower you want the number of bots to be, the harder you have to work. This is intended for quick returns and to get you started on your way to a cleaner and less cluttered website.1) It’s obvious, but the number one way to keep bots from filling up your forum or contact forms with offer for sexual enhancement or fat reducing pills is to NEVER have an unprotected form. If it’s a forum, require registration. If it’s a contact form, have a CAPTCHA. DO NOT have a form with no protection! Once it’s found, it will be abused until your form is useless due to the number of bogus messages being sent.

2) Stop the dumbest of the dumb before they have a chance to strike. One popular way to do this is to block one of the popular methods of contact: libwww-perl, or LWP. This is a perl module that will allow their script to contact the site with no human interaction. This allows them to post comments or forum topics(or register) all without anyone actually having to type anything. Now, before you get too excited, we’re not block all people using libwww. We’re blocking the people using libwww who are too stupid to spoof their user-agent string. Unfortunately, the numbers are lower than we’d like, but remember that we’re going for quick and dirty.

On an apache server, you simply use .htaccess to block the agent. .htaccess is a file that sits in the root directory of your site and tells the server what to do on each page generation. Think of it as a set of rules that the server adheres to each time someone visits. In our case, we’re going to tell the server to block any requests made by anything with a user-agent that contains the libwww signature:

In .htaccess, add:

Like I said, this will only block the most moronic of the morons.

3) Blocking by “keyword”: If you monitor your referrers, you may find thousands of sites referring to yours with URL’s like: http://viagraisthecoolest.info/cialis/tramadol/letmesellyou/illegaldrugs.htm . You’ll find that this page is simply a sales page. Other than lost server resources, the damage isn’t that great, unless you share your referrers with the visitors of your site. Many CMS’ have blocks that share the latest X number of referrers, and this is what the bots are counting on. Here’s how you block the most common of the referrers:

Again, in .htaccess:
longer than a month, you’ve probably witnessed the effects of a bot. Whether it’s bogus referrers in your access logs, Viagra spam in your forum, or sales pitches being sent through your contact form, it’s a pain to the webmaster. The chances are slim that you’ll ever find a way to completely rid yourself of it, but I’m going to give you the top three fastest ways to drop the number of bots to knock on your door. These Will Not get rid of the problem, but will simply lessen the load. The lower you want the number of bots to be, the harder you have to work. This is intended for quick returns and to get you started on your way to a cleaner and less cluttered website.1) It’s obvious, but the number one way to keep bots from filling up your forum or contact forms with offer for sexual enhancement or fat reducing pills is to NEVER have an unprotected form. If it’s a forum, require registration. If it’s a contact form, have a CAPTCHA. DO NOT have a form with no protection! Once it’s found, it will be abused until your form is useless due to the number of bogus messages being sent.

2) Stop the dumbest of the dumb before they have a chance to strike. One popular way to do this is to block one of the popular methods of contact: libwww-perl, or LWP. This is a perl module that will allow their script to contact the site with no human interaction. This allows them to post comments or forum topics(or register) all without anyone actually having to type anything. Now, before you get too excited, we’re not block all people using libwww. We’re blocking the people using libwww who are too stupid to spoof their user-agent string. Unfortunately, the numbers are lower than we’d like, but remember that we’re going for quick and dirty.

On an apache server, you simply use .htaccess to block the agent. .htaccess is a file that sits in the root directory of your site and tells the server what to do on each page generation. Think of it as a set of rules that the server adheres to each time someone visits. In our case, we’re going to tell the server to block any requests made by anything with a user-agent that contains the libwww signature:

In .htaccess, add:

If .htaccess finds one of the words in the referrer link, it will refuse to serve a page to the bot request, instead serving a 403 error page.

Something to consider. A legitimate user may get accidentally caught in your attempt to block the bots. What to do? Well, what I do is create a custom 403(forbidden page) with a link to the page they were trying to get to along with an apology. This allows them to still get to their desired page and will keep the bots from filling up your referrers quite so badly.

Another thing to note is that you can customize this list as your needs change. This list will get you started however, and will cause a noticeable drop in bogus hits.

There’s many more things you can do, but this will give you the most bang for the least effort. Google is your friend for finding other ways to reclaim your website.

[EDIT]New article discussing further measures to combat bots can be found here[/EDIT]