Bad GigaBot! Down Boy, Down

OK, so you’re probably thinking this is a post about some new type of robot that has gone crazy, and in a way you would be right. The recent upgrade (still in progress) to site requires a significant increase in the amount of CPU cycles. Normally this would not be much of a problem, but I noticed that my box was working extra hard last night and getting quite hot to boot. When I went looking, I did find a robot that was hammering my site.

GigaBot is the web spider for the GigaBlast web search site. It was not playing very nicely. I don’t know for how long it has been hitting my server, but so far this month it had downloaded 450MBs of data. After a little bit of research, I updated my robots.txt file to tell GigaBot to not search my site. Once it finally got that change and stopped hitting my site, the machine returned to normal.

Advertisements
Leave a comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: