Help with an annoying bot.

[Ben E Lou]

I'm getting hit hard by a bot from MSN that's really gumming up my web server, and blocking the IP doesn't seem to help.

#	Hits		Files		KBytes		Visits		Hostname
1	31004	43.92%	0	0.00%	0	0.00%	61	9.55%	msnbot-65-52-108-68.search.msn.com

That's just in October. 31,000+ hits already. Looking at my MySQL logs, this bot is hitting dozens of php scripts every second when it's doing its thing. That, uh, slows things down tremendously. Any ideas as to how I can get rid of this thing?

[Ben E Lou]

Oh, and I've already got a robots.txt file in place:

http://www.younglifenorthdekalb.com/robots.txt

That doesn't seem to be helping.

[panerd]

You sure that just isn't Mizzou Basketball Fan in the NCAA Expansion thread?

[MacroGuru]

You might have to get specific with the bot in your robots.txt file as it is known to ignore the wildcard command

User-agent: MSNBot
Disallow: /

Oh, and honestly, the MSNBot is the only one that tends to be a hog..

[Ben E Lou]

OK, I'll try that.

[Ben E Lou]

Heh. That was fast. Made that change and CPU Usage dropped from 100% to 3.6% in about 20 seconds. Thanks!

[MacroGuru]

Your Welcome

[RendeR]

Is Robots.txt supposed to sit in the root dir or in the www or Public_html dir?

I thought I had mine in there but it appears to be missing.

Any suggestions on a good all around robot stopper would be huge too.

[DanGarion]

Originally Posted by RendeR

Is Robots.txt supposed to sit in the root dir or in the www or Public_html dir?

I thought I had mine in there but it appears to be missing.

Any suggestions on a good all around robot stopper would be huge too.

It has to site in the root of the website, a publicly viewable location.

[CrimsonFox]

Awesome fix! Thanks for looking into that! Yeah the Werewolf forum has 30-40 guest bots in it every night

Hopefully that will clear up. Great tip Macro

[Ben E Lou]

CPU Usage		100%
CPU Load Average		29.92, 31.53, 33.65

Same deal. Major robot attacks and I'm not sure how to block them. I've done IP Deny in cPanel, but it appears that they still hit the MySQL server. The process manager shows me that there are dozens of queries running right now, far more than can be attributed to human traffic. Any ideas?

[Ben E Lou]

Dola:

To be clear, it's not msnbot this time. 88.131.106.22, which I have blocked in cPanel, is a major culprit.

[Ben E Lou]

Several that start with 180.76.5 are also problematic.

[MacroGuru]

Originally Posted by Ben E Lou

Dola:

To be clear, it's not msnbot this time. 88.131.106.22, which I have blocked in cPanel, is a major culprit.

This is the entireweb bot, which I do not know why it would be hitting your DB, that is weird.

The 180.76.5 seems to be the baidu spider which is from China and pretty aggressive

I don't know if this article can help you set yours server to block them..

httpx://johannburkard.de/blog/www/spam/introduction-to-blocking-spambots-and-bad-bots.html

[wade moore]

Someone please find a solution, this is painful!

[cuervo72]

Originally Posted by Ben E Lou

Several that start with 180.76.5 are also problematic.

Yeah, I had to block this one in cPanel for FOBL just yesterday.

[JediKooter]

Originally Posted by Ben E Lou

Same deal. Major robot attacks and I'm not sure how to block them. I've done IP Deny in cPanel, but it appears that they still hit the MySQL server. The process manager shows me that there are dozens of queries running right now, far more than can be attributed to human traffic. Any ideas?

They're inside the house..GET OUT!




I've got nothing. Sorry.

[Ben E Lou]

Originally Posted by Ben E Lou

Same deal. Major robot attacks and I'm not sure how to block them. I've done IP Deny in cPanel, but it appears that they still hit the MySQL server. The process manager shows me that there are dozens of queries running right now, far more than can be attributed to human traffic. Any ideas?

Maybe it just took a while for the blocks I put in place to populate? *shurg*

CPU Usage		4.2%
CPU Load Average		0.34, 0.26, 0.21