Hi There,

My system has been absolutely tied up by my (not)friend Google-bot as they tried to scan my Internet fidonet nodes (geez :) ). What a pia.. I took care of it as soon as I noticed it though.. Luckily, bbbs has a feature that lets you keep all connections or some, whichever you like, that try to tie up your tcpip nodes.. I have 7 each of binkp, www, telnet, and raw tcpip.. so the idiots at google-bot were trying anyway they could.. I laughed when I saw them try the port on 24555 (my bbbs binkp port).. still a problem because the attempt is made, but once I got their ip address in with the / at the end of the ip address to specify any port they used, that took care of it.

For those of you that use BBBS, and run into this, this is what I put in inet.bbb:

[bbbsd]

!ftp 66.249.65/$
!www 66.249.65/$
!tcpip 66.249.65/$
!binkp 66.249.65/$

Seemed like it should work fine, and it did <smile>

Take care,
Janis

--- BBBS/LiI v4.01 Flag
* Origin: Prism bbs (1:261/38)

Well, the following didn't work as I thought it was working.. <grin>

Later, Google was able to connect.. more coffee was obviously needed:

For those of you that use BBBS, and run into this, this is what I put in inet.bbb:

[bbbsd]

!ftp 66.249.65/$
!www 66.249.65/$
!tcpip 66.249.65/$
!binkp 66.249.65/$

But THIS works (so far so good <g>):

[bbbsd]
!FTP 66.249.65.83/10
!HTTP 66.249.65.83/10
!raw 66.249.65.83/10
!TCPIP 66.249.65.83/10

Don't know where my brain was last night <g>

Take care,
Janis

--- BBBS/LiI v4.01 Flag
* Origin: Prism bbs (1:261/38)

Hello, Janis.

Friday August 20 2010 at 14:29, you wrote to All:

!FTP 66.249.65.83/10
!HTTP 66.249.65.83/10
!raw 66.249.65.83/10
!TCPIP 66.249.65.83/10

Good idea; I'll just block the entire block from coming in. My web server is professionally hosted so there's no need for Google to go poking around.

Later,
Sean

... You're only young once. How long that once lasts is the question.
--- GoldED/2 3.0.1
* Origin: Nocturnal State BBS - (423) 926-7999 - bbs.kd5col.info (1:18/200)

Hi Sean,

!FTP 66.249.65.83/10
!HTTP 66.249.65.83/10
!raw 66.249.65.83/10
!TCPIP 66.249.65.83/10

Good idea; I'll just block the entire block from coming in. My web server is professionally hosted so there's no need for Google to go poking around.

Exactly :) Mr. Google-bot was KILLING my already crummy DSL system here - They
constantly scan my main apache server on port 80, so it's not like they can't get updates. When they started scanning my BBBS ports (ftp 6071, http 8080, binkp 24555, telnet 23) I'd had enough.. people couldn't even connect to pickup
their mail via the internet since I've only got 7 nodes.. and they didn't take up just "one" node...

Now thankfully, this is what I'm seeing in the log for just one port, but BBBS is blocking all of BBBS's ports when Google-bot tries:

100821 11:22 Got FTP from 66.249.65.137:38215, blocked by inet.bbb!
100821 11:23 Got FTP from 66.249.65.137:43860, blocked by inet.bbb!
100821 11:23 Got FTP from 66.249.65.137:42786, blocked by inet.bbb!
100821 11:23 Got FTP from 66.249.65.137:43327, blocked by inet.bbb!
100821 11:23 Got FTP from 66.249.65.137:39070, blocked by inet.bbb!
100821 11:23 Got FTP from 66.249.65.137:45389, blocked by inet.bbb!
100821 11:23 Got FTP from 66.249.65.137:45336, blocked by inet.bbb!
100821 11:23 Got FTP from 66.249.65.137:46020, blocked by inet.bbb!
100821 11:23 Got FTP from 66.249.65.137:46428, blocked by inet.bbb!
100821 11:23 Got FTP from 66.249.65.137:45457, blocked by inet.bbb!
100821 11:23 Got FTP from 66.249.65.137:43847, blocked by inet.bbb!
100821 11:23 Got FTP from 66.249.65.137:44818, blocked by inet.bbb!
100821 11:23 Got FTP from 66.249.65.137:46356, blocked by inet.bbb!
[yay! <bg> The block then allowed this:]
100821 11:28 Got raw from 173.52.216.2:3038, spawning node 5.
100821 11:28 Closing raw at node 5, 0kB in, 15kB out.
100821 11:28 Got raw from 69.55.228.237:59366, spawning node 6.
100821 11:28 Closing raw at node 6, 0kB in, 7kB out.
100821 11:29 Got raw from 216.66.97.101:4954, spawning node 7.

The above shows them hitting the ftp port on BBBS.. but the dogs were hitting all of the BBBS ports, and constantly as shown above for bbbs's ftp.. I even emailed them first.. no response of course..Robots don't talk much and I guess neither does Googles' admin.

Who needs it.. Once I saw them hitting all the ports of my bbs, I said, enough!


I should just kill them period even on port 80 for apache, and I still may do so <grin>. I have free-find indexing my site automatically, and they do it in such a way that it doesn't interfere with anything.. so it's not like google-bot couldn't do it that way if they wanted to!

Take care,
Janis

--- BBBS/LiI v4.01 Flag
* Origin: Prism bbs (1:261/38)

Hello, Janis.

Saturday August 21 2010 at 11:58, you wrote to me:

I should just kill them period even on port 80 for apache, and I still
may do so <grin>. I have free-find indexing my site automatically,
and they do it in such a way that it doesn't interfere with anything..
so it's not like google-bot couldn't do it that way if they wanted to!

I had to block that Chinese search engine, Baidu, from my site-their bot would hit it nearly every hour...and it was killing my bandwidth.

Later,
Sean

... It's not whether you win or lose -- it's how you look when you play the game.

--- GoldED/2 3.0.1
* Origin: Nocturnal State BBS - (423) 926-7999 - bbs.kd5col.info (1:18/200)

Hi Sean,

I should just kill them period even on port 80 for apache, and I still
may do so <grin>. I have free-find indexing my site automatically,
and they do it in such a way that it doesn't interfere with anything..
so it's not like google-bot couldn't do it that way if they wanted to!

I had to block that Chinese search engine, Baidu, from my site-their bot would
hit it nearly every hour...and it was killing my bandwidth.

I'm not familiar with that one.. I should do a grep for them to see if they've hit over here as well <grin>

Hey, last night I had the greatest chat with a user on my system.. he's from Zone 6 (he's in Tokyo) - and he wonders what happened to Z6.. sad :( Well, he is looking to get it going again.. Super I think :)

Take care,
Janis

--- BBBS/LiI v4.01 Flag
* Origin: Prism bbs (1:261/38)

Janis,

so the idiots at google-bot were trying anyway they could..
I laughed when I saw them try the port on 24555 (my bbbs binkp port)..

Are you sure that it was actually googlebot? It only follows http links, which won't lead to that binkp port... (How'd the IP show up in a reverse dns?)

Also; do you have a robots.txt? I use it to disallow "/bbbs"...




Jame

--- BBBS/LiI v4.01 Flag
* Origin: Rocasa BBS (1:120/546)

Sean,

Hello, Janis.
Friday August 20 2010 at 14:29, you wrote to All:

!FTP 66.249.65.83/10
!HTTP 66.249.65.83/10
!raw 66.249.65.83/10
!TCPIP 66.249.65.83/10

Good idea; I'll just block the entire block from coming in.

Do keep in mind that the IPs have a tendancy to change...

My web server is professionally hosted so there's no need
for Google to go poking around.

I don't see what the first thing as to do with the second...?




Jame
]

--- BBBS/LiI v4.01 Flag
* Origin: Rocasa BBS (1:120/546)

Hi Jame,

Are you sure that it was actually googlebot? It only follows http links,

I guess since the web version of the bbs is http, it went there on my port 8080.. and from there, there are links on that page for ftp, etc. and yes I'm sure it was google bot. MSN-Bot is another one that likes to run all over my site - I've seen them in my bbbs inet.log as well <grin>.

which won't lead to that binkp port... (How'd the IP show up in a reverse dns?)

I'm sure it was Google-bot :)

janis:~$ host 66.249.71.53
53.71.249.66.in-addr.arpa domain name pointer crawl-66-249-71-53.googlebot.com.

That is for sure the google spider :)

Also; do you have a robots.txt? I use it to disallow "/bbbs"...

I haven't put that in there, but I will.. Should work, though there's no need right now. Good idea for the bbs, though, thanks :)

Google called me on the phone quite a while back and their representative asked
me to allow Google-bot to index my main site that runs apache, with robots.txt - I don't mind that since it doesn't involve the bbs nodes.. this is how I allow them to scan my main site on port 80:

User-agent: *
Disallow: /
User-agent: Googlebot
Allow: /

Take care,
Janis

--- BBBS/LiI v4.01 Flag
* Origin: Prism bbs (1:261/38)

Hello, Janis.

Sunday August 22 2010 at 11:56, you wrote to me:

I'm not familiar with that one.. I should do a grep for them to see if they've hit over here as well <grin>

Yeah, it's called BaiduSearch or BaiduBot; I've seen both of them in my logs before.

Hey, last night I had the greatest chat with a user on my system..
he's from Zone 6 (he's in Tokyo) - and he wonders what happened to
Z6.. sad :( Well, he is looking to get it going again.. Super I think
:)

That's great! It'd be nice to see Z6 up and running again.

Later,
Sean

... A good scare is worth more to a man than good advice. - E.E. Howe
--- GoldED/W32 3.0.1
* Origin: Nocturnal State BBS - (423) 926-7999 - bbs.kd5col.info (1:18/200)

Hello, Jame.

Sunday August 22 2010 at 08:46, you wrote to me:

Do keep in mind that the IPs have a tendancy to change...

Yep and I can ban them also by domain name or my system will look it up if need
be. I've got three firewalls on my LAN, so if one doesn't catch it, one of the
others will.

My web server is professionally hosted so there's no need
for Google to go poking around.

I don't see what the first thing as to do with the second...?

Everything. I don't run a web server here so Google has no need to go probing the other ports on my LAN. If port 80 is closed, their bot should move on.

Later,
Sean

... For success: try aspiration, inspiration and perspiration.
--- GoldED/W32 3.0.1
* Origin: Nocturnal State BBS - (423) 926-7999 - bbs.kd5col.info (1:18/200)

Hi Sean,

I'm not familiar with that one.. I should do a grep for them to see if
they've hit over here as well <grin>

Yeah, it's called BaiduSearch or BaiduBot; I've seen both of them in my logs before.

Wow Baidubot is surely hitting on BBBS' webserver over here :( Oh well.. off to inet.bbb to get rid of them haha I mean, I really don't mind if they hit the main web server here.. but bbbs is limited to only those 7 nodes.

Hey, last night I had the greatest chat with a user on my system..
he's from Zone 6 (he's in Tokyo) - and he wonders what happened to
Z6.. sad :( Well, he is looking to get it going again.. Super I think
:)

That's great! It'd be nice to see Z6 up and running again.

Yeah really :) He's got some friends who'll be coming in, and if we count the z6 folks that we've got listed in Z3 right now, it should make a good basis for
reinstating that zone.

Take care,
Janis

--- BBBS/LiI v4.01 Flag
* Origin: Prism bbs (1:261/38)

I'm not familiar with that one.. I should do a grep for them to see if
they've hit over here as well <grin>

Yeah, it's called BaiduSearch or BaiduBot; I've seen both of them in
my logs before.

Wow Baidubot is surely hitting on BBBS' webserver over here :( Oh
well.. off to inet.bbb to get rid of them haha

you could just deny them access to those links in BBBS that lead to places like
the messages, files and games... they are robots.txt friendly...

I mean, I really don't mind if they hit the main web server here..
but bbbs is limited to only those 7 nodes.

i assume by that you mean that it is similar to apache in that there's seven HTTP handlers that are allowed to run and you don't allow any more than that??

Hey, last night I had the greatest chat with a user on my system..
he's from Zone 6 (he's in Tokyo) - and he wonders what happened to
Z6.. sad :( Well, he is looking to get it going again.. Super I think
:)

That's great! It'd be nice to see Z6 up and running again.

Yeah really :) He's got some friends who'll be coming in, and if
we count the z6 folks that we've got listed in Z3 right now, it
should make a good basis for reinstating that zone.

that might be a good thing... especially considering the reasons why Z6 went away last time it was operational...

)\/(ark


* Origin: (1:3634/12)

Wow Baidubot is surely hitting on BBBS' webserver over here :( Oh
well.. off to inet.bbb to get rid of them haha

you could just deny them access to those links in BBBS that lead to
places like the messages, files and games... they are robots.txt friendly...

just to follow up and clarify this... an example is my gallery site... i may not want the spiders trapsing thru the exhibits and looking at the actual photos available... only the microthumbs and thumbs, sure so i just set up a disallow for the exhibits area and let them wander over the others... then, if i see them in the exhibits areas, then i know that they're not following robots.txt and i can then contact their admins with my complaint or just block them at the perimeter and deny them access to everything... depending on my mood, of course ;)

)\/(ark


* Origin: (1:3634/12)

Hi Mark,

Yeah, it's called BaiduSearch or BaiduBot; I've seen both of them in
my logs before.

Wow Baidubot is surely hitting on BBBS' webserver over here :( Oh
well.. off to inet.bbb to get rid of them haha

you could just deny them access to those links in BBBS that lead to places lik
the messages, files and games... they are robots.txt friendly...

Well, that's not exactly the problem.. the content is so not 'private' in other
words.. but when they tie up my bbbs web/telnet/binkp nodes, 'real fido people'
can't connect.. that is a drag <g>

I mean, I really don't mind if they hit the main web server here..
but bbbs is limited to only those 7 nodes.

i assume by that you mean that it is similar to apache in that there's seven
HTTP handlers that are allowed to run and you don't allow any more than that??

Yes, that's right, but it's not that I don't allow any more than 7, it's because the way bbbs works is you register the number of nodes you want for the
bbbs daemons.. so I have 7 nodes registered... that's 1 phone-modem node, 6 http instances, 6 telnet nodes, 6 binkp nodes, etc. I also run BinkD stand-alone mailer on the standard binkp port to pick up more binkp connections
since I know the bbbs binkp nodes get a bit busy with the number of downlinks here <g>. BBBS's binkp daemon runs on port 24555.

I guess what really bugged me was that these spiders were hitting all the ports
here.. not just 80 :( I mean, what could a spider get out of attemping repeated connections to my binkp port on 24555?? <bg>

Great thing this week: when I contacted Kim Heino (he's the author of bbbs) about my dead motherboard and all that, etc., (he uses ftp to connect so we needed to set that up on the main ftp server), he sent me a beta 64 bit version
of BBBS.. Really cool :) It's running really well.

That's great! It'd be nice to see Z6 up and running again.

Yeah really :) He's got some friends who'll be coming in, and if
we count the z6 folks that we've got listed in Z3 right now, it
should make a good basis for reinstating that zone.

that might be a good thing... especially considering the reasons why Z6 went away last time it was operational...

Understand.. these kinds of things take time, but we're hoping.

Take care,
Janis

--- BBBS/Li6 v4.10 Dada-1
* Origin: Prism bbs (1:261/38)

Hi Mark,

just to follow up and clarify this... an example is my gallery site... i may not want the spiders trapsing thru the exhibits and looking at the actual photos available... only the microthumbs and thumbs, sure so i just set up a

Sure, because they are all on the web, and you want some control there.. I understand that :)

disallow for the exhibits area and let them wander over the others... then, if
i see them in the exhibits areas, then i know that they're not following
robots.txt and i can then contact their admins with my complaint or just block
them at the perimeter and deny them access to everything... depending on my mood, of course ;)

<laugh>.. now you sound like me :) :) But seriously, that week that they were hitting BBBS's ports was kind of insane to me <g>

Take care,
Janis

--- BBBS/Li6 v4.10 Dada-1
* Origin: Prism bbs (1:261/38)

Wow Baidubot is surely hitting on BBBS' webserver over here :( Oh
well.. off to inet.bbb to get rid of them haha

you could just deny them access to those links in BBBS that lead to
places like the messages, files and games... they are robots.txt friendly...

Well, that's not exactly the problem.. the content is so not
'private' in other words.. but when they tie up my bbbs
web/telnet/binkp nodes, 'real fido people' can't connect.. that is
a drag <g>

yeah, i can see that with a server that has limited handlers allowed... but the
idea was to limit what they have access to so they're in and out as fast as possible ;)

I mean, I really don't mind if they hit the main web server here..
but bbbs is limited to only those 7 nodes.

i assume by that you mean that it is similar to apache in that
there's seven HTTP handlers that are allowed to run and you don't
allow any more than that??

Yes, that's right, but it's not that I don't allow any more than 7,
it's because the way bbbs works is you register the number of nodes
you want for the bbbs daemons.. so I have 7 nodes registered...
that's 1 phone-modem node, 6 http instances, 6 telnet nodes, 6
binkp nodes, etc.

yup... pretty much the same idea... close enough for the analogy ;)

I also run BinkD stand-alone mailer on the standard binkp port to
pick up more binkp connections since I know the bbbs binkp nodes
get a bit busy with the number of downlinks here <g>. BBBS's
binkp daemon runs on port 24555.

i'd really hate it if the bots started hitting the telnet and binkd stuff... i think that google does the http and ftp stuff now... i know i've seen references to it in my ftp logs somewhere...

I guess what really bugged me was that these spiders were hitting
all the ports here.. not just 80 :( I mean, what could a spider
get out of attemping repeated connections to my binkp port on
24555?? <bg>

trying to connect to a web server that it thinks is running there... if it is doing that, lodge a complaint and/or block it at the perimeter and don't let it
in at all ;) depending on the methods, it could be blocked at the perimeter for
all ports except for 80 :P

Great thing this week: when I contacted Kim Heino (he's the
author of bbbs) about my dead motherboard and all that, etc., (he
uses ftp to connect so we needed to set that up on the main ftp
server), he sent me a beta 64 bit version of BBBS.. Really cool :)
It's running really well.

i saw reference to you running 64bit earlier... that's a GoodThing<tm> ;)

That's great! It'd be nice to see Z6 up and running again.

Yeah really :) He's got some friends who'll be coming in, and if
we count the z6 folks that we've got listed in Z3 right now, it
should make a good basis for reinstating that zone.

that might be a good thing... especially considering the reasons why
Z6 went away last time it was operational...

Understand.. these kinds of things take time, but we're hoping.

word up! :P

)\/(ark


* Origin: (1:3634/12)

Hi Mark,

'private' in other words.. but when they tie up my bbbs
web/telnet/binkp nodes, 'real fido people' can't connect.. that is
a drag <g>

yeah, i can see that with a server that has limited handlers allowed... but th
idea was to limit what they have access to so they're in and out as fast as possible ;)

I think bbbs makes it more than quick.. they connect, they get immediately rejected <g>

I mean, I really don't mind if they hit the main web server here..
but bbbs is limited to only those 7 nodes.

i assume by that you mean that it is similar to apache in that
there's seven HTTP handlers that are allowed to run and you don't
allow any more than that??

Yes, that's right, but it's not that I don't allow any more than 7,
it's because the way bbbs works is you register the number of nodes
you want for the bbbs daemons.. so I have 7 nodes registered...
that's 1 phone-modem node, 6 http instances, 6 telnet nodes, 6
binkp nodes, etc.

yup... pretty much the same idea... close enough for the analogy ;)

Yep, for sure :)

pick up more binkp connections since I know the bbbs binkp nodes
get a bit busy with the number of downlinks here <g>. BBBS's
binkp daemon runs on port 24555.

i'd really hate it if the bots started hitting the telnet and binkd stuff... i
think that google does the http and ftp stuff now... i know i've seen references to it in my ftp logs somewhere...

for the main ftp server, again it's no problem since I don't allow anonymous ftp on my vsftp server..but on bbb's ftp server anonymous is allowed.. not working well for people from what I understand but.. have to look into that :)

I guess what really bugged me was that these spiders were hitting
all the ports here.. not just 80 :( I mean, what could a spider
get out of attemping repeated connections to my binkp port on
24555?? <bg>

trying to connect to a web server that it thinks is running there... if it is
doing that, lodge a complaint and/or block it at the perimeter and don't let i >in at all ;) depending on the methods, it could be blocked at the perimeter fo
all ports except for 80 :P

ah one of these days.. no time for it right now.. just trying to catch up <bg>

Great thing this week: when I contacted Kim Heino (he's the
author of bbbs) about my dead motherboard and all that, etc., (he
uses ftp to connect so we needed to set that up on the main ftp
server), he sent me a beta 64 bit version of BBBS.. Really cool :)
It's running really well.

i saw reference to you running 64bit earlier... that's a GoodThing<tm> ;)

It's a really nice system.. when the conexant internmodem started locking up the system, I decided to back up to the release 32bit version of bbbs.. in case
it was causing problems as well.. for now I'll just let it sit as is <g>.. one of these days I'll reinstall the 64bit version of bbbs..

That's great! It'd be nice to see Z6 up and running again.

Yeah really :) He's got some friends who'll be coming in, and if
we count the z6 folks that we've got listed in Z3 right now, it
should make a good basis for reinstating that zone.

that might be a good thing... especially considering the reasons why
Z6 went away last time it was operational...

Understand.. these kinds of things take time, but we're hoping.

word up! :P

I'll say :)

Take care,
Janis

--- BBBS/LiI v4.01 Flag
* Origin: Prism bbs (1:261/38)

Hello Janis!

21 Aug 2010 11:58, Janis Kracht wrote to Sean Dennis:

with anything.. so it's not like google-bot couldn't do it that way
if they wanted to!

time to get a mirror :)

but:

----- .htaccess begins -----
RewriteEngine on

# Allow only GET and POST verbs
RewriteCond %{REQUEST_METHOD} !^(GET|POST)$ [NC,OR]

# Ban Typical Vulnerability Scanners and others
# Kick out Script Kiddies
RewriteCond %{HTTP_USER_AGENT} ^(java|curl|wget).* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^.*(libwww-perl|curl|wget|python|nikto|wkito|pikto|scan|acunetix).* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^.*(winhttp|HTTrack|clshttp|archiver|loader|email|harvest|extract|grab|miner). * [NC,OR]

# Ban Search Engines, Crawlers to your administrative panel
# No reasons to access from bots
# Ultimately Better than the useless robots.txt
# Did google respect robots.txt?
# Try google: intitle:phpMyAdmin intext:"Welcome to phpMyAdmin *.*.*" intext:"Log in" -wiki -forum -forums -questions intext:"Cookies must be enabled"
RewriteCond %{HTTP_USER_AGENT} ^.*(AdsBot-Google|ia_archiver|Scooter|Ask.Jeeves|Baiduspider|Exabot|FAST.Enter prise.Crawler|FAST-WebCrawler|www\.neomo\.de|Gigabot|Mediapartners-Google|Goog le.Desktop|Feedfetcher-Google|Googlebot|heise-IT-Markt-Crawler|heritrix|ibm.co m\cs/crawler|ICCrawler|ichiro|MJ12bot|MetagerBot|msnbot-NewsBlogs|msnbot|msnbo t-media|NG-Search|lucene.apache.org|NutchCVS|OmniExplorer_Bot|online.link.vali dator|psbot0|Seekbot|Sensis.Web.Crawler|SEO.search.Crawler|Seoma.\[SEO.Crawler \]|SEOsearch|Snappy|www.urltrends.com|www.tkl.iis.u-tokyo.ac.jp/~crawler|Synoo Bot|crawleradmin.t-info@telekom.de|TurnitinBot|voyager|W3.SiteSearch.Crawler|W 3C-checklink|W3C_Validator|www.WISEnutbot.com|yacybot|Yahoo-MMCrawler|Yahoo\!. DE.Slurp|Yahoo\!.Slurp|YahooSeeker).* [NC]
RewriteRule .* - [F]

----- .htaccess ends -----

if bbbs http server cant do this make it as proxy in apache !


Regards Benny

... there can only be one way of life, and it works :)

--- Msged/LNX 6.2.0 (Linux/2.6.34-gentoo-r6 (i686))
* Origin: www.region23.dk/ www.fido.dk/ <128000K ram after bootup (2:237/53)

Hi Benny,

with anything.. so it's not like google-bot couldn't do it that way
if they wanted to!

time to get a mirror :)

Hehe.. Yep, very good idea as we discussed. The more mirrors the better.. That
way my system won't get so boggled down :)

but:

----- .htaccess begins -----
RewriteEngine on

# Allow only GET and POST verbs
RewriteCond %{REQUEST_METHOD} !^(GET|POST)$ [NC,OR]

# Ban Typical Vulnerability Scanners and others
# Kick out Script Kiddies
RewriteCond %{HTTP_USER_AGENT} ^(java|curl|wget).* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^.*(libwww-perl|curl|wget|python|nikto|wkito|pi
to|scan|acunetix).* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^.*(winhttp|HTTrack|clshttp|archiver|loader|ema
l|harvest|extract|grab|miner). * [NC,OR]

# Ban Search Engines, Crawlers to your administrative panel
# No reasons to access from bots
# Ultimately Better than the useless robots.txt
# Did google respect robots.txt?
# Try google: intitle:phpMyAdmin intext:"Welcome to phpMyAdmin *.*.*" intext:"Log in" -wiki -forum -forums -questions intext:"Cookies must be enabled"
RewriteCond %{HTTP_USER_AGENT} ^.*(AdsBot-Google|ia_archiver|Scooter|Ask.Jeeve >|Baiduspider|Exabot|FAST.Enter prise.Crawler|FAST-WebCrawler|www\.neomo\.de|Gi >abot|Mediapartners-Google|Goog le.Desktop|Feedfetcher-Google|Googlebot|heise-I >-Markt-Crawler|heritrix|ibm.co m\cs/crawler|ICCrawler|ichiro|MJ12bot|MetagerBo >|msnbot-NewsBlogs|msnbot|msnbo t-media|NG-Search|lucene.apache.org|NutchCVS|Om >iExplorer_Bot|online.link.vali dator|psbot0|Seekbot|Sensis.Web.Crawler|SEO.sea >ch.Crawler|Seoma.\[SEO.Crawler \]|SEOsearch|Snappy|www.urltrends.com|www.tkl.i >s.u-tokyo.ac.jp/~crawler|Synoo Bot|crawleradmin.t-info@telekom.de|TurnitinBot| >oyager|W3.SiteSearch.Crawler|W 3C-checklink|W3C_Validator|www.WISEnutbot.com|y
cybot|Yahoo-MMCrawler|Yahoo\!. DE.Slurp|Yahoo\!.Slurp|YahooSeeker).* [NC] RewriteRule .* - [F]

----- .htaccess ends -----

if bbbs http server cant do this make it as proxy in apache !

Sure, it can .. :) I have always used mod-rewrite to ban filehogs from places like those where I find companies that are selling the files <g>.

I like the way you used it though :) Thanks for a good idea.

Take care,
Janis

--- BBBS/LiI v4.01 Flag
* Origin: Prism bbs (1:261/38)