Subj : Last couple of days with my binkp mailer..
To   : Janis Kracht
From : Benny Pedersen
Date : Sat Oct 16 2010 03:05 pm

Hello Janis!

21 Aug 2010 11:58, Janis Kracht wrote to Sean Dennis:

 JK> with anything.. so it's not like google-bot couldn't do it that way 
 JK> if they wanted to!

time to get a mirror :)

but:

   ----- .htaccess begins -----
RewriteEngine on

# Allow only GET and POST verbs
RewriteCond %{REQUEST_METHOD} !^(GET|POST)$ [NC,OR]

# Ban Typical Vulnerability Scanners and others
# Kick out Script Kiddies
RewriteCond %{HTTP_USER_AGENT} ^(java|curl|wget).* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^.*(libwww-perl|curl|wget|python|nikto|wkito|pikto|scan|acunetix).* [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^.*(winhttp|HTTrack|clshttp|archiver|loader|email|harvest|extract|grab|miner). * [NC,OR]

# Ban Search Engines, Crawlers to your administrative panel
# No reasons to access from bots
# Ultimately Better than the useless robots.txt
# Did google respect robots.txt?
# Try google: intitle:phpMyAdmin intext:"Welcome to phpMyAdmin *.*.*" 
intext:"Log in" -wiki -forum -forums -questions intext:"Cookies must be 
enabled"
RewriteCond %{HTTP_USER_AGENT} ^.*(AdsBot-Google|ia_archiver|Scooter|Ask.Jeeves|Baiduspider|Exabot|FAST.Enter prise.Crawler|FAST-WebCrawler|www\.neomo\.de|Gigabot|Mediapartners-Google|Goog le.Desktop|Feedfetcher-Google|Googlebot|heise-IT-Markt-Crawler|heritrix|ibm.co m\cs/crawler|ICCrawler|ichiro|MJ12bot|MetagerBot|msnbot-NewsBlogs|msnbot|msnbo t-media|NG-Search|lucene.apache.org|NutchCVS|OmniExplorer_Bot|online.link.vali dator|psbot0|Seekbot|Sensis.Web.Crawler|SEO.search.Crawler|Seoma.\[SEO.Crawler \]|SEOsearch|Snappy|www.urltrends.com|www.tkl.iis.u-tokyo.ac.jp/~crawler|Synoo Bot|crawleradmin.t-info@telekom.de|TurnitinBot|voyager|W3.SiteSearch.Crawler|W 3C-checklink|W3C_Validator|www.WISEnutbot.com|yacybot|Yahoo-MMCrawler|Yahoo\!. DE.Slurp|Yahoo\!.Slurp|YahooSeeker).* [NC]
RewriteRule .* - [F]

   ----- .htaccess ends -----

if bbbs http server cant do this make it as proxy in apache !
 

 Regards Benny

.... there can only be one way of life, and it works :)

--- Msged/LNX 6.2.0 (Linux/2.6.34-gentoo-r6 (i686))
 * Origin: www.region23.dk/ www.fido.dk/ <128000K ram after bootup (2:237/53)

.