Web Spider Traps

When an author does not want his site to be copied or indexed by search engines, he can use:

1. A meta tag as <meta name="robots" content="noindex,nofollow"> (well-behaved bots only).
2. A robots.txt file which indicates the parts of the site not to be explored (well-behaved bots only).
3. .htaccess to ban known or detected robots (any webbot).
4. A java applet, some html, a script written in php, javascript or any other language (any webbot).

These methods are detailed in English at the following addresses:

• How to build a Bot Trap and keep bad bots away from a web site
• Stopping Spambots: A Spambot Trap
• How to keep bad robots, spiders and web crawlers away Apache only
• E-Mail Protector Script (perl script sending 10,000 fake addresses to identified robots)
• Other methods (How to Defeat Bad Web Robots With Apache, Improving Web Spider Trap Efficiency, Blocking Bad User Agents - avec ASP, Robotcop ...) may still be online and searched.

in the documentation of Httrack (abuse FAQ for webmasters).

and www.webmasterworld.com (search for "spider traps" / "Blocking Badly Behaved Bots" or have a look at www.webmasterworld.com/forum24/ or www.webmasterworld.com/forum88/ or http://www.webmasterworld.com/forum92/ ).

Trap?

All these traps are likely to prevent search engines from indexing the pages, make browsing more difficult and discourage the users.

Fighting against the "Spam harvesters", "email grabbers", "email collectors" and "spambots" can easily be understood and quite easily done, but as all spiders are not used for bad purposes why should they all be blocked, even if they consume bandwidth and sometimes block or overload some sites.
Captures can be done for good reasons and good people: this site tries to help those who mirror sites for their students or those who cannot afford staying online...

Mirror?

Often, after some time, protections are removed: those whose navigators do not have the plugins (Macromedia, java -JRE 6.0-) or do not interpret Javascript are lost readers or lost customers.

If you think that the site is interesting enough to be mirrored, ask the author for a copy that you could browse offline.
Indeed, if you activate the option "no robots.txt rules" you may block any access to the site with your IP address or you may copy hundreds of pages without interest - error pages, images, documentations etc -.

In all the cases, locate the useful folders, use reasonable bandwidth limits and connections per second (Options - Limits - Max transfer rate and Options - Limits - Max connections / second), and limit the number of connections.
Examples 12 and 17 of website mirrors may help you.

Identify a robot

You can read about the different robots identifying themselves here:

• Search Engine Spiders List (site and UA)
• Search Engine IP Addresses (UA and IP)
• Search engine robots that visit your web site (site, UA and IP)
• Search Engine Spider Identification (UA and comments at webmasterworld)
entification des robots in French (site, UA and IP)
• E-Mail Collectors List (site and UA)
• For this site : Listed identities (1021 User Agents Strings)

• - None of these sites gives an entire list.
• - Most robots and spiders give MSIE User Agent:
  Mozilla/4.0 (compatible; MSIE 6.0; Windows ...)
  do not read robots.txt
  and are not well-behaved...
• - Robots that regularly request robots.txt (UA).
• - Robots that requested robots.txt this year(UA).

Robots and this site

List of the robots visiting this site (this list indexes the site, tests the link to the site, does surveys or controls for clients' names, plagiarism, spam...):

"1Noonbot search engine" - "50.nu" - "80legs crawler" - "ABACHOBot search engine" - "abcfr_robot search engine" - "Accoona-AI-Agent search engine" - "AcoonBot search engine" - "ActiveBookmark" - "Advanced URL Catalog bookmark manager" - "Advista search engine" - "aiHitBot" - "aipbot search engine" - "alef" - "Aleksika search engine" - "amagit.com search engine" - "Amazonbot crawler" - "Amfibibot search engine" - "Anonymous / Skywalker" - "AnswerBus search engine" - "AntBot search engine" - "antibot crawler" - "appie 1.1 (www.walhello.com) search engine" - "Apple-PubSub RSS monitoring" - "archive.org_bot crawler" - "Argus bookmark managing crawler" - "Art-Online.com 0.9(Beta) crawler" - "Ask Jeeves crawler" - "Asterias crawler" - "atraxbot" - "Baiduspider search engine" - "Bazbot search engine" - "BecomeBot search engine" - "Big Fish %S" - "Biglotron search engine" - "bingbot crawler" - "binlar" - "bitlybot" - "BitSightBot" - "bixolabs  Data Mining" - "BlackMask.Net search engine" - "BlogCorpusCrawler" - "Bloglines RSS monitoring" - "Bluebot crawler" - "BnF" - "bogospider" - "boitho.com-robot search engine" - "Bookdog bookmark manager" - "bot/1.0" - "botmobi search engine" - "BruinBot crawler" - "Butterfly search engine" - "BuzzRankingBot crawler" - "C4PC" - "CacheBot" - "Caliperbot" - "CamontSpider crawler" - "capek crawler" - "Casper Bot Search %Z" - "CatchBot crawler" - "CazoodleBot crawler" - "CCBot crawler" - "ccubee search engine" - "CentiverseBot search engine" - "cfetch" - "Chanceo %S" - "Charlotte search engine" - "Cherchonsbot search engine" - "Cityreview" - "CMS Crawler" - "Combine crawler" - "comBot search engine" - "cometsystems crawler" - "Content Crawler crawler" - "ContextAd Bot" - "Convera  RetrievalWare" - "CorenSearchBot" - "Corpora from the web crawler" - "Cosmix crawler" - "CosmixCrawler search engine" - "Covario crawler" - "Crawl Annu" - "Crawllybot search engine" - "csci_b659  Data Mining" - "CSS/HTML/XTHML  Validator" - "CSSCheck" - "cybercity.dk IE 5.5 Compatible Browser" - "CydralSpider search engine" - "darxi spam / email grabbing" - "DataForSEO Link Bot" - "DataFountains/DMOZ Downloader" - "DAUM Web Robot search engine" - "dcbspider search engine" - "DealGates" - "Declumbot" - "deepak-USC/ISI  spider" - "del.icio.us-thumbnails" - "del.icio.us  bookmark manager link checker" - "DepSpid crawler" - "Diamond search engine" - "Diffbot" - "Directcrawler" - "discobot crawler" - "DLE_Spider spam" - "DMOZ Experiment" - "DNSGroup crawler" - "Domains Project crawler" - "DotBot crawler" - "DTAAgent search engine" - "Dumbot search engine" - "e-SocietyRobot crawler" - "eApolloBot search engine" - "EasyDL/3.04" - "EdisterBot crawler" - "ejupiter.com search engine" - "ellerdale search engine" - "EnaBot crawler" - "envolk search engine" - "ePochta_Extractor spam / email grabbing" - "ETS  translation bot" - "europarchive" - "Exabot crawler" - "Exabot-Thumbnails" - "exactseek-crawler-2.63" - "Exalead NG" - "exooba crawler" - "Ezooms" - "facebookexternalhit" - "Factbot search engine" - "Falconsbot search engine" - "FAST crawler" - "FAST Enterprise Crawler" - "FAST FirstPage retriever" - "fast-search-engine" - "FAST-WebCrawler" - "FAST MetaWeb Crawler" - "FavOrg Link checker" - "favorstarbot Advertising" - "FeedBot" - "FeedBurner" - "FeedFetcher-Google" - "Fetch API Request" - "Filangy bookmark managing crawler" - "Findexa crawler" - "findfiles.net search engine" - "findlinks" - "flatlandbot" - "fleck" - "Flight Deck" - "FlightDeckReports" - "Fluffy (searchhippo) search engine" - "flyindex search engine" - "Focal crawler" - "FollowSite" - "Friend search engine" - "FurlBot search engine" - "Gaisbot/3.0 search engine" - "Galbot crawler" - "Generalbot" - "genevabot search engine" - "geniebot search engine" - "GeoBot" - "Gigabot crawler" - "Gigamega.bot search engine" - "GingerCrawler" - "Girafabot" - "Gnomit crawler" - "GOFORITBOT search engine" - "gold crawler" - "Google Desktop RSS monitoring" - "Google-Site-Verification" - "Google-Sitemaps" - "Googlebot crawler" - "Googlebot-Image" - "Googlebot-Mobile" - "Google Web Preview" - "GPTBot crawler" - "grub search engine" - "grub crawler" - "grub.org" - "gsa-crawler" - "GT::WWW/1." - "gURLChecker Link checker" - "GurujiBot search engine" - "GUSbot" - "GVC-SPIDER" - "Hailoobot search engine" - "Haste" - "hclsreport crawler" - "Helix crawler" - "HenriLeRobotMirago crawler" - "Heritrix crawler" - "hoge" - "Holmes search engine" - "HooWWWer crawler" - "htdig" - "HuaweiSymantecSpider crawler" - "ia_archiver crawler" - "ICC-Crawler crawler" - "ichiro search engine" - "icsbot-0.1" - "IlTrovatore search engine" - "imbot" - "INA dlweb crawler" - "IndoCrew %Z" - "Indy Library  Internet Direct Library for Borland - often spambot" - "InelaBot crawler" - "inet library" - "inktomi Slurp crawler" - "InsiteRobot" - "integromedb.org crawler" - "InternetSeer Connectivity checker" - "Interseek" - "IntranooBot" - "IP*Works Link checker" - "IRLbot crawler" - "iSearch search engine" - "istarthere search engine" - "IXE Crawler" - "Jakarta Commons" - "Jetbot/1.0 crawler" - "JungleKeyBot search engine" - "Jyxobot search engine" - "KaloogaBot search engine" - "Killou.com search engine" - "KiwiStatus search engine" - "kmccrew Bot Search %Z" - "Knowledge.com search engine" - "knowmore" - "KomodiaBot" - "Lachesis" - "larbin crawler" - "ldspider" - "leak" - "lemurwebcrawler" - "librabot search engine" - "libwww-perl" - "LinguaBot search engine" - "linkaGoGo crawler" - "LinkChecker" - "Link Commander bookmark manager" - "linkdex.com" - "Linkman Link checker" - "Links SQL" - "Link Valet Online Link checker" - "LiteFinder search engine" - "livemark.jp Link checker" - "lmspider crawler" - "Look.com search engine" - "Loopy.fr search engine" - "Loserbot" - "Lsearch/sondeur" - "lwp-request" - "lwp-trivial" - "LWP::Simple" - "MagpieRSS" - "Mail.Ru" - "MaMa CaSpEr %Z" - "MaMa CyBer %Z" - "MapoftheInternet search engine" - "Marvin search engine" - "Me.dium OneRiot crawler" - "Mediapartners-Google" - "Megaglobe search engine" - "Megite  news aggregator" - "MetaGeneratorCrawler" - "Metaspinner search engine" - "MileNSbot search engine" - "Mirago (HenriLeRobot) crawler" - "MJ12bot crawler" - "MLBot" - "MnogoSearch/3.2.11" - "MojeekBot search engine" - "Monrobot crawler" - "MOSBookmarks Link checker" - "mozDex crawler" - "Mozilla/4.0 (compatible; MSIE 6.0)" - "Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.0;)" - "Mp3Bot search engine" - "MQbot crawler" - "ms research robot" - "MSIE 4.5 %S" - "MSIE 6.0 (compatible; MSIE 6.0;... %S" - "MSIE 7.01 %S" - "MSMOBOT crawler" - "msnbot crawler" - "MSNPTC  MSN search robot" - "MSR-ISRCCrawler" - "MSRBOT crawler" - "MultiCrawler search engine" - "mxbot" - "MyFamilyBot crawler" - "Nambu" - "NaverBot search engine" - "NaverRobot search engine" - "Nelian Pty Ltd" - "Netcraft survey" - "netEstate crawler" - "NetID Bot Advertising" - "NetResearchServer search engine" - "NetSprint search engine" - "NetWhatCrawler search engine" - "newsg8 RSS monitoring" - "NEWT ActiveX spam / email grabbing" - "NG-Search search engine" - "NG/1.0" - "NG/2.0 crawler" - "NGBot crawler" - "nicebot" - "Nigma search engine" - "NimbleCrawler search engine" - "NjuiceBot" - "Norbert the Spider search engine" - "NoteworthyBot" - "NPBot  NameProtect crawler" - "nrsbot search engine" - "NuSearch Spider search engine" - "Nutch crawler" - "Nutch (Princeton) crawler" - "ObjectsSearch search engine" - "oBot crawler" - "octopodus search engine" - "Octora crawler" - "ODP::/0.01 Link checker" - "ODP entries" - "ODP links test" - "OmniExplorer_Bot search engine" - "onalytica" - "onCHECK" - "OnetSzukaj search engine" - "OOZBOT search engine" - "Openbot search engine" - "OpenindexSpider" - "OpenISearch search engine" - "OpenTaggerBot  social bookmarks" - "OpenX Spider Advertising" - "OrangeBot-Mobile search engine" - "OutfoxBot" - "ozelot" - "page-store" - "Pagebull search engine" - "pagepeeker" - "page_verifier" - "Paleoweb crawler" - "PanopeaBot/1.0 (UCLA CS Dpt.)" - "panopta.com Connectivity checker" - "Pathtraq search engine" - "PEERbot search engine" - "PeerFactor crawler" - "petalbot crawler" - "Pete-Spider crawler" - "pflab" - "PHP/4." - "PHP version tracker web stats" - "PicSpider" - "PipeLine  spider" - "Pita crawler" - "plaNETWORK Bot Search %Z" - "Plukkie search engine" - "PollettSearch crawler" - "polybot crawler" - "Pompos - dir.com crawler" - "Popdexter crawler" - "PostFavorites" - "PostRank" - "Powermarks Link checker" - "PrivacyFinder search engine" - "PROBE! search engine" - "Program Shareware" - "psbot crawler" - "Python-urllib" - "QEAVis" - "QihooBot search engine" - "Qualidator.com Bot" - "quickobot crawler" - "RAMPyBot search engine" - "RankurBot" - "Rapid-Finder search engine" - "Reaper/2.06 search engine" - "RedBot crawler" - "RedCarpet" - "RixBot search engine" - "robotgenius  malware detection?" - "Robozilla/1.0" - "RSSMicro search engine" - "RTGI  Data Mining" - "RufusBot" - "sagool search engine" - "savvybot search engine" - "SBIder crawler" - "schibstedsokbot search engine" - "Scooter search engine" - "ScoutJet search engine" - "Scrubby search engine" - "search.updated.com search engine" - "Search17Bot search engine" - "SearchByUsa search engine" - "SearchIt.Bot search engine" - "SearchWebLinks" - "Seekbot crawler" - "Semager search engine" - "SemrushBot" - "Sensis search engine" - "seocompany crawler" - "SEOENGBot" - "SEOprofiler bot crawler" - "SETOOZBOT search engine" - "SeznamBot" - "ShablastBot search engine" - "Shelob" - "sherlock search engine" - "Shim-Crawler" - "ShrinkTheWeb crawler" - "ShunixBot crawler" - "silk search engine" - "Sindup RSS monitoring" - "SISTRIX crawler" - "SiteBot %S" - "SiteIntel.net Bot" - "Skywalker / Anonymous" - "sledink Bot Search %Z" - "Slurpy Verifier" - "snap.com search engine" - "Snapbot search engine" - "SnapPreviewBot" - "socbay search engine" - "sogou spider" - "sohu-search search engine" - "sohu agent search engine" - "Solomono search engine" - "Sosospider search engine" - "SpeedySpider search engine" - "SpiderLing crawler" - "Spinn3r" - "sproose crawler" - "SpurlBot bookmark managing crawler" - "sSearch Crawler" - "statbot" - "StatusCheckBot Link checker" - "Steeler crawler" - "SuperBot search engine" - "Susie  bookmark manager link checker" - "sygol search engine" - "SynapticWalker spam / email grabbing" - "SynooBot search engine" - "Syntryx ANT Chassis crawler" - "Szukacz/1.5 search engine" - "T-H-U-N-D-E-R-S-T-O-N-E" - "TargetYourNews Link checker" - "Teemer" - "Teoma search engine" - "TerraSpider" - "test" - "TFC" - "Theophrastus" - "Thriceler search engine" - "Thumbnail.CZ robot" - "thumbshots-de-bot" - "TinEye crawler" - "TranSGeniKBot" - "trexmod" - "Tubenowbot Link checker" - "TurnitinBot crawler" - "TutorGigBot crawler" - "Tutorial Crawler" - "TweetmemeBot" - "TwengaBot crawler" - "Twiceler crawler" - "Twisted PageGetter" - "Twitterbot" - "Twitturl" - "TygoBot search engine" - "uberbot crawler" - "UnChaosBot search engine" - "Unicorn  Validator" - "updated search engine" - "Update Profile Bot search engine" - "Updownerbot" - "UptimeAuditor Connectivity checker" - "UptimeBot" - "URLBase bookmark manager" - "Valizbot crawler" - "VDSX.nl search engine" - "VelenPublicWebCrawler" - "versus crawler" - "Visbot search engine" - "VoilaBot crawler" - "Voluniabot" - "Vortex crawler" - "voyager search engine" - "VSE/1.0 crawler" - "W3C-checklink" - "WASALive search engine" - "WebAlta crawler" - "WebarooBot crawler" - "WebCorp search engine" - "webcrawl search engine" - "WebFilter" - "WebIndexer search engine" - "WebRACE/1.1" - "Webscan" - "WebsiteWorth %S" - "WebwikiBot" - "wikiwix search engine" - "Willow Internet Crawler" - "Windows-Live-Social-Object-Extractor-Engine" - "WinkBot search engine" - "Winsey search engine" - "WIRE" - "WongBot" - "woorank" - "woriobot search engine" - "WorQmada Link checker" - "Wotbox search engine" - "wpbot crawler" - "wume_crawler" - "www.almaden.ibm.com/cs/crawler" - "www.IsMySiteUp.Net" - "www.pisoc.com search engine" - "Xenu Link checker" - "Xerka  Data Mining" - "xirq search engine" - "XmarksFetch bookmark manager search engine" - "yacybot search engine" - "Yahoo! Slurp crawler" - "Yahoo! Mindset" - "Yahoo-MMCrawler" - "Yahoo-Test crawler" - "YahooSeeker search engine" - "YahooVideoSearch search engine" - "Yandex search engine" - "Yanga search engine" - "yellowJacket Link checker" - "YesupBot" - "Yeti search engine" - "Yooda" - "yoono search engine" - "YottaCars search engine" - "YottaShopping search engine" - "YoudaoBot search engine" - "YRSpider" - "ZeBot search engine" - "zerxbot search engine" - "Zeus search engine" - "Zion crawler" - "ZipppBot search engine" - "ZyBorg/1.0 search engine" - "101.32.115.96" - "101.32.240.178" - "101.32.244.173" - "101.44.160.***" - "101.44.161.***" - "101.44.162.***" - "101.44.163.***" - "101.44.249.247" - "101.44.25.76" - "101.44.66.78" - "101.44.67.183" - "101.44.69.199" - "101.44.70.157" - "101.46.0.9" - "101.46.1.20" - "101.46.14.79" - "101.46.15.123" - "101.46.15.124" - "101.46.2.11" - "101.46.2.41" - "101.46.3.96" - "101.46.5.37" - "101.46.5.43" - "101.46.6.125" - "101.46.6.162" - "101.46.7.72" - "101.46.9.139" - "101.46.9.255" - "101.46.9.56" - "101.47.8.*** crawler" - "102.129.145.***" - "102.129.152.***" - "102.129.235.***" - "103.122.90.219" - "103.139.17.122" - "103.159.188.120" - "103.159.188.123" - "103.204.211.131" - "103.204.211.135" - "103.244.251.164" - "104.207.32.12" - "104.232.221.246" - "104.239.44.*** crawler" - "104.28.209.115" - "107.151.182.54" - "107.151.182.58" - "107.151.182.62" - "108.165.243.23" - "109.196.161.*** crawler" - "109.207.130.*** crawler" - "110.238.104.225" - "110.238.104.227" - "110.238.105.253" - "110.238.106.18" - "110.238.107.231" - "110.238.108.0" - "110.238.108.65" - "110.238.109.***" - "110.238.110.***" - "111.119.192.161" - "111.119.192.35" - "111.119.193.219" - "111.119.193.27" - "111.119.195.12" - "111.119.195.232" - "111.119.201.14" - "111.119.201.71" - "111.119.202.91" - "111.119.203.112" - "111.119.203.244" - "111.119.204.117" - "111.119.204.60" - "111.119.206.108" - "111.119.206.172" - "111.119.207.190" - "111.119.207.228" - "111.119.209.115" - "111.119.209.13" - "111.119.209.60" - "111.119.210.***" - "111.119.211.249" - "111.119.211.73" - "111.119.213.142" - "111.119.213.223" - "111.119.214.105" - "111.119.215.27" - "111.119.216.126" - "111.119.216.17" - "111.119.216.36" - "111.119.217.17" - "111.119.219.138" - "111.119.220.232" - "111.119.221.109" - "111.119.221.9" - "111.119.222.127" - "111.119.222.231" - "111.119.222.5" - "111.119.233.***" - "111.119.235.123" - "111.119.235.247" - "111.119.236.147" - "111.119.237.139" - "111.119.237.6" - "111.119.237.63" - "111.119.240.188" - "111.119.243.17" - "111.119.243.67" - "111.119.250.125" - "111.119.253.115" - "114.115.158.156" - "114.115.159.156" - "114.115.175.113" - "114.116.110.95" - "114.119.175.154" - "114.119.181.242" - "114.119.183.4" - "114.119.184.239" - "114.119.185.250" - "114.119.186.247" - "114.119.187.77" - "114.119.189.133" - "114.119.191.215" - "116.202.182.111" - "116.204.117.83" - "116.204.119.227" - "116.204.120.56" - "116.204.122.132" - "116.204.77.47" - "117.50.193.39" - "117.50.197.145" - "117.50.47.189" - "117.50.47.205" - "119.13.100.***" - "119.13.101.100" - "119.13.101.155" - "119.13.101.171" - "119.13.102.90" - "119.13.104.14" - "119.13.104.140" - "119.13.105.163" - "119.13.105.238" - "119.13.105.88" - "119.13.106.117" - "119.13.106.181" - "119.13.106.71" - "119.13.107.222" - "119.13.107.54" - "119.13.108.132" - "119.13.110.181" - "119.13.110.84" - "119.13.91.46" - "119.15.83.218" - "119.8.127.117" - "119.8.162.110" - "119.8.163.102" - "119.8.163.75" - "119.8.166.250" - "119.8.169.96" - "119.8.182.195" - "119.8.190.174" - "119.8.190.185" - "119.8.190.246" - "119.8.232.212" - "119.8.33.67" - "121.237.36.***" - "121.37.105.39" - "122.161.49.56" - "122.8.185.67" - "122.8.186.91" - "122.96.30.65" - "124.243.132.***" - "124.243.133.***" - "124.243.134.***" - "124.243.135.***" - "124.243.136.***" - "124.243.137.***" - "124.243.138.***" - "124.243.139.***" - "124.243.144.***" - "124.243.145.***" - "124.243.146.***" - "124.243.147.***" - "124.243.148.***" - "124.243.149.***" - "124.243.150.***" - "124.243.151.***" - "124.243.168.***" - "124.243.169.132" - "124.243.169.158" - "124.243.169.197" - "124.243.170.150" - "124.243.170.47" - "124.243.171.231" - "124.243.171.48" - "124.243.172.174" - "124.243.172.185" - "124.243.172.191" - "124.243.174.158" - "124.243.174.21" - "124.243.176.105" - "124.243.177.168" - "124.243.177.227" - "124.243.178.***" - "124.243.179.151" - "124.243.180.***" - "124.243.181.***" - "124.243.182.221" - "124.243.182.3" - "124.243.183.***" - "124.243.184.***" - "124.243.185.***" - "124.243.186.***" - "124.243.187.***" - "124.243.189.***" - "124.243.190.***" - "124.243.191.***" - "128.14.159.252" - "128.14.188.106" - "128.14.209.26" - "128.14.209.42" - "128.199.167.175" - "128.241.235.***" - "129.153.237.237" - "129.226.146.19" - "129.226.150.55" - "129.226.151.215" - "129.226.154.196" - "129.226.156.129" - "129.226.158.117" - "129.226.192.111" - "129.226.192.224" - "129.226.193.30" - "129.226.209.118" - "129.226.91.207" - "129.226.92.236" - "129.226.92.4" - "129.80.129.244" - "129.80.228.2" - "129.80.230.124" - "129.80.63.18" - "13.58.11.68" - "13.58.150.59" - "13.58.252.8" - "13.58.82.79" - "13.75.147.105" - "13.90.150.239" - "131.153.142.170 RSS monitoring" - "131.153.143.50 RSS monitoring" - "131.153.240.*** RSS monitoring" - "138.197.139.195" - "138.201.17.*** crawler" - "139.99.62.***" - "141.94.194.132" - "141.94.194.132 RSS monitoring" - "142.44.167.89" - "143.110.250.220" - "143.178.156.15" - "144.76.108.212" - "144.76.14.48" - "146.190.108.97" - "146.190.146.83" - "146.70.189.199" - "146.70.189.207" - "146.70.189.213" - "147.78.181.*** crawler" - "149.102.232.***" - "149.102.246.101" - "149.102.246.43" - "149.102.246.50" - "149.71.246.193" - "15.235.15.135" - "150.107.201.110" - "150.109.11.222" - "150.109.12.106" - "150.109.13.249" - "150.109.17.45" - "150.109.20.64" - "150.109.21.76" - "150.109.23.33" - "150.109.24.245" - "150.109.25.235" - "150.136.246.147" - "150.136.87.92" - "151.252.25.100" - "152.42.245.*** crawler" - "152.53.112.127" - "152.53.112.132" - "152.53.113.1" - "152.53.114.205" - "152.53.16.147" - "152.53.50.50" - "152.53.52.167" - "152.53.52.53" - "152.70.200.104" - "154.16.206.***" - "154.194.8.*** crawler" - "154.30.107.182" - "154.92.124.*** crawler" - "156.224.168.2" - "157.97.126.228" - "159.138.100.151" - "159.138.102.121" - "159.138.103.201" - "159.138.104.242" - "159.138.105.125" - "159.138.105.133" - "159.138.106.205" - "159.138.110.12" - "159.138.111.209" - "159.138.111.76" - "159.138.121.61" - "159.138.29.236" - "159.138.34.28" - "159.138.85.78" - "159.138.86.203" - "159.138.98.152" - "159.138.98.91" - "159.138.99.105" - "159.203.160.220" - "165.154.254.20" - "165.154.255.62" - "165.22.247.105" - "165.232.173.160" - "166.108.192.240" - "166.108.194.166" - "166.108.194.199" - "166.108.196.109" - "166.108.200.133" - "166.108.200.170" - "166.108.200.231" - "166.108.203.17" - "166.108.203.187" - "166.108.225.152" - "166.108.226.24" - "166.108.228.4" - "166.108.229.204" - "166.108.229.39" - "166.108.230.220" - "166.108.230.240" - "166.108.232.117" - "166.108.233.127" - "166.108.233.135" - "166.108.233.28" - "166.108.234.88" - "166.108.239.131" - "166.108.239.43" - "17.241.219.36" - "17.241.227.191" - "17.241.75.166" - "17.241.75.252" - "170.253.53.115" - "172.104.166.140" - "172.172.104.187" - "172.56.66.154" - "173.252.70.114" - "173.252.83.***" - "173.252.87.***" - "174.254.48.23" - "176.125.229.***" - "176.125.230.***" - "178.128.93.5" - "178.175.128.40 Link checker" - "178.175.129.***" - "178.215.224.157" - "178.215.224.228" - "18.117.165.66" - "18.117.186.92" - "18.117.81.240" - "18.118.120.204" - "18.118.145.114" - "18.118.184.237" - "18.118.200.86" - "18.119.111.9" - "18.119.125.7" - "18.119.131.178" - "18.119.139.50" - "18.119.213.235" - "18.188.61.223" - "18.189.170.17" - "18.189.178.37" - "18.189.180.244" - "18.190.156.212" - "18.206.12.31" - "18.216.239.46" - "18.216.32.116" - "18.217.144.32" - "18.218.127.141" - "18.219.22.169" - "18.219.224.103" - "18.221.146.223" - "18.221.174.248" - "18.221.187.121" - "18.223.0.53" - "18.224.32.86" - "18.227.190.93" - "18.227.228.95" - "18.232.185.167" - "18.234.55.154" - "181.214.107.93" - "181.41.206.***" - "182.160.12.209" - "183.134.163.43" - "183.134.59.130" - "183.241.134.115" - "185.102.48.*** crawler" - "185.117.225.***" - "185.180.141.***" - "185.182.235.44" - "185.202.108.*** crawler" - "185.214.198.243" - "185.226.197.58" - "185.226.197.70" - "185.226.197.9" - "185.241.208.116" - "185.254.97.109" - "185.54.229.56" - "185.61.217.*** crawler" - "185.61.223.*** crawler" - "185.65.134.132" - "185.65.134.156" - "185.88.100.*** crawler" - "186.75.152.12" - "188.165.251.162" - "188.208.222.253" - "188.68.34.94" - "188.68.53.142" - "188.68.58.161" - "189.1.223.251" - "190.141.34.139" - "190.92.198.201" - "190.92.199.19" - "190.92.199.75" - "190.92.200.71" - "190.92.200.8" - "190.92.201.109" - "190.92.201.147" - "190.92.203.***" - "190.92.204.35" - "190.92.205.141" - "190.92.205.8" - "190.92.206.113" - "190.92.206.161" - "190.92.208.129" - "190.92.209.***" - "190.92.210.141" - "190.92.211.230" - "190.92.212.198" - "190.92.212.251" - "190.92.213.192" - "190.92.214.107" - "190.92.215.18" - "190.92.215.37" - "190.92.216.***" - "190.92.218.12" - "190.92.221.146" - "190.92.221.210" - "190.92.240.91" - "191.96.106.***" - "191.96.37.***" - "191.96.67.***" - "193.118.52.22" - "193.118.52.30" - "193.118.52.34" - "193.118.53.109" - "193.118.53.91" - "193.118.55.180" - "193.124.188.83" - "193.124.190.18" - "193.31.126.*** crawler" - "193.37.32.213" - "193.37.32.228" - "193.37.32.81" - "193.42.225.*** crawler" - "194.146.15.80" - "194.31.162.*** crawler" - "194.38.22.71" - "194.38.23.16" - "195.133.20.8" - "195.82.146.250" - "196.196.53.***" - "196.251.112.209" - "196.251.118.195" - "196.251.69.153" - "196.251.69.33" - "196.251.71.229" - "196.251.71.46" - "196.251.71.7" - "196.251.72.46" - "196.251.73.83" - "196.251.80.216" - "196.251.80.22" - "196.251.80.228" - "196.251.86.***" - "198.145.54.189" - "198.15.119.93" - "198.235.24.***" - "199.101.196.102" - "199.101.196.77" - "2.57.169.***" - "20.125.101.243" - "20.14.139.144" - "20.163.52.87" - "20.172.46.206" - "20.191.202.4" - "20.223.212.89" - "20.232.164.19" - "20.232.164.35" - "20.232.165.102" - "20.25.162.233" - "20.41.116.71" - "20.5.104.63" - "200.119.177.77" - "204.10.193.85" - "205.210.31.*** crawler" - "207.148.68.97" - "207.32.217.219" - "207.46.13.107" - "209.240.99.195" - "212.102.60.162" - "212.109.132.196" - "213.111.144.131" - "216.73.161.***" - "216.73.216.***" - "217.76.56.193" - "220.158.233.***" - "221.194.149.*** crawler" - "221.194.179.*** crawler" - "23.20.220.59" - "24.144.124.27" - "3.12.71.237" - "3.126.116.154" - "3.128.199.162" - "3.128.94.171" - "3.133.79.70" - "3.136.154.103" - "3.137.161.222" - "3.137.180.32" - "3.137.192.3" - "3.137.218.230" - "3.138.101.95" - "3.138.141.202" - "3.139.97.157" - "3.14.254.103" - "3.140.185.147" - "3.140.185.170" - "3.141.100.120" - "3.141.193.158" - "3.142.135.86" - "3.142.144.40" - "3.142.53.68" - "3.143.0.157" - "3.143.168.172" - "3.143.4.181" - "3.143.9.115" - "3.144.17.45" - "3.144.172.115" - "3.144.189.177" - "3.144.202.167" - "3.144.26.83" - "3.145.119.199" - "3.145.131.28" - "3.145.152.98" - "3.145.173.112" - "3.145.47.253" - "3.145.52.86" - "3.145.94.130" - "3.146.65.212" - "3.149.252.37" - "3.15.143.181" - "3.15.149.45" - "3.15.218.254" - "3.15.219.217" - "3.15.225.173" - "3.16.15.149" - "3.16.66.206" - "3.16.70.101" - "3.16.81.94" - "3.17.162.247" - "3.17.183.24" - "3.17.5.68" - "3.17.6.75" - "3.18.109.77" - "3.19.54.41" - "3.209.56.116" - "3.21.104.109" - "3.21.248.47" - "3.21.76.0" - "3.22.181.209" - "3.22.249.158" - "3.23.101.60" - "3.230.128.106" - "3.230.147.225" - "3.235.199.19" - "3.235.243.45" - "3.236.111.234" - "3.237.31.131" - "3.237.51.235" - "3.238.195.81" - "3.239.206.191" - "3.252.83.45" - "3.253.142.93" - "3.67.80.25" - "3.70.183.103" - "3.71.95.121" - "3.72.246.125" - "3.76.105.213" - "3.76.248.47" - "3.79.103.34" - "3.80.155.163" - "3.81.79.135" - "3.88.16.192" - "3.90.35.86" - "3.91.19.28" - "3.93.173.205" - "31.129.110.237" - "34.207.208.186" - "34.220.250.119" - "34.226.66.3" - "34.228.240.6" - "34.230.84.106" - "34.237.245.80" - "34.70.1.206" - "35.159.23.188" - "35.159.84.66" - "35.168.113.41" - "35.171.22.220" - "35.172.194.25" - "35.173.178.60" - "35.173.233.176" - "35.82.31.2" - "35.85.34.241" - "35.86.139.13" - "35.88.212.83" - "35.88.52.201" - "35.89.149.192" - "35.91.120.68" - "35.94.157.182" - "35.94.85.211" - "35.95.1.216" - "35.95.46.17" - "36.150.60.24" - "36.27.95.48" - "360Spider search engine" - "37.120.184.163" - "37.120.213.216" - "37.140.223.137" - "37.140.223.9" - "37.167.91.236" - "37.19.221.***" - "37.46.113.*** Link checker" - "4.147.168.150" - "4.210.91.56" - "4.227.121.203" - "40.71.170.134" - "43.128.84.42" - "43.128.88.143" - "43.128.89.170" - "43.130.12.103" - "43.130.33.54" - "43.130.42.164" - "43.130.62.137" - "43.133.38.100" - "43.133.43.121" - "43.133.43.154" - "43.133.43.227" - "43.133.56.146" - "43.133.57.8" - "43.133.59.248" - "43.133.60.115" - "43.133.60.97" - "43.133.62.111" - "43.133.62.221" - "43.134.0.62" - "43.134.107.106" - "43.134.109.11" - "43.134.112.111" - "43.134.118.145" - "43.134.119.86" - "43.134.12.237" - "43.134.121.104" - "43.134.15.134" - "43.134.16.138" - "43.134.165.87" - "43.134.167.226" - "43.134.176.114" - "43.134.184.91" - "43.134.229.118" - "43.134.231.229" - "43.134.236.64" - "43.134.26.191" - "43.134.26.28" - "43.134.41.2" - "43.134.46.116" - "43.134.48.88" - "43.134.56.250" - "43.134.57.196" - "43.134.61.126" - "43.134.61.238" - "43.134.63.65" - "43.134.64.76" - "43.134.69.123" - "43.134.69.90" - "43.134.72.28" - "43.134.73.181" - "43.134.75.217" - "43.134.77.29" - "43.134.91.203" - "43.134.91.49" - "43.134.99.61" - "43.143.200.85" - "43.143.7.112" - "43.153.112.164" - "43.153.192.129" - "43.153.193.211" - "43.153.35.128" - "43.153.62.242" - "43.156.107.145" - "43.156.12.8" - "43.156.2.243" - "43.156.29.120" - "43.156.29.145" - "43.156.3.195" - "43.156.5.207" - "43.156.6.103" - "43.159.130.175" - "43.159.32.86" - "43.159.37.213" - "43.159.41.139" - "43.159.41.195" - "43.163.0.23" - "43.163.8.72" - "44.197.113.64" - "44.200.101.170" - "44.200.196.114" - "44.202.90.91" - "44.204.164.147" - "44.214.187.82" - "44.218.102.77" - "44.220.41.140" - "44.222.122.246" - "44.222.125.114" - "44.222.149.13" - "44.234.8.38" - "45.10.155.226" - "45.10.155.233" - "45.138.16.202" - "45.141.215.116" - "45.156.128.***" - "45.156.129.***" - "45.156.130.21" - "45.156.130.32" - "45.156.131.14" - "45.156.131.23" - "45.156.131.25" - "45.192.141.*** crawler" - "45.192.145.*** crawler" - "45.192.146.*** crawler" - "45.3.37.226" - "45.3.38.77" - "45.33.51.228" - "45.41.130.192" - "45.76.163.17" - "45.80.158.218" - "45.83.31.80" - "45.84.224.217" - "45.88.13.*** crawler" - "45.92.229.***" - "46.19.138.210" - "46.250.250.119" - "47.238.13.*** crawler" - "47.242.148.*** crawler" - "47.242.149.*** crawler" - "47.242.167.*** crawler" - "47.242.209.*** crawler" - "47.243.228.*** crawler" - "47.243.62.*** crawler" - "47.243.78.*** crawler" - "47.76.222.*** crawler" - "47.79.0.***" - "47.79.1.***" - "47.79.116.***" - "47.79.117.***" - "47.79.118.***" - "47.79.119.***" - "47.79.120.***" - "47.79.121.***" - "47.79.122.***" - "47.79.123.***" - "47.79.2.***" - "47.79.3.***" - "47.79.4.***" - "47.79.5.***" - "47.79.7.***" - "47.79.98.***" - "47.82.0.*** crawler" - "47.82.10.*** crawler" - "47.82.11.*** crawler" - "47.82.12.***" - "47.82.13.***" - "47.82.14.***" - "47.82.15.***" - "47.82.16.***" - "47.82.17.***" - "47.82.18.***" - "47.82.20.***" - "47.82.23.***" - "47.82.24.***" - "47.82.25.***" - "47.82.26.***" - "47.82.28.***" - "47.82.9.*** crawler" - "49.0.203.114" - "49.0.203.70" - "49.0.204.134" - "49.0.205.36" - "49.0.205.66" - "49.0.205.79" - "49.0.206.145" - "49.0.206.244" - "49.0.206.82" - "49.0.207.105" - "49.0.207.20" - "5.183.255.129" - "51.222.158.108" - "51.75.206.225" - "51.75.206.94" - "52.164.231.122" - "52.178.204.143" - "52.224.217.143" - "52.226.135.84" - "52.38.92.236" - "52.90.181.205" - "54.144.81.21" - "54.159.186.146" - "54.160.243.44" - "54.166.234.171" - "54.172.169.199" - "54.185.168.115" - "54.213.253.217" - "54.221.69.42" - "54.224.124.217" - "54.225.1.66" - "54.234.136.147" - "54.234.83.134" - "54.242.75.224" - "54.39.190.168" - "54.69.185.226" - "54.85.255.74" - "57.141.0.***" - "57.141.5.***" - "57.141.7.***" - "58.144.138.11" - "62.72.43.36" - "62.72.45.43" - "63.177.67.193" - "63.177.87.18" - "64.137.121.*** crawler" - "64.23.233.179" - "64.71.179.*** crawler" - "65.109.116.*** crawler" - "66.220.149.***" - "69.171.230.28" - "69.171.230.31" - "69.171.230.4" - "69.171.249.***" - "74.208.88.236" - "74.235.223.64" - "77.74.177.114" - "77.74.177.118" - "77.74.177.119" - "78.141.211.98" - "79.142.79.47 Link checker" - "79.142.79.55 Link checker" - "8.210.154.*** crawler" - "8.210.187.*** crawler" - "8.210.190.*** crawler" - "8.218.91.*** crawler" - "80.76.51.217" - "80.85.139.166" - "80.85.142.44" - "81.21.233.*** crawler" - "82.67.157.9" - "83.99.151.64" - "83.99.151.67" - "83.99.151.71" - "84.17.35.72" - "84.247.60.*** crawler" - "85.208.96.194" - "85.208.96.205" - "85.239.37.*** crawler" - "86.72.52.243" - "87.106.214.89" - "87.120.114.171" - "87.120.126.100" - "87.121.86.54" - "89.58.12.49" - "91.222.174.116" - "91.240.118.252" - "91.246.195.*** crawler" - "91.90.44.22" - "91.92.243.138" - "91.92.243.241" - "91.92.245.181" - "91.92.246.225" - "91.92.246.247" - "91.92.247.168" - "91.92.250.96" - "91.92.253.183" - "91.92.253.80" - "91.92.254.123" - "91.92.254.43" - "91.92.255.159" - "93.157.29.195" - "93.22.38.74" - "94.103.125.240" - "94.156.67.39" - "94.228.145.229" - "94.74.122.138" - "94.74.80.132" - "94.74.80.161" - "94.74.80.165" - "94.74.81.63" - "94.74.82.42" - "94.74.83.173" - "94.74.84.120" - "94.74.84.72" - "94.74.85.217" - "94.74.85.240" - "94.74.85.248" - "94.74.86.81" - "94.74.87.185" - "94.74.88.***" - "94.74.89.31" - "94.74.90.22" - "94.74.92.***" - "94.74.94.113" - "94.74.94.172" - "95.142.121.46" - "98.159.234.***" - "98.96.193.***" - "Gaisbot/3.0 search engine" - , "www.dir.com"

You can see their last visits or find their identity (1021 User Agent strings) or download a list.

Some robots regularly request robots.txt but link checkers (inbound links from other sites or search engines), validation tools and log spamming do not read robots.txt.

Recently for this site:

• Robots visits
• Origin of the robots
• IP adresses and origin of the robots spamming the site
• Origin the robots looking for vulnerabilities

Older visits:

Detecting a robot

The undesirables

First of all, it is necessary to ensure that the requested page actually exists.
For this site, Apache returns /index.php in $_SERVER['SCRIPT_NAME'] if the page does not exist. If $_SERVER['REQUEST_URI'] without the query string is not the homepage, then we are dealing with a robot looking for a CMS known vulnerability.

It is then necessary to ensure that the query string is not different from what is used by the site operation or tested for undesirable elements using $_SERVER['QUERY_STRING']. The regular expressions #(\b|%..)(AND|UNION|SELECT|CASE|ORDER( |%20)BY)(\b|%..)|(CHA?R|SLEEP|ELT|CONCAT|SCRIPT)\(\d|\d{1,4}=\d{1,4}#i and #(author|cmd|mode|option|task|view)=|admin|user|w(ord)?p(ress)?|\.env|%#i identify the most common ones.
The user agent may also contain these same elements.

We can also verify that the contact page is not the only one requested and using a POST method.

Checking the "referer" ($_SERVER['HTTP_REFERER']) helps to identify spammers. The method used is often HEAD, the second-level domain of the URL, the directory or the main domain name can be significant. For this site, the second-level domain is most often ru, ua, kz, su, by or club, finance, fun... The directory and the domain name also use words related to marketing, gambling, finance...
The log files allow to make a list fairly quickly.

Many robots regularly return without identifying themselves and request, always in the same order, often with the site URL as "referer", the same pages, sometimes one or two pages, sometimes a large part of the site.
They are more difficult to identify because they change their IP address, their user agent...

This identification of undesirable robots is not bullet-proof because the robots of the best search engines suffer from bugs.

Using its User Agent

Here is a PHP script (which is used by the site stats) allowing you to know if a robot or a search engine is requesting a page:

$UA=$_SERVER["HTTP_USER_AGENT"]; $no_ip=$_SERVER["REMOTE_ADDR"]; $brow = strtolower($UA);//to lowercase as there are many versions $bots = array ("googlebot","webcrawler","grub.org","slurp","openfind","antibot", "netresearchserver","nutch","ia_archiver","scooter","fluffy"); //Here you can add the other robots names in lower case $n=0; do { $robot = strchr($brow,$bots[$n]); $n=$n+1; }while ((!$robot) && ($n<count($bots))); //if $robot is a nul string, you can do the same to spot spiders, "Spamrats", //or other visitors who identify themselves and then take a decision. //if $robot contains the name of the robot you want to stop, // redirect before any output towards the browser by using header("Location:anypage.html"); //this page will have no links, so the spider will stop searching. //In the site the page http://danzcontrib2.free.fr/no_msie6.php redirects any MSIE 6 compatible to the home page //and displays the code to other browsers.

A script using the User Agent is now online here

It is more difficult to spot robots that do not identify:

• give a MSIE 6 (as UUNET or Websense) or Mozilla 5 (net-sweeper) or Konqueror (twtc / Websense - RegExp: 3\.[0-1](-rc[1-6])?; i686 Linux; 2002[0-9]{4}-, exabot - Exalead User Preview) or Mozilla 4.01 (NOOS) identification,
• change IP address each time they load a page (a WHOIS search on Ripe or Whois Source or Openrbl may give you a clue),
• combine all the methods (qwest.net, .ev1servers.net as well).

Using its host

A good example seems to be the www.dir.com (search engine) robot which uses many IP addresses (from 212.27.33.164 to 212.27.33.173 in May 2003, 212.27.41.18 in November 2003). Its activity could be seen on the page logging servers, but is filtered now by the following PHP routine.

if (!$robot)
{
$robot=strchr(gethostbyaddr($no_ip),".dir.com");
}
//if it's the www.dir.com robot then $robot is set as .dir.com

The regular expression /semrush|^crawl|\.(google|msn|spider|apple|sketchengine)\./ helps to avoid identification errors related to a bug or an unusual user agent.

Using its IP address

A robot requesting pages from a few IP addresses can be spotted likewise:

if (!$robot)
{
$robot=strchr($no_ip,"208.53.138.");
}
/*
if the IP address is between 208.53.138. and 208.53.138.
$robot is set as 208.53.138.
*/

In any case, maintaining a list of User Agents, hosts and IP addresses noticed as having a strange behaviour will be necessary.

Using the request method

It seems that, at the present time (June 2005), only robots and download utilities use a HEAD request (then a GET if the page exists or has been modified). Thus $_SERVER["REQUEST_METHOD"] can allow the identification of a robot using a browser User Agent. (Read RSS feed for tests in progress).

/*this method must come first*/
if ($_SERVER["REQUEST_METHOD"]=="HEAD") {$robot="robot";};
/*if head is used, $robot will not be empty*/

All these methods seem to be rather accurate.

Blocking a robot with PHP

When some Apache modules are not available for use and having access to .htaccess files is restricted (my case) or if we want to cut down the size of the file .htaccess and let the server do what's useful, PHP allows us to redirect or block a robot.

If we want to stop a robot (here Fetch API Request) , we just have begin all our pages (before any output to the browser) with the following script so that the webbot is redirected toward the page bye.html, any other page or send a 403 Access Denied status message.

<?php
$UA=getenv("HTTP_USER_AGENT");
if (stristr($UA,"Fetch API Request")!="")
{
header("Location:http://mydomain/bye.html");
die(); /*this line can be replaced by the HTML redirection*/
}
?>

This page not being linked, the spidering will immediately stop.
The same can be done with an IP address by using getenv("REMOTE_ADDR");.
More sophisticated techniques are listed above.

About two thirds of the robots will follow the redirection if the domain name does not change, almost none if it changes.
A redirection in HTML will be necessary if we want to redirect all of them or let them know where the new page is:

<?php
echo"<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
 "http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<title>Redirection</title>
<meta http-equiv="Refresh" content="0;URL=http://mydomain/bye.html">
</head>
<body>
<p>
Redirection: <a href="http://mydomain/bye.html">http://mydomain/bye.html</a>
</p>
</body>
</html>";
die();
?>

Allowing some robots and blocking others

A function to include and call at the begining of each page can allow us to manage robots.

/*start*/
function redirect_robots()
{
$requested_page=$_SERVER["REQUEST_URI"];
if (preg_match("/([enptux\d]|\b)(ftp|https?|php)(:\/\/|%3A%2F%2F)/i",$requested_page))
   {die();} /*blocks the majority of zombies*/

When we are unlucky and visited by zombies, or when we are using a CMS, the best is to block all these requests.

if ($_SERVER["REQUEST_METHOD"]=="HEAD") return;

Why should we block this type of request? The "harm's done", link checkers toward our site (Xenu, Powermarks, Link Commander, HTTrack, IRLbot...) and search engines (Speedy Spider, sygol...) will have a positive answer and their case, if they come back with a GET or POST request, will be considered later.
There, we can store the IP address in a MySQL table to block any comeback of the utility or webbot.

$UA=getenv("HTTP_USER_AGENT");
if (preg_match("/Googlebot|Yahoo|VoilaBot|Ask Jeeves|SpeedySpider/i",$UA)) return;

No problem for the robots we accept: those who identify themselves and are named in the regular expression above. The host can be checked to see if it matches the User Agent.

/*
Including bot in the expression will block aipbot, antibot, boitho, OmniExplorer...
As for this site, up to 408 robots!
*/
if (preg_match("/[^e]crawler|spider|bot|custo |web(cow|moni|capture)|wysigot|httrack|wget|xenu/i",$UA))
{
header("Location:http://mydomain/bye.html");die();
/*another option is to send a 403 Access Denied status message
handled by Apache .htaccess
header("Status: 403 Forbidden");die();*/
}

Even if I am not convinced by the necessity to block the ones that do not exaggerate, all those in the regular expression will be redirected.
Many utilities like Wysigot leave their name in the User Agent even when they are not active.

$no_ip=getenv("REMOTE_ADDR");
$host=gethostbyaddr($no_ip);
if (preg_match("/(becquerel|66-132|64-225)\.noos\.(net|fr)/i",$host) && (strchr($UA,"MSIE 4.01"))
 {
 header("Location:http://mydomain/bye.html");die();
 }
if (preg_match("/exabot|lehigh/i",$host))
 {
 header("Location:http://mydomain/bye.html");die();
 }

We can test the host and ban a few badly-behaved robots or the reading by a request from a search engine. Is it really useful?

//$no_ip=getenv("REMOTE_ADDR");
if (preg_match("/63\.247\.72\.42|208\.53\.138\.1/",$no_ip))die();

We can ban an IP address or a group of IP addresses, get from a MySQL database the IP address to ban...

return;
}
/*end*/

Now, those who are still here can browse.
We can optimize the code, add a few rules for the referrer, the number of pages requested (stored with MySQL)... It will be easy to update or modify the code, but how many errors?

A few ideas...

As indexing activity shouldn't be blocked (even if no one can stop a web spider user to declare a robot identifier), knowing whether a human being is viewing a page is done in the site with two bot traps in the French home page (and only one robot trap in the English home page):
They consist in links without text so that no one can see them.
- The first is in an allowed folder. Any access to the file allows me to update the list above.
- The second is in a folder marked as prohibited to robots in the file robots.txt ( Disallow: /interdit/). Even if all indexing robots do not always respect the rules, if the page is hit it must be a web copier.
As the site is rarely copied and even if few users follow robots.txt rules, these two traps do not initiate an action.
If some people find the site interesting enough to be mirrored, they can archive it but I could stop them with a script from the sites mentioned above, the methods following the detection script, an anti-mirroring PHP script, I could limit the number of pages per session or per IP address (robots usually follow the same route), or slow them by counting the number of pages visitors or robots are trying to get by second and allow less than a page per second which will be a problem for web spiders and people who do not read.
Using the IP address to do so works if the visitor's provider gives a unique IP address. This is not the case with AOL and many big companies.
Changing provider is one option: some filter web spiders (just as www.free.fr sometimes does!!!).

Therefore preventing or stopping website mirroring is difficult or risky.

If you prefer offline browsing, you can download the static part of the site (extension of compressed files : exe~597k or bz2~631k - December 2005 / use the site map).