Webbot activity on Jun 26, 2017

name		activity
IP 46.29.251.11	log spam	1 page
IP 198.27.91.223	log spam	1 page
Googlebot	crawler	1 page
DotBot	crawler	1 page
Googlebot	crawler	1 page
bingbot	crawler	1 page
BUbiNG	crawler	10 pages in  1' 13''
DotBot	crawler	1 page
Yahoo! Slurp	crawler	1 page
Yahoo! Slurp	crawler	1 page
IP 195.154.21.201	log spam	1 page
Baiduspider	search engine	1 page
DuckDuckGo	search engine	1 page
Yahoo! Slurp	crawler	1 page
IP 80.241.223.216	log spam	1 page
Yahoo! Slurp	crawler	1 page
Googlebot	crawler	1 page
Python-urllib		1 page
bingbot	crawler	1 page
CCBot	crawler	33 pages in  9' 19''
661,478 visits of identified bots about 107 a day in 2014 38 today at 14:08 (+55 visitors) zombies : 3 visits / 31 requests - spammers : 5 visits

What do these statistics mean?

They have been extracted from $_SERVER["HTTP_USER_AGENT"] -$HTTP_USER_AGENT with PHP 3- and gethostbyaddr().
As this website host (free.fr) sometimes filters access, they are biased.
The site statistics do not consider robots as visitors. The browser and country they show are ignored. Therefore it is quite easy to log the activity of those which are not banned.
Even if they read a few pages they are stored once except if they come back after more than 10 minutes (for Google Desktop after more than 30 minutes).
This list gives approximate information as it assumes a perfect connection to MySQL which is not the case for this site. But web hosting here is free, so...

Robot Detection

This routine is commented in the page about webbot traps.
Logging other visitors' user agent is necessary to update the lists of webbots visiting the site and of their User Agents.
$UA=$_SERVER["HTTP_USER_AGENT"]; $no_ip=$_SERVER["REMOTE_ADDR"]; $brow = strtolower($UA);//in lower case to spot any version $bots = array ("googlebot","webcrawler","grub.org","slurp","openfind","antibot", "netresearchserver","nutch","ia_archiver","scooter","fluffy"); /* The list of user agents allows me to add the most significant part of robots names in lower case in the array $bot As the connection to MySQL is not perfect 24 hours a day, I use an array so that robots always get a light page. */ $n=0; do { $robot = strchr($brow,$bots[$n]); $n=$n+1; }while ((!$robot) && ($n<count($bots))); //for robots who do not identify with $HTTP_USER_AGENT if (!$robot) { $robot=strchr(gethostbyaddr($no_ip),".dir.com"); //etc } //

The Data Table

Here is the structure of the table robots I use:

#
# Structure of the table `robots`
#
CREATE TABLE `robots` (
  `timeoflastaccess` int(10) unsigned NOT NULL default '0',
  `timeofarrival` int(10) unsigned NOT NULL default '0',
  `nameofbot` varchar(64) NOT NULL default '',
  `lastpage` varchar(30) NOT NULL default '',
  `numberofpages` mediumint(8) unsigned NOT NULL default '0',
  KEY `timeoflastaccess` (`timeoflastaccess`),
  KEY `timeofarrival` (`timeofarrival`),
  KEY `nameofbot` (`nameofbot`),
  KEY `numberofpages` (`numberofpages`)
) TYPE=MyISAM;

You can use double or datetime for times, double or int for numberofpages, if necessary, increase the number of caracters for lastpage.

Table Update

//connection to the server $serveur=mysql_connect("your host","user name","password"); $base=mysql_select_db("your base",$serveur); //we set time $timeoflastaccess=time(); $timeoflastaccesslessten=$timeoflastaccess-600; if ($robot!="") { $result=@mysql_query("SELECT COUNT(`timeoflastaccess`) FROM `robots`"); $test=@mysql_fetch_row($result); //Some cleaning from time to time if ($test[0]>100) { $result=@mysql_query("SELECT `timeoflastaccess` FROM `robots` ORDER BY `timeoflastaccess` LIMIT 80,1"); $test=@mysql_fetch_row($result); $result=@mysql_query("DELETE FROM `robots` WHERE `timeoflastaccess`<'$test[0]'"); $result=@mysql_query("OPTIMIZE TABLE `robots`"); } /* Many robots have a dynamically allocated IP, thus the name of the robot is a better field to store even if many versions arrive simultaneously! Did the robot load a page in the last 600 seconds? */ $result=@mysql_query("SELECT * FROM `robots` WHERE `nameofbot`='$robot' AND `timeoflastaccess`>$timeoflastaccesslessten"); $test=@mysql_fetch_row($result); $page_self=$_SERVER["REQUEST_URI"];//$PHP_SELF for PHP 3 if (empty($test[0])) { //no, it is the first page $result=@mysql_query("INSERT INTO `robots` (`timeoflastaccess`,`timeofarrival`,`nameofbot`,`lastpage`,`numberofpages`) VALUES ('$timeoflastaccess','$timeoflastaccess','$robot','$page_self','1');"); } else { //time of last access is updated $result=@mysql_query("UPDATE `robots` SET `timeoflastaccess`=$timeoflastaccess WHERE `nameofbot`='$robot' AND `timeoflastaccess`>$timeoflastaccesslessten"); //we get lastpage and numberofpages $result=@mysql_query("SELECT * FROM `robots` WHERE `nameofbot`='$robot' AND `timeoflastaccess`='$timeoflastaccess'"); $res=@mysql_fetch_row($result); if ($page_self!=$res[3]) { //This is not the same page //we update $result=@mysql_query("UPDATE `robots` SET `lastpage`='$page_self',`numberofpages`=numberofpages+1 WHERE `nameofbot`='$robot' AND `timeoflastaccess`='$timeoflastaccess'"); } //Et voilà ! You surely can do better. } } else { /* you can do the same for visitors with their IP or store the $HTTP_USER_AGENT to update the array $bots. */ }

Data Display

We have the name of the robot, the time of its arrival, the time of the last page it loaded and the total number of pages indexed.
If there is one page and different values for timeoflastaccess and timeofarrival, then the page was reloaded.
I chose to display the number of pages loaded and the length of reading time.
//connection to the server $server=mysql_connect("your host","user name","password"); $base=mysql_select_db("your base",$server); //selection of the last 20 insertions $result=@mysql_query("SELECT COUNT('timeoflastaccess') FROM `robots`"); $numberofrows=@mysql_fetch_row($result); $start=$numberofrows[0]-20;if ($start<0){$start=0;} $result=@mysql_query("SELECT * FROM `robots` ORDER BY `timeofarrival` LIMIT $start,20"); // while ($tableau=@mysql_fetch_row($result)) { //time spent $tpslect=$tableau[0]-$tableau[1]; if ($tpslect>60) { $tpslect=Floor($tpslect/60)."'".$tpslect % 60; } //display of name number and time spent echo "$tableau[2] $tableau[4] pages in $tpslect''<br>"; }

A similar script is now online here