从一分钟的蜘蛛抓取看搜索引擎

2shoushichang · 发表于 2007-3-6

2007-03-05 23:59:59 218.104.*.* GET /type/l-haikou-98.html - 80 - 202.160.179.90 Mozilla/5.0+(compatible;+Yahoo!+Slurp+China;+<a href="http://misc.yahoo.com.cn/help.html)" target="_blank">http://misc.yahoo.com.cn/help.html)</a> 200 0 0 2007-03-06 00:00:06 218.104.*.* GET /lend/721.html - 80 - 202.160.178.131 Mozilla/5.0+(compatible;+Yahoo!+Slurp+China;+<a href="http://misc.yahoo.com.cn/help.html)" target="_blank">http://misc.yahoo.com.cn/help.html)</a> 200 0 0 2007-03-06 00:00:09 218.104.*.* GET /lend/10172.html - 80 - 60.12.227.58 Mozilla/5.0+(compatible;+YodaoBot/1.0;+<a href="http://www.yodao.com/help/webmaster/spider/;+)" target="_blank">http://www.yodao.com/help/webmaster/spider/;+)</a> 200 0 0 2007-03-06 00:00:16 218.104.*.* GET /lend/6921.html - 80 - 60.12.227.58 Mozilla/5.0+(compatible;+YodaoBot/1.0;+<a href="http://www.yodao.com/help/webmaster/spider/;+)" target="_blank">http://www.yodao.com/help/webmaster/spider/;+)</a> 200 0 0 2007-03-06 00:00:17 218.104.*.* HEAD /lend/27870.html - 80 - 202.108.22.142 Baiduspider+(+<a href="http://www.baidu.com/search/spider.htm)" target="_blank">http://www.baidu.com/search/spider.htm)</a> 200 0 0 2007-03-06 00:00:17 218.104.*.* GET /smalltype/s-xiamen-4-22.html - 80 - 202.160.178.190 Mozilla/5.0+(compatible;+Yahoo!+Slurp+China;+<a href="http://misc.yahoo.com.cn/help.html)" target="_blank">http://misc.yahoo.com.cn/help.html)</a> 200 0 0 2007-03-06 00:00:19 218.104.*.* GET /lend/9956.html - 80 - 202.160.178.92 Mozilla/5.0+(compatible;+Yahoo!+Slurp+China;+<a href="http://misc.yahoo.com.cn/help.html)" target="_blank">http://misc.yahoo.com.cn/help.html)</a> 200 0 0 2007-03-06 00:00:20 218.104.*.* HEAD /type/s-nanjing-146.html - 80 - 202.108.22.142 Baiduspider+(+<a href="http://www.baidu.com/search/spider.htm)" target="_blank">http://www.baidu.com/search/spider.htm)</a> 200 0 0 2007-03-06 00:00:25 218.104.*.* GET /lend/30849.html - 80 - 66.249.72.101 Mozilla/5.0+(compatible;+Googlebot/2.1;++<a href="http://www.google.com/bot.html)" target="_blank">http://www.google.com/bot.html)</a> 200 0 0 2007-03-06 00:00:26 218.104.*.* GET /lend/9794.html - 80 - 202.160.179.132 Mozilla/5.0+(compatible;+Yahoo!+Slurp+China;+<a href="http://misc.yahoo.com.cn/help.html)" target="_blank">http://misc.yahoo.com.cn/help.html)</a> 200 0 0 2007-03-06 00:00:30 218.104.*.* GET /lend/9184.html - 80 - 202.160.178.91 Mozilla/5.0+(compatible;+Yahoo!+Slurp+China;+<a href="http://misc.yahoo.com.cn/help.html)" target="_blank">http://misc.yahoo.com.cn/help.html)</a> 200 0 0 2007-03-06 00:00:31 218.104.*.* GET /sale/28890.html - 80 - 60.12.227.58 Mozilla/5.0+(compatible;+YodaoBot/1.0;+<a href="http://www.yodao.com/help/webmaster/spider/;+)" target="_blank">http://www.yodao.com/help/webmaster/spider/;+)</a> 200 0 0 2007-03-06 00:00:32 218.104.*.* HEAD /sale/6169.html - 80 - 202.108.22.142 Baiduspider+(+<a href="http://www.baidu.com/search/spider.htm)" target="_blank">http://www.baidu.com/search/spider.htm)</a> 200 0 0 2007-03-06 00:00:34 218.104.*.* GET /smalltype/l-nanjing-1-27.html - 80 - 202.160.179.121 Mozilla/5.0+(compatible;+Yahoo!+Slurp+China;+<a href="http://misc.yahoo.com.cn/help.html)" target="_blank">http://misc.yahoo.com.cn/help.html)</a> 200 0 0 2007-03-06 00:00:34 218.104.*.* HEAD /sale/4886.html - 80 - 202.108.22.142 Baiduspider+(+<a href="http://www.baidu.com/search/spider.htm)" target="_blank">http://www.baidu.com/search/spider.htm)</a> 200 0 0 2007-03-06 00:00:35 218.104.*.* GET /lend/9986.html - 80 - 202.160.178.230 Mozilla/5.0+(compatible;+Yahoo!+Slurp+China;+<a href="http://misc.yahoo.com.cn/help.html)" target="_blank">http://misc.yahoo.com.cn/help.html)</a> 200 0 0 2007-03-06 00:00:37 218.104.*.* HEAD /lend/1418.html - 80 - 202.108.22.142 Baiduspider+(+<a href="http://www.baidu.com/search/spider.htm)" target="_blank">http://www.baidu.com/search/spider.htm)</a> 200 0 0 2007-03-06 00:00:40 218.104.*.* HEAD /sale/1331.html - 80 - 202.108.22.142 Baiduspider+(+<a href="http://www.baidu.com/search/spider.htm)" target="_blank">http://www.baidu.com/search/spider.htm)</a> 200 0 0 2007-03-06 00:00:43 218.104.*.* HEAD /lend/347.html - 80 - 202.108.22.142 Baiduspider+(+<a href="http://www.baidu.com/search/spider.htm)" target="_blank">http://www.baidu.com/search/spider.htm)</a> 200 0 0 2007-03-06 00:00:43 218.104.*.* GET /smalltype/l-guangzhou-2-9.html - 80 - 202.160.178.150 Mozilla/5.0+(compatible;+Yahoo!+Slurp+China;+<a href="http://misc.yahoo.com.cn/help.html)" target="_blank">http://misc.yahoo.com.cn/help.html)</a> 200 0 0 2007-03-06 00:00:44 218.104.*.* GET /lend/2076.html - 80 - 202.160.180.65 Mozilla/5.0+(compatible;+Yahoo!+Slurp+China;+<a href="http://misc.yahoo.com.cn/help.html)" target="_blank">http://misc.yahoo.com.cn/help.html)</a> 200 0 0 2007-03-06 00:00:45 218.104.*.* HEAD /lend/516.html - 80 - 202.108.22.142 Baiduspider+(+<a href="http://www.baidu.com/search/spider.htm)" target="_blank">http://www.baidu.com/search/spider.htm)</a> 200 0 0 2007-03-06 00:00:47 218.104.*.* HEAD /lend/8841.html - 80 - 202.108.22.142 Baiduspider+(+<a href="http://www.baidu.com/search/spider.htm)" target="_blank">http://www.baidu.com/search/spider.htm)</a> 200 0 0 2007-03-06 00:00:49 218.104.*.* HEAD /lend/12985.html - 80 - 202.108.22.142 Baiduspider+(+<a href="http://www.baidu.com/search/spider.htm)" target="_blank">http://www.baidu.com/search/spider.htm)</a> 200 0 0 2007-03-06 00:00:52 218.104.*.* GET /sale/12149.html - 80 - 60.12.227.58 Mozilla/5.0+(compatible;+YodaoBot/1.0;+<a href="http://www.yodao.com/help/webmaster/spider/;+)" target="_blank">http://www.yodao.com/help/webmaster/spider/;+)</a> 200 0 0 2007-03-06 00:00:53 218.104.*.* HEAD /sale/31397.html - 80 - 202.108.22.142 Baiduspider+(+<a href="http://www.baidu.com/search/spider.htm)" target="_blank">http://www.baidu.com/search/spider.htm)</a> 200 0 0 2007-03-06 00:00:55 218.104.*.* HEAD /lend/32143.html - 80 - 202.108.22.142 Baiduspider+(+<a href="http://www.baidu.com/search/spider.htm)" target="_blank">http://www.baidu.com/search/spider.htm)</a> 200 0 0 这是早上零点1分钟内的蜘蛛来访记录,从里面我们可以看出一下几个问题: 1.百度仍是国内最大的蜘蛛来源,也是国内抓取页面最多的搜索引擎.(12次) 2.google抓取页面的频率相对比较低(1次,在零点这一小时内最多的一分钟(0点9分)抓了6次) 3.yahoo抓取页面也很多,基本和百度持平(10次) 4.出现了一个相对较新的蜘蛛YodaoBot(以前把它当作yahoo了),抓取频率也较高 5.没有出现iask和TencentTraveler,他们抓取频率相对较低(在零点开始的前2个小时,iask只抓取了十几次,TencentTraveler抓取了一百多次),这也和他们的市场份额相符 6.所有蜘蛛都抓取js页面,不过yahoo好像喜欢js文件,基本见一个抓一个,百度相对抓的少一些. 7.百度和yahoo对参数类型的页面也是照抓不误,google好像不是很感冒. (数据按照早上零点到5点35分1万零18次的抓取数据分析)

xiaoyeah · 发表于 2007-3-6

如何抓到这些记录的？基本都在同个时间抓你站？

2shoushichang · 发表于 2007-3-6

如果你自己有独立服务器的话,可以从C:\WINDOWS\system32\LogFiles\W3SVC908923928找到这些记录. 蜘蛛是一直在抓站的,我的站每天大概被抓取4~5万次,其中google大概1万次(虚拟主机可以通过查看google管理员工具得到),其余的是百度和yahoo,别的几个较少

2shoushichang · 发表于 2007-3-6

当然,从这里我们也可以看出来,如果服务器性能不好的话,是得不到好的收录的.

impop · 发表于 2007-3-6

白愁飞 · 发表于 2007-3-6

服务器性能不好的话,是得不到好的收录的. ——应该是这样的

xuexiaowei · 发表于 2007-3-9

好的,使劲顶,加油了各位

		自动登录	找回密码
密码			注册