|
这些是我监控到的数据,郁闷的是爬虫爬了几天了,不收录一个页面
17:54:43,309 INFO [STDOUT] Mediapartners-Google/2.1
17:54:43,309 INFO [STDOUT] 66.249.72.78
17:54:46,237 INFO [STDOUT] Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
17:54:46,237 INFO [STDOUT] 66.249.72.71
17:54:48,182 INFO [STDOUT] Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
17:54:48,182 INFO [STDOUT] 66.249.72.68
17:54:49,810 INFO [STDOUT] Mediapartners-Google/2.1
17:54:49,811 INFO [STDOUT] 66.249.72.79
17:54:51,437 INFO [STDOUT] Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
17:54:51,437 INFO [STDOUT] 66.249.72.84
..... |
|