Yuriko.Net 個別記事

2006-10-19
晴れ

Baiduspider を拒否

ゆりこ による 23:51:35 の投稿
カテゴリー: ネットワーク

9月下旬ごろからアクセスログが7倍ぐらいに増えてしまったのですが、どうやら中国の百度という検索エンジンのクローラー (Baiduspider) が原因のようでした。存在しないサブディレクトリーを執拗に検索するという、めちゃくちゃな動作をするので、 robots.txt で拒否することにしました。

User-agent: baiduspider
Disallow: /

とりあえず、こう書いておいてどうなるか見物です。もしアクセスに変化がなければ Apache の設定で拒否することを検討します。

百度は中国では Google を凌ぐ人気のようですが、確かに以下のような執拗な探索をしていれば、隠れた情報も見付けてしまいそうですね;-) Yuriko.Net の arc/ 以下は yyyy/mm/dd/ などの URI 構成なのに、それより深いディレクトリーを掘るなんてひどすぎ。深いディレクトリーにアクセスしても、それより上位ディレクトリーに記事が存在すればページを生成してしまう Yuriko.Net のシステムにも原因がありますが……。

60.28.17.33 - - [19/Oct/2006:23:48:37 +0900] "GET /arc/2005/08/22/06/10/02/ HTTP/1.1" 200 11964 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:48:42 +0900] "GET /arc/2005/08/26/30/23/14/26/ HTTP/1.1" 200 22038 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:48:43 +0900] "GET /arc/2005/08/09/07/27/03/14/ HTTP/1.1" 200 14834 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:48:44 +0900] "GET /arc/2005/02/04/26/2005/08/ HTTP/1.1" 200 15400 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:48:45 +0900] "GET /arc/2005/08/31/07/23/25/31/ HTTP/1.1" 200 10976 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:48:46 +0900] "GET /arc/2005/01/22/25/23/03/17/ HTTP/1.1" 200 11568 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:48:51 +0900] "GET /arc/2005/08/22/06/09/31/24/ HTTP/1.1" 200 12192 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:48:51 +0900] "GET /arc/2005/04/06/20/13/03/30/ HTTP/1.1" 200 16794 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:48:53 +0900] "GET /arc/2005/04/12/04/22/06/07/ HTTP/1.1" 200 11361 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:48:52 +0900] "GET /arc/2005/04/06/22/08/19/affiriate HTTP/1.1" 200 16554 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:48:55 +0900] "GET /arc/2005/08/09/06/25/13/14/10/ HTTP/1.1" 200 15068 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:49:00 +0900] "GET /arc/2004/09/14/20/14/ HTTP/1.1" 200 10996 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:49:02 +0900] "GET /arc/2005/01/22/23/03/12/30/ HTTP/1.1" 200 11568 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:49:04 +0900] "GET /arc/2005/04/06/02/01/08/affiriate HTTP/1.1" 200 16554 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:49:07 +0900] "GET /arc/2005/08/23/04/06/05/ HTTP/1.1" 200 11299 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:49:06 +0900] "GET /arc/2005/02/21/16/26/04/ HTTP/1.1" 200 13958 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:49:08 +0900] "GET /arc/2005/07/02/02/01/28/ HTTP/1.1" 200 11330 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:49:08 +0900] "GET /arc/2006/08/26/27/21/affiriate HTTP/1.1" 200 10895 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:49:10 +0900] "GET /arc/2004/09/18/01/11/30/ HTTP/1.1" 200 12214 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:49:15 +0900] "GET /arc/2005/03/01/20/13/13/ HTTP/1.1" 200 11701 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:49:15 +0900] "GET /arc/2005/08/16/23/10/17/ HTTP/1.1" 200 12648 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:49:17 +0900] "GET /arc/2005/08/09/05/04/06/ HTTP/1.1" 200 14600 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:49:19 +0900] "GET /arc/2005/08/28/02/27/22/23/ HTTP/1.1" 200 19878 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:49:17 +0900] "GET /arc/2006/03/05/19/30/ HTTP/1.1" 200 13751 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:49:23 +0900] "GET /arc/2005/01/2005/12/25/17/ HTTP/1.1" 200 8952 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:49:23 +0900] "GET /arc/2005/08/09/06/25/27/04/25/ HTTP/1.1" 200 15068 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:49:25 +0900] "GET /arc/2005/08/09/10/16/06/29/ HTTP/1.1" 200 14834 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:49:27 +0900] "GET /arc/2005/08/09/27/11/07/30/ HTTP/1.1" 200 14834 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:49:30 +0900] "GET /arc/2004/09/14/06/09/15/ HTTP/1.1" 200 11200 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:49:32 +0900] "GET /arc/2005/04/24/03/23/ HTTP/1.1" 200 12663 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:49:33 +0900] "GET /arc/2005/08/02/21/17/02/05/ HTTP/1.1" 200 10676 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:49:34 +0900] "GET /arc/2005/08/09/13/31/17/ HTTP/1.1" 200 14600 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:49:34 +0900] "GET /arc/2005/02/04/21/19/21/23/ HTTP/1.1" 200 15628 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:49:37 +0900] "GET /arc/2005/03/06/17/19/24/ HTTP/1.1" 200 15160 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:49:36 +0900] "GET /arc/2005/08/09/27/30/ HTTP/1.1" 200 14366 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:49:42 +0900] "GET /arc/2005/08/28/31/13/06/06/ HTTP/1.1" 200 19878 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:49:44 +0900] "GET /arc/2005/08/31/07/29/14/ HTTP/1.1" 200 10754 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:49:44 +0900] "GET /arc/2005/08/31/04/14/04/ HTTP/1.1" 200 10754 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:49:48 +0900] "GET /arc/2005/08/09/24/03/23/ HTTP/1.1" 200 14600 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:49:50 +0900] "GET /arc/2005/09/01/17/21/ HTTP/1.1" 200 10937 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:49:56 +0900] "GET /arc/2004/09/09/11/06/13/ HTTP/1.1" 200 10863 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:49:58 +0900] "GET /arc/2005/10/11/24/24/ HTTP/1.1" 200 11689 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:50:02 +0900] "GET /arc/2005/07/27/26/28/ HTTP/1.1" 200 11306 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:50:03 +0900] "GET /arc/2005/04/06/24/24/26/25/ HTTP/1.1" 200 16794 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:50:05 +0900] "GET /arc/2005/04/06/20/07/07/ HTTP/1.1" 200 16554 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:50:10 +0900] "GET /arc/2005/03/13/21/21/20/24/ HTTP/1.1" 200 14206 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:50:13 +0900] "GET /arc/2005/12/26/06/18/01/17/04/ HTTP/1.1" 200 14284 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:50:15 +0900] "GET /arc/2005/11/17/04/24/25/ HTTP/1.1" 200 11992 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:50:13 +0900] "GET /arc/2005/02/04/06/05/11/19/ HTTP/1.1" 200 15628 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:50:18 +0900] "GET /arc/2005/04/24/18/09/16/25/ HTTP/1.1" 200 13107 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:50:22 +0900] "GET /arc/2005/08/28/31/07/20/22/ HTTP/1.1" 200 19878 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:50:35 +0900] "GET /arc/2005/04/12/19/13/24/26/affiriate HTTP/1.1" 200 11361 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:50:38 +0900] "GET /arc/2005/12/30/22/04/09/ HTTP/1.1" 200 12711 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"
60.28.17.33 - - [19/Oct/2006:23:50:39 +0900] "GET /arc/2006/08/26/27/19/10/15/26/ HTTP/1.1" 200 11579 "-" "Baiduspider+(+http://www.baidu.com/search/spider.htm)"

トラックバック・コメント »

コメントはありません。

上に戻る

※スパム対策プラグインの影響により、すぐにトラックバックが反映されない場合があります。お手数ですが、半日ほど待ってみてください。

コメント投稿

※発言の責任を明確にするため「名無し」「通りすがり」「匿名希望」等の匿名は不可とします。捨てハンドルでもいいので必ず名乗ってください。
XHTML (使えるタグ): <a href="" title="" ktai=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong> <img localsrc="" alt=""> .
※スパム対策プラグインの影響により、すぐにコメント内容が表示されない場合があります。お手数ですが、半日ほど待ってみてください。

上に戻る