[analog-help] Identifying Known Spiders?

[Michael Crawford]

On Thu, Jul 3, 2008 at 8:37 AM, Jeremy Wadsack
<jeremy at 7simplemachines.com> wrote:
> The robots list from which that page was built no longer exists. The group that was maintaining it decided that it didn't make sense to maintain a database of "known robots" any more as anyone can make a robot.

In my personal case, it's not so much that I want to watch all the
bots, as to monitor my progress at getting a new site indexed by the
search engines.

While Google, Yahoo and MSN together provide the vast majority of
search engine referrals, there are still a few small, independent
players such as JGDO.

There are lots of reasons for running a bot, some good, some bad.  I'd
be happy if I could get a report of visits by the bots belong to, say,
the top half-dozen search engines.

Note that it often happens, with new sites, that a search engine
spider may not visit at all for months, and even then will only fetch
the home page.  By creating config files for each of my pages, I hope
to monitor spider visits throughout my site.

If this isn't yet possible with analog, I don't think it would be hard
to implement, and would be very popular, and so would get Analog a lot
more users, and maybe some consulting fees for Analog experts.
-- 
Michael David Crawford
mdcrawford at gmail dot com

 Enjoy my art, photography, music and writing at
http://www.geometricvisions.com/
 --- Free Music Downloads ---