[analog-help] Identifying Known Spiders?

[Aengus]

On 7/3/2008 3:48 AM, Michael Crawford wrote:
> I'd like to know the success of my efforts to submit a new site to all
> the search engines; some spiders won't visit a site until it's been
> online for a while, and some will only visit the home page.
> 
> I can see some of the spiders in the BROWSERREP and BROWSERSUM, but
> it's missing some because it's definitely missing Googlebot and Yahoo
> Slurp.
> 
> Also the BROWSERREP shows all the browsers used by my human visitors;
> it will get hard to spot spiders when my traffic picks up.
> 
> Is there a report specifically for known spiders?

No, the only special treatment for spiders in Analog is the ROBOTINCLUDE 
command which tells Analog to count the requests with the specified 
User-Agents as Search Engines in the OS Report.

There used to be a list of Spider User-Agents at 
http://www.wadsack.com/robot-list.html but it seems to be empty at the 
moment. There's a list from may 2007 at 
http://www2.owen.vanderbilt.edu/mike.shor/diversions/analog/RobotInclude.txt

You might want to do a report with FILEINCLUDE /robots.txt, which should 
give you a good indication of which search engines are hitting your site.

Aengus