[analog-help] Very complex apache log format

[Aengus]

At Friday, November 16, 2007 9:08 PM, Joshua S. Freeman 
<jf2412 at columbia.edu> wrote:

> Hi,
>
> I've read the Analog docs pretty closely.  I've used Analog on and
> off over the last 10 years and am always pleased to come back...
>
> Here are the facts:
>
> Httpd.conf logformat line:
>
> LogFormat "%h %l %u %t \"(%r)\" %>s %b \"(ref %{Referer}i)\" \
> \"(client %{User-agent}i)\" \"(elapsed %D)\"" mainserver
>
>
> What I put in my analog.cfg file:
>
> APACHELOGFORMAT (%h %l %u %t \"(%r)\" %>s %b \"(ref %{Referer}i)\"
> \"(client %{User-agent}i)\" \"(elapsed %D)\")
>
>
> Which, as far as I can read, *should* work.. However, when I run
> analog:
>
> $ analog -G +g./analog.cfg -v
> analog: analog version 6.0/Unix
> analog: Warning C: Too many arguments for configuration command:
> ignoring end of line starting:
>  APACHELOGFORMAT (%h %l %u %t \"(%r) \"
>  (For help on all errors and warnings, see docs/errors.html)

You're using () as a delimiter for your LOGFORMAT, even though you have () 
characters in the format string. Use a different delimiter, such as [].

APACHELOGFORMAT [%h %l %u %t \"(%r)\" %>s %b \"(ref %{Referer}i)\" 
\"(client %{User-agent}i)\" \"(elapsed %D)\"]

> analog: Warning C: Ignoring long configuration line starting
>  # cuscon11608.tstt.net.tt - - [04/Nov/2007:00:00:06 -0400] "(GET
> /imag

You probably put a line from your logfile into the .cfg so you could 
compare it to your LOGFORMAT.

> analog: Warning M: Logfile /u/2/j/jf2412/logfiles/*.20071110
>  contains lines with no bytes: byte counts may be low

Many of the reports will report on the bytes transferred - if you're not 
recording that information, Analog can't report on it.

> I do see that the hosts.txt file I set up is getting bigger and
> bigger so SOMETHING's happening..
>
> Of course I won't know until the file is done being parsed, etc.. But
> I'm concerned about this error... Is there ANYTHING you can suggest I
> do to make analog happier with the apache logformat?

That sounds like you're having Analog do it's own DNS lookups. You really 
need to look into some of the DNS Helper apps that will create the 
"hosts.txt" file for you much faster than Analog can do on it's own.

http://www.analog.cx/helpers/#dns

Aengus