Results 1 to 9 of 9
  1. #1
    Super Moderator
    Join Date
    Jun 2002
    Location
    Mt Macedon, Victoria, Australia
    Posts
    3,993
    Thanks
    1
    Thanked 45 Times in 44 Posts

    Analysing web logs

    For a couple of small web sites I look after (with Windows hosting) the web stats (AWstats) provided by the hosts only provides detailed reporting on the current month.
    so If I want a detailed report on October, I have to go and look at in on Oct 31 or perhaps Nov 1. Needless to say, I usually forget.

    The host has agreed to provide the raw logs to me so I can do it myself. I went looking today for a tool I can use to analyse the logs myself - preferably free. I did not find anything that was free. I downloaded one cheap tool, but I did not like its display much.

    Does anyone know of a good free tool for analysing web logs.

    Alternatively, I though I would just read the logs into an Access database, and do my own analysis. This is actually my preferred solution because that way I can ask just the questions I want to ask.

    I started to to build it, but I suspected that I was reinventing something that has been done many times. So I went looking, but again could not find any.
    Does anyone know of one?
    Regards
    John



  2. #2
    Platinum Lounger
    Join Date
    Feb 2002
    Location
    A Magic Forest in Deepest, Darkest Kent
    Posts
    5,681
    Thanks
    0
    Thanked 1 Time in 1 Post

    Re: Analysing web logs

    John

    I have had a hunt around and I have found that there is:

    Log Analyzer

    It appears to be shareware, I haven't checked it out and uses Excel as a viewer. It appears there are a few like it out there but it is a start.
    Jerry

  3. #3
    Super Moderator
    Join Date
    Jun 2002
    Location
    Mt Macedon, Victoria, Australia
    Posts
    3,993
    Thanks
    1
    Thanked 45 Times in 44 Posts

    Re: Analysing web logs

    Thanks Jerry

    I have downloaded Log Analyzer, but it seems that unless you pay to register the programme you can't get any reports out of it at all. I could not anyway.

    I remembered the Freeware Search engine mentioned a while ago in the Software Finds and Wants forum.

    This produced Web Log Expert Lite a freeware version of Web Log Expert. This produces a good range of well presented reports. As far as I can tell however, the lite version has a hidden list of file types, that results in mp3 files being excluded from the reports, and I am particularly interested in mp3 files.

    I have now found Alter Wind Log Analyzer Lite which is similar to Web Log Expert. This tells me a bit about about mp3 and pdf files, but not much. It again seems to be the case that there is a hidden list of ifle extensions that are used for different reports.
    Regards
    John



  4. #4
    Platinum Lounger
    Join Date
    Feb 2002
    Location
    A Magic Forest in Deepest, Darkest Kent
    Posts
    5,681
    Thanks
    0
    Thanked 1 Time in 1 Post

    Re: Analysing web logs

    <img src=/S/grin.gif border=0 alt=grin width=15 height=15>, I saw you looking at the Freeware Search Engine earlier, but I am glad you have found a solution of sorts at the moment.
    Jerry

  5. #5
    Super Moderator jscher2000's Avatar
    Join Date
    Feb 2001
    Location
    Silicon Valley, USA
    Posts
    23,112
    Thanks
    5
    Thanked 93 Times in 89 Posts

    Re: Analysing web logs

    I recently was using the program visitors to analyze some IIS logs. Because visitors doesn't read the IIS format directly, I had to convert the logs first. I created a little VBA routine that lets me read in as many IIS logs as I like, pumps out an NCSA-format log compatible with visitors, and then creates batch files that cause visitors to generate two reports and provides links to open them in my default browser. If you'd like to try it, you can download these two items and try it:
    1. <LI>visitors.exe: http://www.hping.org/visitors/visitors.exe (command-line program)

      <LI>Word doc with macro: http://jscher2000.home.att.net/webdev/iis2visitorsA.doc
    Note that the Word code uses a file picker that comes with Word 2002 and 2003; it doesn't support Word 2000. One could rewrite the code to use a different file picker, but my priority was speed of implementation because this was all "nights and weekends" development.

    By the way, I did start trying to install AWStats on my computer. I downloaded ActivePerl, but I didn't see any clear instructions on "how to run AWStats on Windows without the Apache server," so I just stopped trying. If you much prefer its reports, however, I'm sure there is a way. <img src=/S/smile.gif border=0 alt=smile width=15 height=15>

  6. #6
    Super Moderator
    Join Date
    Jun 2002
    Location
    Mt Macedon, Victoria, Australia
    Posts
    3,993
    Thanks
    1
    Thanked 45 Times in 44 Posts

    Re: Analysing web logs

    Thank you

    I had previously downloaded Visitors, but thought that trying to convert the log files seemed a lot of trouble.

    Whenever I try to use your Word Macro, it gives me a subscript out of range error at the line :

    strFields(15) = Lcase(strFields(15))

    My log files seem to have only 15 fields. Your macro seems to be expecting them to have 21. Here is the top of a log file.

    #Software: Microsoft Internet Information Services 6.0
    #Version: 1.0
    #Date: 2005-10-03 00:43:50
    #Fields: date time s-ip cs-method cs-uri-stem cs-username c-ip cs-version cs(User-Agent) cs(Referer) cs-host sc-status sc-bytes cs-bytes time-taken
    2005-10-03 00:43:50 66.179.153.16 GET /bandstyles.css - 220.233.164.90 HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+MyIE2;+ SV1;+Maxthon;+.NET+CLR+1.1.4322) http://www.melbournehighlandpipes.com.au/ www.melbournehighlandpipes.com.au 200 5240 377 703
    Regards
    John



  7. #7
    Super Moderator jscher2000's Avatar
    Join Date
    Feb 2001
    Location
    Silicon Valley, USA
    Posts
    23,112
    Thanks
    5
    Thanked 93 Times in 89 Posts

    Re: Analysing web logs

    Yes, I was using files from 1&1, which I thought were in a standard "IIS W3C Extended" format. Perhaps your format also is a standard format, but not quite the same one. <img src=/S/grin.gif border=0 alt=grin width=15 height=15> I probably can look at this after hours.

    Note to self: here's a handy Microsoft page with IIS 6 format info: Log File Formats in IIS (IIS 6.0)

  8. #8
    Super Moderator
    Join Date
    Jun 2002
    Location
    Mt Macedon, Victoria, Australia
    Posts
    3,993
    Thanks
    1
    Thanked 45 Times in 44 Posts

    Re: Analysing web logs

    Thanks

    I realized that your Word macro had done most of the work needed to import the logs into a database, so I have modified it to do that (using my log file format).

    At the moment it imports all the fields, but I think I will skip a few that always have the same information.

    The two Analysis tools in mentioned above both do a good job, and having the logs in a database means I can answer other add hoc questions (specifically about the downloads of mp3 and pdf files) using queries.
    Attached Files Attached Files
    Regards
    John



  9. #9
    Super Moderator jscher2000's Avatar
    Join Date
    Feb 2001
    Location
    Silicon Valley, USA
    Posts
    23,112
    Thanks
    5
    Thanked 93 Times in 89 Posts

    Re: Analysing web logs

    That's a great idea, and if you find that the query language is not efficient for something that visitors.exe can do quickly, you can export the data into the NCSA format. Win-win!

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •