Encourage FUNIX

The consultation of the site is completely free and without advertisements. Donations are nevertheless appreciated to pay for hosting and encourage its author


Analyzing apache logs with goaccess and awstats
[ Presentation | Analyzing Apache logs with goaccess( installation   ,  configuration , usage ) | Analyze logs with awstats ( Overview , installation , usage ) | Analyze logs of a site not hosted locally ]

Last modified August 11, 2024

Analyzing Apache logs with goaccess and Awstats

LAMP Configuration

This site has been automatically translated with Google Translate from this original page written in french, there may be some translation errors


Presentation

This page presents two tools for analyzing Apache logs , from the most recent goaccess to the oldest awstats, each with its advantages and disadvantages. The first allows daily analysis or even real-time analysis and allows visualization in a terminal, which can be practical depending on how you administer your system. The second allows monthly reports.
 
The official site is https://goaccess.io/ where you can retrieve the archive and decompress it by typing

tar xvfz goaccess-1.9.3.tar.gz

this gives the directory goaccess-1.9.3/

But first, we will install the library to read MaxMind DB files, including the GeoIP2 databases , which will be used to geolocate IP addresses. The official website https://github.com/maxmind/libmaxminddb we retrieve the archive that we unzip by typing

tar xvfz libmaxminddb-1.10.0.tar.gz

This gives the directory libmaxminddb-1.10.0

We will first install the ncursesw-devel package

Apparently there are some missing bits of code under libmaxminddb-1.10.0 , we will get a copy of libtap here https://github.com/zorgnax/libtap/tree/b53e4ef5257f80e881762b6143834d8aae29da1a , which we will uncompress under libmaxminddb-1.10.0/t/libtap by successively typing the following commands

unzip libtap-b53e4ef5257f80e881762b6143834d8aae29da1a.zip
cd libtap-b53e4ef5257f80e881762b6143834d8aae29da1a/
mv * ..
mv.gitignore..
mv.travis.yml..
cd..
rmdir libtap-b53e4ef5257f80e881762b6143834d8aae29da1a/

under libmaxminddb-1.10.0 we finally type

./configure

make

and as root make install

We return to the  goaccess-19.3 directory and type

./configure --enable-utf8 --enable-geoip=mmdb

here the result

Your build configuration:

  Prefix         : /usr/local
  Package        : goaccess
  Version        : 1.9.3
  Compiler flags :  -pthread
  Linker flags   : -lncursesw -lmaxminddb -lpthread -lintl -lintl 
  UTF-8 support  : yes
  Dynamic buffer : no
  ASan           : no
  Geolocation    : GeoIP2
  Storage method : In-Memory with On-Disk Persistent Storage
  TLS/SSL        : no
  Bugs           : hello@goaccess.io

type make then as root make install

Configuration

Everything happens in the file /usr/local/etc/goaccess/goaccess.conf

I only changed three parameters which can be found below

######################################
# Time Format Options (required)
######################################
#
# The hour (24-hour clock) [00,23]; leading zeros are permitted but not required.
# The minute [00,59]; leading zeros are permitted but not required.
# The seconds [00,60]; leading zeros are permitted but not required.
# See `man strftime` for more details
#
# The following time format works with any of the
# Apache/NGINX's log formats below.
#
time-format %H:%M:%S

(...)

######################################
# Date Format Options (required)
######################################
#
# The date-format variable followed by a space, specifies
# the log format date containing any combination of regular
# characters and special format specifiers. They all begin with a
# percentage (%) sign. See `man strftime`
#
# The following date format works with any of the
# Apache/NGINX's log formats below.
#
date-format %d/%b/%Y

(...)

######################################
# Log Format Options (required)
######################################

log-format %v %h %^[%d:%t %^] (%T s) "%r" %s %b "%R" "%u"' --date-format=%d/%b/%Y --time-format=%T

This last option is probably the most important option, it is the format of the log file. The custom format that is indicated corresponds to my funix logs.

www.funix.org:80 74.125.217.134 - - [10/Aug/2024:02:04:33 +0200] "GET /images/style.css HTTP/1.0" 301 561 "http://www.funix.org/" "Mozilla/5.0 (Linux; Android 4.0.4; Galaxy Nexus Build/IMM76B) AppleWebKit/537.36 (KHTML, like Gecko; Mediapartners-Google) Chrome/126.0.6478.182 Mobile Safari/537.36"

Use

Just type goaccess with the name of the log file after it, everything happens in a terminal

This site will give you an idea of ​​all the information you can have. Now if you prefer a visualization via html, just type

goaccess www.funix.org.log -a -o funixlog.html

and this will give something more attractive, below two views



 


 

Analyzing Apache logs with awstats

Presentation

Awstats is a tool to analyze Apache logs, you can find it at the URL awstats.sourceforge.net, it is an archive in tarball format awstats-7.9.tar.gz that we will retrieve.

Installation and configuration

We unzip the archive by typing:

tar xvfz awstats-7.9.tar.gz

This will give us the awstats-7.9 directory . First we will have to slightly modify the apache httpd.conf configuration file in case you are using virtual hosts, here is an example configuration

<VirtualHost 192.168.13.11:1234>
ServerName funix.homelinux.org
DocumentRoot /roger/homepage/online
ErrorLog logs/online-error_log
TransferLog logs/online-access_log
CustomLog logs/online-access_log combined
</VirtualHost>

We must put combined for the CustomLog parameter . Don't forget to restart apache in case of modification

systemctl restart httpd.service

Now we need to create the directory /etc/awstats

mkdir /etc/awstats

For the configuration of awstats we will go to the tools directory

cd awstats-7.9/tools

In the case of an update we will type

perl awstats_updateall.pl now -awstatsprog=/absolute-path/awstats-7.9/wwwroot/cgi-bin/awstats.pl -configdir=/etc/awstats/

/etc/awstats is the directory where your servers configuration files are located. Here is the result

Running '"/home/olivier/web/awstats-7.9/wwwroot/cgi-bin/awstats.pl" -update -config=funix.kervao.fr -configdir="/etc/awstats/"' to update config funix.kervao.fr
Failed to open blacklist: No such file or directory

Running '"/home/olivier/web/awstats-7.9/wwwroot/cgi-bin/awstats.pl" -update -config=www.funix.org -configdir="/etc/awstats/"' to update config www.funix.org
Create/Update database for config "/etc/awstats/awstats.www.funix.org.conf" by AWStats version 7.9 (build 20230108)
From data in log file "/usr/local/apache2/logs/funix.org/www.funix.org.log"...
Phase 1: First bypass old records, searching new record...
Direct access after last parsed record (after line 6479)
Jumped lines in file: 6479
 Found 6479 already parsed records.
Parsed lines in file: 0
 Found 0 dropped records,
 Found 0 comments,
 Found 0 blank records,
 Found 0 corrupted records,
 Found 0 old records,
 Found 0 new qualified records.

Running '"/home/olivier/web/awstats-7.9/wwwroot/cgi-bin/awstats.pl" -update -config=www.hoarau.org -configdir="/etc/awstats/"' to update config www.hoarau.org
Create/Update database for config "/etc/awstats/awstats.www.hoarau.org.conf" by AWStats version 7.9 (build 20230108)
From data in log file "/usr/local/apache2/logs/hoarau.org/www.hoarau.org.log"...
Phase 1: First bypass old records, searching new record...
Direct access after last parsed record (after line 7827)
Jumped lines in file: 7827
 Found 7827 already parsed records.
Parsed lines in file: 0
 Found 0 dropped records,
 Found 0 comments,
 Found 0 blank records,
 Found 0 corrupted records,
 Found 0 old records,
 Found 0 new qualified records.

In the case of a first installation, we will type (in the example below, I have two Apache servers ( version 1.3 and 2) with the virtual web servers http://funix.homelinux.org:1234 and http://ohoarau.homelinux.org:1234 ) as root

perl awstats_configure.pl

here is the result

----- AWStats awstats_configure 1.0 (build 20140126) (c) Laurent Destailleur -----
This tool will help you to configure AWStats to analyze statistics for
one web server. You can try to use it to let it do all that is possible
in AWStats setup, however following the step by step manual setup
documentation (docs/index.html) is often a better idea. Above all if:
- You are not an administrator user,
- You want to analyze downloaded log files without web server,
- You want to analyze mail or ftp log files instead of web log files,
- You need to analyze load balanced servers log files,
- You want to 'understand' all possible ways to use AWStats...
Read the AWStats documentation (docs/index.html).

-----> Running OS detected: Linux, BSD or Unix
Warning: AWStats standard directory on Linux OS is '/usr/local/awstats'.
If you want to use standard directory, you should first move all content
of AWStats distribution from current directory:
/usr/local/linux/web/awstats-7.9
to standard directory:
/usr/local/awstats
And then, run configure.pl from this location.
Do you want to continue setup from this NON standard directory [yN] ? y

here it finds the configuration files of my two Apache servers

-----> Check for web server install
  Found Web server Apache config file '/usr/local/apache/conf/httpd.conf'
  Found Web server Apache config file '/usr/local/apache2/conf/httpd.conf'

It modifies both configuration files, it also modifies files to update the log archiving format

-----> Check and complete web server config file '/usr/local/apache2/conf/httpd.conf'
Warning: You Apache config file contains directives to write 'common' log files
This means that some features can't work (os, browsers and keywords detection).
Do you want me to setup Apache to write 'combined' log files [y/N] ? y

Add 'Alias /awstatsclasses "/usr/local/linux/web/awstats-7.9/wwwroot/classes/"'
  Add 'Alias /awstatscss "/usr/local/linux/web/awstats-7.9/wwwroot/css/"'
  Add 'Alias /awstatsicons "/usr/local/linux/web/awstats-7.9/wwwroot/icon/"'
  Add 'ScriptAlias /awstats/ "/usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin/"'
  Add '<Directory>' directive
  AWStats directives added to Apache config file.

It updates its own configuration file

-----> Update model config file '/usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin/awstats.model.conf'
  File awstats.model.conf updated.

-----> Need to create a new config file ?
Do you want me to build a new AWStats config/profile
file (required if first install) [y/N] ? y

Here we indicate the URL of the web server to analyze (address:port)

-----> Define config file name to create
What is the name of your web site or profile analysis ?
Example: www.mysite.com
Example: demo
Your web site, virtual server or profile name:
> funix.homelinux.org:1234

Defining the path to store the configuration files for each server to be analyzed

-----> Define config file path
In which directory do you plan to store your config file(s) ?
Default: /etc/awstats
Directory path to store config file(s) (Enter for default):
>

Creating the server configuration file mentioned above

-----> Create config file '/etc/awstats/awstats.funix.homelinux.org:1234.conf'
 Config file /etc/awstats/awstats.funix.homelinux.org:1234.conf created.

Apache is restarted automatically

-----> Restart Web server with '/sbin/service httpd restart'
Redirecting to /bin/systemctl restart httpd.service

-----> Add update process inside a scheduler
Sorry, configure.pl does not support automatic add to cron yet.
You can do it manually by adding the following command to your cron:
/usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin/awstats.pl -update -config=funix.homelinux.org:1234
Or if you have several config files and prefer having only one command:
/usr/local/linux/web/awstats-7.9/tools/awstats_updateall.pl now
Press ENTER to continue...

A SIMPLE config file has been created: /etc/awstats/awstats.funix.homelinux.org:1234.conf
You should have a look inside to check and change manually main parameters.
You can then manually update your statistics for 'funix.homelinux.org:1234' with command:
> perl awstats.pl -update -config=funix.homelinux.org:1234
You can also read your statistics for 'funix.homelinux.org:1234' with URL:
> http://localhost/awstats/awstats.pl?config=funix.homelinux.org:1234


Press ENTER to finish...

He created a configuration file for the site concerned /etc/awstats/awstats.funix.homelinux.org:1234.conf in this file I modified the following parameters:

- The one which indicates the path of the raw log file to analyze

LogFile="/usr/local/apache/logs/online-access_log"

- to enable DNS name resolution

DNSLookup=1

- Directory where awstats will store its statistics

DirData="/usr/local/var/awstats"

- To indicate the index file of your site

DefaultFile="index.php"

- To ignore the machines of your domain

SkipHosts="127.0.0.1 REGEX[^192\.168\.13\.] REGEX[\.kervao\.fr$]"

- If your site is identified by another URL, you must add it here (put a space as a separator field)

HostAliases="funix.homelinux.org:1234"

To combat spam referrers , you need to enable the following variable

SkipReferrersBlackList="/etc/awstats/blacklist.txt"

with a blacklist.txt file that we will retrieve under awstats-7.9/wwwroot/cgi-bin/lib

A much more recent blacklist can be found here : https://perishablepress.com/blacklist/ultimate-referrer-blacklist.txt

in this file according to my tests it would seem that the first part which begins with RewriteCond is useless for awstats, it is only useful if you filter SPAM referrers directly on access to your site with a .htaccess be careful this method will result in slowing down access to your site. See here an interesting discussion on the subject.

It's only the second part that is actually useful and works with awstats , it starts with

# This is the URL blacklist from the chongqed.org database
# it is available from http://blacklist.chongqed.org/
# You can use each line below as a regular expression
# that can be tested against URLs on your wiki.
# The last spammer was added on 2008-09-11 10:14:51
# Check http://blacklist.chongqed.org/ for updates

I started to complete the list with the following lines:

https?:\/\/([^\/]*\.)?proxtrail\.ru
https?:\/\/([^\/]*\.)?denterum\.ru
https?:\/\/([^\/]*\.)?profeservice\.ru
https?:\/\/([^\/]*\.)?bazakanstovarov\.com
https?:\/\/([^\/]*\.)?balkanfarma\.ru
https?:\/\/([^\/]*\.)?sobervoditel\.ru
https?:\/\/([^\/]*\.)?avtokor-23\.ru
https?:\/\/([^\/]*\.)?avtokor23\.ru
https?:\/\/([^\/]*\.)?xn--j1at1a.xn--p1ai
https?:\/\/([^\/]*\.)?rupolitshow\.ru
https?:\/\/([^\/]*\.)?vyezd-viyezd\.ru
https?:\/\/([^\/]*\.)?igru-2015\.ru
https?:\/\/([^\/]*\.)?jeribetejewu\.c0\.pl
https?:\/\/([^\/]*\.)?creditservise\.ru
https?:\/\/([^\/]*\.)?megamashiny\.com
https?:\/\/([^\/]*\.)?seoxbeep\.ru
https?:\/\/([^\/]*\.)?woman3050\.ru
https?:\/\/([^\/]*\.)?www.vselgoty\.ru
https?:\/\/([^\/]*\.)?playpokeronline\.dk
https?:\/\/([^\/]*\.)?medical\.in\.ua
https?:\/\/([^\/]*\.)?viagralevitradzheneriki\.ru
https?:\/\/([^\/]*\.)?\.com\.ua
https?:\/\/([^\/]*\.)?tasgroup\.it
https?:\/\/([^\/]*\.)?\.co\.ua
https?:\/\/([^\/]*\.)?kiev\.ua
https?:\/\/([^\/]*\.)?bringtwo\.net
https?:\/\/([^\/]*\.)?cleaningservices\.kiev\.ua
https?:\/\/([^\/]*\.)?studio-topkapi\.eu
https?:\/\/([^\/]*\.)?kruchen\.com\.ua
https?:\/\/([^\/]*\.)?freedom\.co\.ua
https?:\/\/([^\/]*\.)?bringtwo\.net
https?:\/\/([^\/]*\.)?studio-topkapi\.eu
https?:\/\/([^\/]*\.)?c0\.pl
https?:\/\/([^\/]*\.)?bazakanstovarov\.com
https?:\/\/([^\/]*\.)?xikiz\.com
https?:\/\/([^\/]*\.)?lesbianmilf\.xblog\.in
https?:\/\/([^\/]*\.)?pamyatniki-in-kiev\.com\.ua
https?:\/\/([^\/]*\.)?carivka\.com\.ua
https?:\/\/([^\/]*\.)?agent-05\.su
https?:\/\/([^\/]*\.)?websolution\.com\.ua
https?:\/\/([^\/]*\.)?pamjatnik\.com\.ua
https?:\/\/([^\/]*\.)?startimes\.com
https?:\/\/([^\/]*\.)?arktech\.co
https?:\/\/([^\/]*\.)?goohey\.com
https?:\/\/([^\/]*\.)?himalayan-imports\.com
https?:\/\/([^\/]*\.)?pizza-imperia\.com
https?:\/\/([^\/]*\.)?nowellgroup\.com
https?:\/\/([^\/]*\.)?ddrgame\.com
https?:\/\/([^\/]*\.)?torrinomedica\.it
https?:\/\/([^\/]*\.)?oliveriobalcells\.com
https?:\/\/([^\/]*\.)?giocagiocagioca\.com

It quickly got on my nerves because every day there are new sites that appear, the easiest thing is to block certain extensions outright!

^https?://[^/]+\.ru
^https?://[^/]+\.ua
^https?://[^/]+\.su ^
https?://[^/]+\.link
^https?://[^/]+\.cc
^https?://[^/]+\.in


there we will block all sites in .ru , in .ua , etc. While we're at it, since it's not necessarily interesting to see the referrers of your own domain, we also block

https?:\/\/([^\/]*\.)?hoarau\.org
https?:\/\/([^\/]*\.)?funix\.org


and there I can guarantee you that the list of referrers is cleaned up and finally usable.

for this to be taken into account with awstats you have to delete what is under /usr/local/var/awstats/ and restart awstats for all the weekly log files

Remember to create this directory beforehand

mkdir /usr/local/var/awstats

To see if everything works, we now place ourselves as root under awstats-7.9/wwwroot/cgi-bin and type

./awstats.pl -config=funix.homelinux.org:1234 -update

This is the result

Create/Update database for config "/etc/awstats/awstats.www.hoarau.org.conf" by AWStats version 7.9 (build 20230108)
From data in log file "/usr/local/apache2/logs/hoarau.org/www.hoarau.org.log"...
Phase 1: First bypass old records, searching new record...
Direct access after last parsed record (after line 7827)
Jumped lines in file: 7827
 Found 7827 already parsed records.
Parsed lines in file: 0
 Found 0 dropped records,
 Found 0 comments,
 Found 0 blank records,
 Found 0 corrupted records,
 Found 0 old records,
 Found 0 new qualified records.

On previous versions there was a slight bug, here is a typical log line

192.168.0.11 - - [19/Feb/2004:10:58:17 +0100] "GET /images/journal.gif HTTP/1.1" 304 - "http://funix.homelinux.org:1234/fr/linux/index.php?ref=tips&page=menu" "Mozilla/5.0 (X11; U; Linux i686; fr-FR; rv:1.4) Gecko/20
030630"

Or sometimes Apache archives this

192.168.0.11 - - [19/Feb/2004:10:58:24 +0100] "GET / HTTP/1.1" 200 17493

In the absence of information, it leaves empty fields and awstats does not appreciate it (it does not analyze the logs...). I do not have enough perspective to know if it was corrected.

If you have a second virtual site, create another file /etc/awstats/awstats.my-virtual-domain.conf on the same model as before. In my case, just type

cp /etc/awstats/awstats.funix.homelinux.org:1234.conf  /etc/awstats/awstats.ohoarau.homelinux.org:1234.conf

In this file remember to modify the parameters for the URL and the Apache log file to analyze.

Note that given the method used (see next chapter) to use awstats , I deleted the lines that awstats added to my Apache configuration files.
If you want to use awstats as an accessible cgi-bin from Apache, you must leave these lines.

Use

This is how I use awstats, I created an awstats file (executable rights) under /etc/cron.daily which therefore runs every day and allows me to produce the stats for my two virtual web addresses

#!/bin/bash
/usr/local/linux/web/awstats-7.9/tools/awstats_buildstaticpages.pl -config=ohoarau.homelinux.org:1234 -update -lang=fr -dir=/usr/local/apache2/htdocs/awstats/ohoarau -awstatsprog=/usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin/awstats.pl

/usr/local/linux/web/awstats-7.9/tools/awstats_buildstaticpages.pl -config=funix.homelinux.org:1234 -update -lang=fr -dir=/usr/local/apache2/htdocs/awstats/online -awstatsprog=/usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin/awstats.pl


The -dir option allows you to save the statistics obtained in html format in the appropriate directory. Finally, the awstats icon directory must be accessible on your web server. For this, in the Apache configuration file, I added the following directives

Alias ​​/awstatsicons/ "/usr/local/linux/web/awstats-7.9/wwwroot/icon/"

<Directory "/usr/local/linux/web/awstats-7.9/wwwroot/icon">
    Options Indexes MultiViews
    AllowOverride None
    Order allow,deny
    Allow from all
</Directory>

You will of course replace with the absolute path of awstats. By the way I put the rights that were right for the awstats directory awstats

chmod 755 awstats-7.9/

This is what it gives when I launch my script /etc/cron.daily/awstats

Launch update process: "/usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin/awstats.pl" -config=ohoarau.homelinux.org:1234 -update -configdir=
Build main page: "/usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin/awstats.pl" -config=ohoarau.homelinux.org:1234 -staticlinks -lang=fr -output
Build alldomains page: "/usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin/awstats.pl" -config=ohoarau.homelinux.org:1234 -staticlinks -lang=fr -output=alldomains
Build allhosts page: "/usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin/awstats.pl" -config=ohoarau.homelinux.org:1234 -staticlinks -lang=fr -output=allhosts
Build lasthosts page: "/usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin/awstats.pl" -config=ohoarau.homelinux.org:1234 -staticlinks -lang=fr -output=lasthosts
Build unknownip page: "/usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin/awstats.pl" -config=ohoarau.homelinux.org:1234 -staticlinks -lang=fr -output=unknownip
Build allrobots page: "/usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin/awstats.pl" -config=ohoarau.homelinux.org:1234 -staticlinks -lang=fr -output=allrobots
Build lastrobots page: "/usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin/awstats.pl" -config=ohoarau.homelinux.org:1234 -staticlinks -lang=fr -output=lastrobots
Build session page: "/usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin/awstats.pl" -config=ohoarau.homelinux.org:1234 -staticlinks -lang=fr -output=session
Build urldetail page: "/usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin/awstats.pl" -config=ohoarau.homelinux.org:1234 -staticlinks -lang=fr -output=urldetail
Build urlentry page: "/usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin/awstats.pl" -config=ohoarau.homelinux.org:1234 -staticlinks -lang=fr -output=urlentry
Build urlexit page: "/usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin/awstats.pl" -config=ohoarau.homelinux.org:1234 -staticlinks -lang=fr -output=urlexit
Build osdetail page: "/usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin/awstats.pl" -config=ohoarau.homelinux.org:1234 -staticlinks -lang=fr -output=osdetail
Build unknownos page: "/usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin/awstats.pl" -config=ohoarau.homelinux.org:1234 -staticlinks -lang=fr -output=unknownos
Build browserdetail page: "/usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin/awstats.pl" -config=ohoarau.homelinux.org:1234 -staticlinks -lang=fr -output=browserdetail
Build unknownbrowser page: "/usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin/awstats.pl" -config=ohoarau.homelinux.org:1234 -staticlinks -lang=fr -output=unknownbrowser
Build refererse page: "/usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin/awstats.pl" -config=ohoarau.homelinux.org:1234 -staticlinks -lang=fr -output=refererse
Build refererpages page: "/usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin/awstats.pl" -config=ohoarau.homelinux.org:1234 -staticlinks -lang=fr -output=refererpages
Build keyphrases page: "/usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin/awstats.pl" -config=ohoarau.homelinux.org:1234 -staticlinks -lang=fr -output=keyphrases
Build keywords page: "/usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin/awstats.pl" -config=ohoarau.homelinux.org:1234 -staticlinks -lang=fr -output=keywords
Build errors404 page: "/usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin/awstats.pl" -config=ohoarau.homelinux.org:1234 -staticlinks -lang=fr -output=errors404
20 files built.
Main HTML page is 'awstats.ohoarau.homelinux.org:1234.html'.
Launch update process : "/usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin/awstats.pl" -config=funix.homelinux.org:1234 -update -configdir=
Build main page: "/usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin/awstats.pl" -config=funix.homelinux.org:1234 -staticlinks -lang=fr -output
Build alldomains page: "/usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin/awstats.pl" -config=funix.homelinux.org:1234 -staticlinks -lang=fr -output=alldomains
Build allhosts page: "/usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin/awstats.pl" -config=funix.homelinux.org:1234 -staticlinks -lang=fr -output=allhosts
Build lasthosts page: "/usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin/awstats.pl" -config=funix.homelinux.org:1234 -staticlinks -lang=fr -output=lasthosts
Build unknownip page: "/usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin/awstats.pl" -config=funix.homelinux.org:1234 -staticlinks -lang=fr -output=unknownip
Build allrobots page: "/usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin/awstats.pl" -config=funix.homelinux.org:1234 -staticlinks -lang=fr -output=allrobots
Build lastrobots page: "/usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin/awstats.pl" -config=funix.homelinux.org:1234 -staticlinks -lang=fr -output=lastrobots
Build session page: "/usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin/awstats.pl" -config=funix.homelinux.org:1234 -staticlinks -lang=fr -output=session
Build urldetail page: "/usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin/awstats.pl" -config=funix.homelinux.org:1234 -staticlinks -lang=fr -output=urldetail
Build urlentry page: "/usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin/awstats.pl" -config=funix.homelinux.org:1234 -staticlinks -lang=fr -output=urlentry
Build urlexit page: "/usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin/awstats.pl" -config=funix.homelinux.org:1234 -staticlinks -lang=fr -output=urlexit
Build osdetail page: "/usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin/awstats.pl" -config=funix.homelinux.org:1234 -staticlinks -lang=fr -output=osdetail
Build unknownos page: "/usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin/awstats.pl" -config=funix.homelinux.org:1234 -staticlinks -lang=fr -output=unknownos
Build browserdetail page: "/usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin/awstats.pl" -config=funix.homelinux.org:1234 -staticlinks -lang=fr -output=browserdetail
Build unknownbrowser page: "/usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin/awstats.pl" -config=funix.homelinux.org:1234 -staticlinks -lang=fr -output=unknownbrowser
Build refererse page: "/usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin/awstats.pl" -config=funix.homelinux.org:1234 -staticlinks -lang=fr -output=refererse
Build refererpages page: "/usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin/awstats.pl" -config=funix.homelinux.org:1234 -staticlinks -lang=fr -output=refererpages
Build keyphrases page: "/usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin/awstats.pl" -config=funix.homelinux.org:1234 -staticlinks -lang=fr -output=keyphrases
Build keywords page: "/usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin/awstats.pl" -config=funix.homelinux.org:1234 -staticlinks -lang=fr -output=keywords
Build errors404 page: "/usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin/awstats.pl" -config=funix.homelinux.org:1234 -staticlinks -lang=fr -output=errors404
20 files built.
Main HTML page is 'awstats.funix.homelinux.org:1234.html'.

if you have an error like

bash: /usr/local/linux/web/awstats-7.9/tools/awstats_buildstaticpages.pl: /usr/bin/perl^M: bad interpreter: No such file or directory

just edit the file awstats_buildstaticpages.pl and remove the ^M after perl (with mcedit for example).

now if you have the following error awstats.pl permission not granted, it is because the rights of awstats are not good. You have to set it to 755 like this

chmod 755 /usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin/awstats.pl

Now under the directories /usr/local/apache2/htdocs/awstats/online and /usr/local/apache2/htdocs/awstats/ohoarau I end up with html files, for the first the input file is awstats.funix.homelinux.org:1234.html and for the other awstats.ohoarau.homelinux.org:1234.html
From a web page accessible on your web server (home page for example), I made a link to these statistics pages.
You may need to add the /usr/local/apache2/htdocs/awstats/online directory in a Directory directive in the httpd.conf file if necessary. I also had to create the following directive for the icons to be visible

<Directory "/usr/local/linux/web/awstats-7.9/wwwroot/icon">
   Options Indexes FollowSymLinks Includes ExecCGI
   AllowOverride All
   Require all granted
</Directory>

Here are some (non-exhaustive) overviews








To be able to do the stats of a site that you do not host locally, it's here .

Disadvantage of this method the pages built are static and we cannot access the history of previous months. It will be necessary to go through the Apache CGI extension that will have to be activated, then we add the following lines in the httpd.conf file

Alias /awstatsclasses "/usr/local/linux/web/awstats-7.9/wwwroot/classes/"
Alias /awstatscss "/usr/local/linux/web/awstats-7.9/wwwroot/css/"
Alias /awstatsicons "/usr/local/linux/web/awstats-7.9/wwwroot/icon/"
ScriptAlias /awstats/ "/usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin/"

<Directory "/usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin">
    Options ExecCGI
    AllowOverride None
    Order allow,deny
    Allow from all
</Directory>

<Directory "/usr/local/linux/web/awstats-7.9/wwwroot">
   Options Indexes FollowSymLinks Includes ExecCGI
   AllowOverride All
   Require all granted
</Directory>


The site will be accessible by typing as URL http://mywebserver/awstats/awstats.pl?config= funix.homelinux.org:1234 and here is the result


we find a menu banner on the left and we can choose the month of analysis with a drop-down menu.

Analyse logs of a non locally hosted site

To be concrete, I have my site funix.org which is hosted at online , I have a way to retrieve the raw Apache logs, each day a log is created and is found in a tree structure year/month/day_access.log.gz. With cron I request a daily update of the stats, I request it for the day before.
Here is the script that allows you to retrieve the stats from the previous day and launch the analyses by awstats and goaccess to make them accessible via a local Apache server, and which places it in the directory /usr/local/apache2/logs/funix.org

#!/bin/bash
# date format construction
day=`date +%e`
month=`date +%m`
year=`date +%Y`
month31="may july october december"
month30="february april june september november"
if [ $day -eq "1" ];
 then
   monthb=`date +%B`
   for i in $month31
   do
        if [ $monthb == $i ]
        then
             yesterday=30
         month=`expr $month - 1`
        fi
   done
   for i in $month30
   do
        if [ $monthb == $i ]
        then
              yesterday=31
          month=`expr $month - 1`
        fi
   done
   if [ $monthb == "August" ]
   then
       yesterday=31
       month=07
   fi
   if [ $monthb == "March" ]
    then
       yesterday=28
       month=02
   fi
   if [ $monthb == "January" ]
   then
           yesterday=31
           month=12
       year=`expr $year - 1`
   fi
 else
   yesterday=`expr $day - 1`
 fi
if [ "${#yesterday}" = 1 ]
   then yesterday="0"$yesterday
fi
if [ "${#month}" = 1 ]
   then month="0"$month
fi
# I move to the correct directory and delete the old log files
cd /usr/local/apache2/logs/funix.org
rm -f /usr/local/apache2/logs/funix.org/*.log
# retrieve logs by ftp
wget -c ftp://stats.online.net/archives/"$annee"/"$mois"/"$hier"_access.log.gz --user=webmaster@funix.org --password=mot-de-passe
# I unzip the archive and rename it
gunzip "$hier"_access.log.gz
mv "$hier"_access.log www.funix.org.log

#analysis with awstats
/usr/local/linux/web/awstats-7.9/tools/awstats_buildstaticpages.pl -config=www.funix.org -update -lang=fr -dir=/usr/local/apache2/htdocs/awstats/funix.org -awstatsprog=/usr/local/linux/web/awstats-7.9/wwwroot/cgi-bin/awstats.pl

#analysis with goaccess
goaccess /usr/local/apache/logs/funix.org/www.funix.org.log -a -o  /usr/local/apache/logs/funix.org/funixlog.log

To make stats with awstats, I created a file /etc/awstats/awstats.www.funix.org.conf in which I modified the following parameters

LogFile="/usr/local/apache/logs/funix.org/www.funix.org.log"
SiteDomain="www.funix.org"
HostAliases="www.funix.org"

My raw log file looks like this

www.funix.org:80 109.8.158.150 - - [01/Jan/2017:01:02:29 +0100] "GET /fr/unix/awk.htm HTTP/1.0" 200 8711 "https://www.google.fr/" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:50.0) Gecko/20100101 Firefox/50.0"
www.funix.org:80 109.8.158.150 - - [01/Jan/2017:01:02:29 +0100] "GET /favicon.ico HTTP/1.0" 404 468 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:50.0) Gecko/20100101 Firefox/50.0"
www.funix.org:80 109.8.158.150 - - [01/Jan/2017:01:02:29 +0100] "GET /favicon.ico HTTP/1.0" 404 468 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:50.0) Gecko/20100101 Firefox/50.0"
www.funix.org:80 163.172.66.155 - - [01/Jan/2017:01:02:52 +0100] "GET /fr/linux/main-linux.php?ref=xinetd&page=full HTTP/1.0" 200 5751 "-" "Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)"
www.funix.org:80 86.252.235.82 - - [01/Jan/2017:01:04:29 +0100] "GET /fr/linux/main-linux.php?ref=video&page=menu HTTP/1.0" 200 14052 "https://www.google.fr/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/55.0.2883.87 Safari/537.36"
www.funix.org:80 86.252.235.82 - - [01/Jan/2017:01:04:29 +0100] “GET /images/funix-icone.gif HTTP/1.0" 200 4320 "http://www.funix.org/fr/linux/main-linux.php?ref=video&page=menu" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36"
www.funix.org:80 86.252.235.82 - - [01/Jan/2017:01:04:29 +0100] "GET /images/journal.gif HTTP/1.0" 200 354 "http://www.funix.org/fr/linux/main-linux.php?ref=video&page=menu" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTM
L, like Gecko) Chrome/55.0.2883.87 Safari/537.36"

by default there will always be only one unique visitor per day! So I had to adapt the log format with the LogFormat variable

LogFormat="%other %host - - %time1 %methodurl %code %bytesd %refererquot %uaquot"

This means that we drop the first field and only take into account the following ones.

The result files will be found under /usr/local/apache2/htdocs/awstats/funix.org, it seemed useful to me in this directory to create the following link

ln -s awstats.www.funix.org.html index.html

Now place this script under /etc/cron.daily so that it is called every day (at 4am for a default Mageia). So that all this is visible from my local Apache server, I added the following aliases in the file /usr/local/apache2/conf/httpd.conf

Alias ​​/awstats-funix-org "/usr/local/apache2/logs/funix.org/"

Alias ​​/goaccess-funix-org " /usr/local/apache2/logs/funix.org/funixlog.html"

and for the above directories to be accessible

<Directory "/usr/local/apache2/logs/funix.org/">
   Options Indexes FollowSymLinks Includes ExecCGI
   AllowOverride All
   Require all granted
</Directory>

<Directory "/usr/local/apache2/htdocs/awstats/funix.org/">

   Options Indexes FollowSymLinks Includes ExecCGI
   AllowOverride All
   Require all granted
</Directory>


we restart Apache in the process

systemctl restart httpd.service

with the CGI method my stats are accessible from the URL http://myserver/awstats/awstats.pl?config=www.funix.org
 
[ Back to FUNIX home page ] [ back to top of page ]