OPTIMIZE Transfer.log : Cool Impacts on iworx daily cron !

Hello,

The iworx.pex --daily script is a cronjob which run every days.

Basicly it performs a lot of things and one of them is to treat the transfer.log files of every vhost. (it uses the logresolve.pl script)

The transfer.log format and contents are defined in your httpd.conf file and is set for every vhosts in the vhost config file

CustomLog /home/account/var/domain.com/logs/transfer.log combined

You might have HUGE transfer.log and in this case the iworx.pex --daily can take a long long time to run.

For example if you have a live support there will be in this log file every checks between your workstation and your support live application. More if you have a lot of visitors, and you have a live support script on all your web page you’ll have HUGE entries in your transfer.log (hundred of thousand lines) !

So it might be very interesting to filter the data that fill up the transfer.log

For example you might consider :

- Not take in account all entry with no referer
- Not take in account all entry that match your live support tool

Ok stop talking, I show you how do this in your vhost config file:

SetEnvIf Request_URI ^/support/winapp/ myfiler=the-agent
SetEnvIf Request_URI ^/support/visitor/ myfilter=the-visitor
SetEnvIf Referer ^$ myfilter=refisempty
CustomLog /home/account/var/domain.com/logs/transfer.log combined env=!myfilter

  • The 3 first lines defined an environment variable called myfilter and set to it different value (just to tell is set)
  • The 2 first lines seton the environement variable myfilter if the request URI matches these paths : /support/winapp/ and /support/visitor/
  • The third line seton the environment variable myfilter if the referer is not set !
  • Finally we have added env=!myfilter to the Customlog entry. This means run the customlog but not for all cases set by this environment variable !

This an example and you might find a lot of cases that could fit your needs !

So, to conclude :
1- If you have a “iworx --daily” cronjob that take a long long time
2- Find huge transfer.log sizes
3- Study them and defined what you want to track or not !
4- Defined some Environment variable to filter your transfer.log contents !

Hope you learned something and it might help you

Pascal

Excellent tip, Pascal. Thanks :slight_smile:

No pbm :wink:

Pascal