SpamAssassins Setup

General Interworx Control Panel Info General Inquries
1

Moved this over from the 1.9 thread:

I’ve been playing around with the setup and think I figured out some things (please correct me if I am wrong on any points)

NodeWorx Main Email Setup
(/nodeworx/mailsetup.php)

If you turn on “Enable SpamAssassin” this does two things:

  1. You are enabling the SMTP level scanning. The SMTP level scanning drops messages based on the “SMTP Spam Score” below. This scans the mail before local delivery and if it is higher than the “SMTP Spam Score” you set it will drop the email (never reaching the inbox of the user[s]). So I would set this to at least 15 or above so your clients don’t complain about missing emails.

  2. You turned on all SiteWorx account to scan on local delivery based on the spam preferences (see below for more on spam preferences) even if “SpamAssassin Status” is off in the SiteWorx Account.

NodeWorx Spam Preferences
(/nodeworx/mailsetup.php?mode=spamprefs)
This is the default global settings for local delivery (basically this overrides the SpamAssassins default settings for all SiteWorx Accounts).

*NOTE: These settings are active even if you have “Enable SpamAssassin” turned off in the NodeWorx Main Email Setup

SiteWorx Spam Preferences
(/siteworx/email.php?mode=spamprefs)

These settings override any default SpamAssassins or NodeWorx preferences. For example if you set “required_score 10” in the NodeWorx Spam Preferences and the SiteWorx account sets “required_score 5” the 5 will take precedence for that domain.

Sample Setup

Force all SiteWorx accounts to scan and make the default local spam level set to 10 without dropping any messages at the SMTP level.

  1. Set the NodeWorx Main Email Setup ON to FORCE (even if turned off at the SiteWorx level) all the SiteWorx account to scan for Spam.

[I]*Note: SiteWorx Spam Preferecnes are still active and override the Global preference even if SpamAssassin status is off in SiteWorx

FOR THE IWORX TEAM
Could you just have the SiteWorx SpamAssassins status to ON and have it greyed out (radio buttons disabled) if it is on by default in Nodeworx[/I]

  1. We don’t want to drop any messages at the SMTP level so we will set the SMTP Spam Score very high (ie. 999999).

FOR THE IWORX TEAM
Even tough it won’t drop at STMP if set to 99999, it will still take CPU time to scan, maybe you can have a way in NodeWorx to force all SiteWorx account to turn on without scanning SMTP.

  1. Then we go to NodeWorx Spam Preferences and add an entry for " required_score 10"

And were done.

Questions, comments, correction please!!!

OTHER STUFF:

Just noticed a White/Black list setting in horde, what is this tied to?

Question that I am not sure about
Does the SMTP scan at the NodeWorx level just use some default SpamAssassins settings or does it also take into account the NodeWorx Spam Preferences (SiteWorx Spam Preferences can’t be use as Paul explained above: “The bad thing about SpamAssassin scanning at the SMTP level is that if the e-mail that comes in has multiple recipients, there is no way to determine which recipient’s Spam Preferences should be used when scanning.”?

So if you have an email black listed on the NodeWorx Spam Preferences would this drop the email so it doesn’t even reach the inbox?

On this same note one bad thing is if you have a white list in SiteWorx Spam Preferences becasue this would be ignored at the SMTP level.

2

If you turn on “Enable SpamAssassin” this does two things:
…
2) You turned on all SiteWorx account to scan on local delivery based on the spam preferences (see below for more on spam preferences) even if “SpamAssassin Status” is off in the SiteWorx Account.

Not quite. The message won’t be scanned again during local delivery, UNLESS it is also turned on at the SiteWorx level.

FOR THE IWORX TEAM
Could you just have the SiteWorx SpamAssassins status to ON and have it greyed out (radio buttons disabled) if it is on by default in Nodeworx

As mentioned above these are actually two different settings, so greying it out doesn’t really apply.

FOR THE IWORX TEAM
Even tough it won’t drop at SMTP if set to 99999, it will still take CPU time to scan, maybe you can have a way in NodeWorx to force all SiteWorx account to turn on without scanning SMTP.

Indeed, SpamAssassin scanning at the SMTP level can definately lead to higher CPU usage if there was a lot of e-mail going to and from the server, and I wouldn’t recommend it in that case unless you have CPU cycles to spare.

Does the SMTP scan at the NodeWorx level just use some default SpamAssassins settings or does it also take into account the NodeWorx Spam Preferences (SiteWorx Spam Preferences can’t be use as Paul explained above:

It does take the NodeWorx Spam Preferences into account, AND, SiteWorx Spam Preferences will also be used, UNLESS there is more than one recipient of the e-mail.

So if you have an email black listed on the NodeWorx Spam Preferences would this drop the email so it doesn’t even reach the inbox?

I believe the blacklist increases the spam score by 100 points (this is from memory only), so if the SMTP Spam Score was set to 95, it should reject it.

On this same note one bad thing is if you have a white list in SiteWorx Spam Preferences becasue this would be ignored at the SMTP level.

Yes, that’s true, if there is more than one recipient for the e-mail.

Just noticed a White/Black list setting in horde, what is this tied to?

It’s currently not related to SpamAssassin at all. BUT: As of version 1.9.2 (not yet released) the white and blacklist in Horde Webmail WILL be tied to the SpamAssassin white/blacklist, and it will be a preference for just that e-mail user (not domain wide). But that’s another story for another post for another (not very far off) day.

I think I got all the questions - let me know if I missed anything.

Paul

3

I’m am a little confused by this. I did serveral test and it appears to be using the rules of siteworx to scan even if Spam is OFF in SiteWorx.

So maybe that means the NodeWorx scan was taking the SiteWorx preferences (even when Spam is off) because it was to one receipient.

If you set the SiteWorx email preferences to a spam level of 5 and lets say it is 10 in NodeWorx. When I get the test email the spam score setting in the email headers is 5 even with the SiteWorx spam setting turned off.

The difference I do see on the server is the default qmail changes to:
|/usr/bin/spamc -u $EXT@$HOST | /home/vpopmail/bin/vdelivermail ‘’ delete and another file (empty) is created called spamon

Some more questions:

-When we say “SMTP Level” is that referring to just a server wide scan (like at the NodeWorx level) or is there a SMTP scan, a NodeWorx scan, and then also a SiteWorx scan?

  • If an email is scanned at the SMTP level it is dropped an never delivered and is based on default, NodeWorx, and SiteWorx settings (does SiteWorx Spam have to be on for this to work?)

  • How does the SiteWorx scan happen if it is turned off and the .qmail-default is the normal | /home/vpopmail/bin/vdelivermail ‘’ bounce-no-mailbox

  • Is there a way to drop mail (like at the NodeWorx level) using some kind of Spam Preference.

4

That’s correct.

If you set the SiteWorx email preferences to a spam level of 5 and lets say it is 10 in NodeWorx. When I get the test email the spam score setting in the email headers is 5 even with the SiteWorx spam setting turned off.

Correct, if there is one recipient, it will still use any defined preferences from SiteWorx when scanning only at the SMTP level (and when it’s “off” in SiteWorx.

Some more questions:

-When we say “SMTP Level” is that referring to just a server wide scan (like at the NodeWorx level) or is there a SMTP scan, a NodeWorx scan, and then also a SiteWorx scan?

Enabling SpamAssassin in NodeWorx means enabling it at the SMTP level.
Enabling SpamAssassin in SiteWorx means at the MDA (mail delivery agent) level, AKA during local delivery, when we know there will be only one recipient.

  • If an email is scanned at the SMTP level it is dropped an never delivered and is based on default, NodeWorx, and SiteWorx settings (does SiteWorx Spam have to be on for this to work?)

If it is scanned, and the Spam Score is HIGHER than the SMTP Spam Score setting in NodeWorx, the message will be dropped, and not queued for local delivery - and it will use the NodeWorx Preferences every time, and it will use the NodeWorx Prefs + SiteWorx Prefs if it is able to (if there’s one recipient). SpamAssassin does not need to be “on” in SiteWorx for this to happen.

  • How does the SiteWorx scan happen if it is turned off and the .qmail-default is the normal | /home/vpopmail/bin/vdelivermail ‘’ bounce-no-mailbox

The “SiteWorx Scan” doesn’t happen if it’s “off” in SiteWorx, which means messages aren’t scanned during local delivery - however, the Preferenecs defined in SiteWorx WILL still be used, if possible, during the SMTP level scan (assuming it’s enabled).

  • Is there a way to drop mail (like at the NodeWorx level) using some kind of Spam Preference.

No, but with 1.9.2 (coming soon) if there is a “Spam” IMAP folder, it will deliver spam-tagged messages to that folder rather than the user’s INBOX.

Thanks for all the great questions!
Paul

5

Not sure if this is linked to SpamAssassin but would like someone to confirm this is correct.

I am watching the var/log/maillog and seeing this alot.

Mar 16 19:57:38 delly spamd[10664]: connection from delly.**Server**.net [127.0.0.1] at port 38726
Mar 16 19:57:38 delly spamd[10664]: handle_user: unable to find user 'someone@domain.co.uk'!
Mar 16 19:57:38 delly spamd[10664]: Still running as root: user not specified with -u, not found, or set to root.  Fall back to nobody.
Mar 16 19:57:38 delly spamd[10664]: processing message <000601c52a62$750d61a0$2101a8c0@a**in> for someone@domain.co.uk:99.
Mar 16 19:57:39 delly spamd[10664]: clean message (0.0/5.0) for someone@domain.co.uk:99 in 0.2 seconds, 2012 bytes.
Mar 16 19:57:39 delly spamd[10664]: result: .  0 -  scantime=0.2,size=2012,mid=<000601c52a62$750d61a0$2101a8c0@a**in>,autolearn=failed

Thanks in advance.

Perry

6

Hi Perry,

I don’t see anything out of the ordinary here.

Paul

7

A question about the SMTP level scan and the Vpopmail scan.

I found this flowchart and doesn’t seem to be the same as what I know how the InterWorx one is setup.

http://www.tnpi.biz/internet/mail/toaster/filter.shtml

Towards the top where it says “is message clean?”. It says “no” and then goes to reject spam. I assume that this is the server wide setting in NodeWorx (SMTP Spam Score) for the spam. So if the email is above that SMTP Spam Score (or if there is a virus detected), the email is dropped (“deleted”) before every being sent to the Vpopmailbox

That seems to be in line with what I know of InterWorx. The part that doesn’t seem to be the same is towards the bottom.

It says “Is spam filtering enabled”, now I am guessing this is at the Vpop mail (aka siteworx scanning).

This is the part that is different because…

  1. In this chart it is not scanned twice if spam headers are already present.
  2. There seems to be a way to drop mail at the Vpopmail level. From what I understand with the InterWorx setup if you want to completely drop mail (not just ‘tag’ it) you can only do that if you enable SMTP (NodeWorx) scanning.

I am currently using NodeWorx scanning so I can have the ability to drop spam, I know everyone says to just send it to a junk folder, but for the higher marked spam I would rather just not see it at all.

The downside of this setup is that I disabled SiteWorx scanning on all the accounts I manage so that I dont get the 2x scan, butfor clients that manage their own account they may turn on SiteWorx Spamassassins (which would put an extra uneeded load on my server).

So even if #1 above isn’t possilbe and it will be scanned 2x if it’s on in SiteWorx, is it possible to do #2 and drop mail at the Vpopmail (SiteWorx) level?

Thanks for the info! :smiley:

[EDIT]
Something else I noticed is that even on a non-enabled siteworx account, the .qmail-default devliever to the maildrop for spam scanning.

|/usr/bin/maildrop /home/interworx/lib/maildrop/spamfilter

[EDIT - 21:53]
Doing some more digging I see that maildrop runs /home/interworx/lib/maildrop.

This is the “script” that does the ‘spamc’ scan at the SiteWorx level. So Im thinking if I disable the NodeWorx scan, I can edit the Maildrop script to check the spam score after it is run on SiteWorx level and if it is above a certain score it can be delivered to

"|/home/vpopmail/bin/vdelivermail /dev/null

or maybe you can just put the /dev/null without the vdelivermail?

I know messing with this file could screw up mailing being delivered and I haven’t actually done anything yet. But I want to know if Im on the right track and if maybe in the future we can control more details of the maildrop through NodeWorx instead of manually editing it?

One other thing I noticed (maybe :rolleyes: ). If you turn on Spam for SiteWorx and spam is detected it will be delivered to a Spam folder. But if a client is only using POP3 instead of webmail or IMAP they would never see this email.

8

I’m pretty sure Maildrop can solve both (#1 and #2) problems above. Not exactly sure how, but you could probably have maildrop check the headers of the mail to see if its already been scanned so it wont be scanned twice.

Also, for #2 you could check the spam score after the SiteWorx scan (in maildrop) and then deliver to /dev/null. Below is the code, but not exactly sure how to do it yet.


      if ( $SIZE < $IW_MAXSIZE )
      {
         exception {
            xfilter "/usr/bin/spamc -u $EXT@$HOST"
         }
      }

      if (/^X-Spam-Level: **********/)  
      {
         #drop message
      }

I dont know how I can make it check the actual score number out of something like "X-Spam-Status: Yes, score=15.3 required=10.0 " and even what I have above would drop mail that has a score of 10.0 - 10.9. Not sure how I could say 10 or above.

[EDIT]
Maybe something like this???


      # Drop if spam score is not 0-10.9
      if (! /^X-Spam-Status:.*score=\/[0-10\.\-]+/)  
      {
         #drop message
      }
9

You’re definately on the right track. You’re correct that that flow chart is somewhat similar to the current InterWorx setup, but it is not identical as you noticed.

Yes, that’s true. Unfortunately if it isn’t scanned a second time (assuming it was scanned once at the SMTP/NodeWorx level), they can’t guarantee that the user’s spamassassin preferences will be used, since at the SMTP level there can be more than one recipient. That’s the downside of not spamassassin scanning at the SiteWorx / maildrop level.

  1. There seems to be a way to drop mail at the Vpopmail level. From what I understand with the InterWorx setup if you want to completely drop mail (not just ‘tag’ it) you can only do that if you enable SMTP (NodeWorx) scanning.

Yes, that’s correct, with the current default InterWorx setup - version 1.9.2 - the only way to reject spam is at the NodeWorx/SMTP level.

I’m pretty sure Maildrop can solve both (#1 and #2) problems above.

Yes, it can.

      # Drop if spam score is not 0-10.9
      if (! /^X-Spam-Status:.*score=\/[0-10\.\-]+/)  
      {
         #drop message
      }

Something like this would get the job done - I assume you found this maildrop code somewhere online? I’d need to be convinced that regex actually does what the comment says - I’m not sure that [0-10.-]+ will really match 0-10.9 - but if it does, that would work. In place of the #drop message you could just put “exit” and it would stop all maildrop processing at that point, and in effect the mail would be “dropped” at that point.

I know messing with this file could screw up mailing being delivered and I haven’t actually done anything yet. But I want to know if Im on the right track and if maybe in the future we can control more details of the maildrop through NodeWorx instead of manually editing it?

Yes, it definately could mess up mail being delivered, so I’m glad you realize that :slight_smile: and yes, you’re on the right track, and there will be more powerful “siteworx level” options in the default maildrop script in future releases.

One other thing I noticed (maybe :rolleyes: ). If you turn on Spam for SiteWorx and spam is detected it will be delivered to a Spam folder. But if a client is only using POP3 instead of webmail or IMAP they would never see this email.

That’s right, pop3 users would have to check their webmail occasionally to see the spam in the Spam folder. Alternatively they could delete (or not create) the Spam imap folder, in which case the tagged spam will go to their inbox instead. Unfortunately there is no way to deliver a message to a pop3 folder, other than the inbox, on the server side, since the pop3 folders are all client side.

Hopefully I’ve helped clarify some things for you. Let me know if there are more questions, or any questions I overlooked in this post.

Paul

10

Paul, thanks for your always detailed answers!

That makes sense, but I still am not sure if that would cause problems because then you would have two different spam headers in the email which may confuse a client program trying to filter the email. Or it could confuse the maildrop check “if /X-Spam-Flag: YES/”. What would happen in the case where NodeWorx doesn’t detect spam, but the SiteWorx does? Does the SiteWorx scan remove the spam headers of the NodeWorx basically making it look like the NodeWorx never ran? Or something else…?

I found the [0-10.-]+ part of the code online. For some reason I still can’t figure out how the wildcard stuff works for these test with all the * .
I will either have to play around with it until I figure it out or find a webpage that explains it, but Im not even sure what I should google for that.

Maybe I will set it up in the maildrop file special test for a specific email address and try out my Mod’d spam code only for that one email to test it out. Not sure how to do this yet because the Maildrop script seems to have it’s own syntax that is not the same as a shell script.

So I dont know for a test if I would do:

if($EXT@$HOST==“myemail@domain.com”)
{
#Run MyTest
}

Yeah, this will definately be a weekend project for testing out the different stuff. And I will definately will do a:
cp /home/interworx/lib/maildrop/spamfilter /home/interworx/lib/maildrop/spamfilter_iworx_backup

Good to hear!!! You guys are always one step ahead of me.

How exactly does this work? If a client has NEVER logged into webmail and has only done POP3, would the Maildrop still automatically create a spam folder and drop spam in there?

I just went over the Maildrop code again and see that it does NOT create a spam folder, it just checks if it exist. So I guess this would not be a problem unless they went back and forth between IMAP/Webmail and POP3.

On a more just Linux note, how do you get the result of the test into $RETURNCODE automatically?

Yes, thank you Paul. The fee I pay for InterWorx is worth it just for the support and infomoration you guys provide. So I basically feel like I have a free control panel :wink:

And the only problem when you answer my questions is as soon as you answer them I have a few more waiting :smiley:

[EDIT - May 21 - 13:33]
Another good thing about doing SiteWorx only scanning for spam is you dont waste scanning. I was just tail -f’n my maillog and I saw a bunch of spam come into one of my sites. But the email address were not ones that are actually there, so they wouldn’t be delivered anyway, but they were still scanned for spam.

[EDIT - May 21 - 15:47]
Wanted to add yet another Question :smiley:

If SMTP (NodeWorx) scanning is turned off would alias emails still be scanned? I know the alias through SiteWorx are stored in the Vpopmail database, so I assume for alias it is still “delivered” from qmail-que to Maildrop (Spam scan takes place) which then calls vdelivermail which finds that it is sent to an alias rather than a local vpopmail Mailbox folder.

Is that right?

11

I figured out how to setup my testing environment which I will post just for fun (as it isn’t needed for anything besides testing)


#$EXT and $host are set by script

$IW_TEST_ACCOUNT="yourtestemail@domain.com"

`test "$EXT@$HOST" = "$IW_TEST_ACCOUNT"`
#If it's a match do test code
if( $RETURNCODE != 1 )
{
  #test code here
}

And the code to determine the Spam-Score is the following:


#IW_DROP_SCORE is set to 15, but maybe in a future Iworx this could be set 
#in Iworx database for each mail user to be able to specify at which score 
#they want to drop messages.
$IW_DROP_SCORE = 15

#Checks for Spam and stores the score in $MATCH2 variable
if ( /^X-Spam-Status: Yes, score=![0-9]+\.[0-9]+! /:h)
{
    #If above Drop Score silently drop the message
    if($MATCH2 > $IW_DROP_SCORE)
    {
        exit
    }
}
12

Is there a good way to do a SiteWorx (Maildrop) scan of aliased email?

I disabled my NodeWorx (SMTP) scan since I now can drop messages at the SiteWorx level. But now that I have NodeWorx off my alias accounts are not getting scanned because the IW_VHOME is not set because there is no local mailbox by that name.

I know I could just have spam scanning when IW_VHOME is not set, but I dont want to waste time scanning emails that are sent to a non-existance mailbox. I have one account on my server that gets about 10-20 emails every 5 mins to bad address.

Is there a way I could do a check if an alias account exist in addtional to the IW_VHOME existing?

[EDIT]


#original code
if( $IW_VHOME )



#Need to create this variable
IW_VALIAS=`/home/vpopmail/bin/valias -s $EXT@$HOST`

#Add IW_VALIAS to the if statment
if( $IW_VHOME || $IW_VALIAS )