Ham/Spam learning accounts not working

[jgiles@…]

I have enabled the learning Ham account and learning spam account and saved the changes, however, when I forward a mail note to the spam@… account, I get this error:

postfix/smtpd[2738]: NOQUEUE: reject: RCPT from unknown[192.168.0.165]: 550 5.1.1 <spam@domain.com>: Recipient address rejected: User unknown in virtual mailbox table; from=<username@domain.com> to=<spam@domain.com> proto=ESMTP helo=<machinename.localnet>

cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=10.04
DISTRIB_CODENAME=lucid
DISTRIB_DESCRIPTION="Ubuntu 10.04.3 LTS"

dpkg -l |grep ebox
ii  asterisk                             1:1.6.2.7-1+ebox2                        Open Source Private Branch Exchange (PBX)
ii  asterisk-config                      1:1.6.2.7-1+ebox2                        Configuration files for Asterisk
ii  asterisk-sounds-main                 1:1.6.2.7-1+ebox2                        Core Sound files for Asterisk (English)
ii  ebox                                 2.0.23                                   Zentyal - Core
ii  ebox-antivirus                       2.0.5                                    Zentyal - Antivirus
ii  ebox-asterisk                        2.0.1                                    Zentyal - VoIP
ii  ebox-ca                              2.0.5                                    Zentyal - Certification Authority
ii  ebox-communication                   2.0-1                                    Zentyal - Communications Suite
ii  ebox-dhcp                            2.0.6                                    Zentyal - DHCP Service
ii  ebox-dns                             2.0.5                                    Zentyal - DNS Service
ii  ebox-ebackup                         2.0.13                                   Zentyal - Backup
ii  ebox-firewall                        2.0.3                                    Zentyal - Firewall
ii  ebox-ftp                             2.0.1                                    Zentyal - FTP
ii  ebox-gateway                         2.0-1                                    Zentyal - Gateway Suite
ii  ebox-ids                             2.0.1                                    Zentyal - Intrusion Detection System
ii  ebox-infrastructure                  2.0-1                                    Zentyal - Network Infrastructure Suite
ii  ebox-jabber                          2.0                                      Zentyal - Jabber (Instant Messaging)
ii  ebox-l7-protocols                    2.0                                      Zentyal - Layer-7 Filter
ii  ebox-mail                            2.0.6                                    Zentyal - Mail Service
ii  ebox-mailfilter                      2.0.2                                    Zentyal - Mail Filter
ii  ebox-monitor                         2.0.5                                    Zentyal - Monitor
ii  ebox-network                         2.0.10                                   Zentyal - Network Configuration
ii  ebox-ntp                             2.0.3                                    Zentyal - NTP Service
ii  ebox-objects                         2.0                                      Zentyal - Network Objects
ii  ebox-office                          2.0-1                                    Zentyal - Office Suite
ii  ebox-openvpn                         2.0.11                                   Zentyal - VPN Service
ii  ebox-printers                        2.0.3                                    Zentyal - Printer Sharing
ii  ebox-radius                          2.0                                      Zentyal - RADIUS
ii  ebox-remoteservices                  2.0.18                                   Zentyal - Zentyal Cloud Client
ii  ebox-samba                           2.0.12                                   Zentyal - File Sharing
ii  ebox-security                        2.0-1                                    Zentyal - UTM Suite
ii  ebox-services                        2.0                                      Zentyal - Network Services
ii  ebox-software                        2.0.12                                   Zentyal - Software Management
ii  ebox-squid                           2.0.8                                    Zentyal - HTTP Proxy (Cache and Content Filt
ii  ebox-trafficshaping                  2.0.3                                    Zentyal - Traffic Shaping
ii  ebox-usersandgroups                  2.0.12                                   Zentyal - Users and Groups
ii  ebox-webmail                         2.0.1                                    Zentyal - Web Mail Service
ii  ebox-webserver                       2.0.5                                    Zentyal - Web Server
ii  ebox-zarafa                          2.0.5                                    Zentyal - Groupware (Zarafa)
ii  john                                 1.7.6-1~ebox0~lucid0                     active password cracking tool
ii  john-data                            1.7.6-1~ebox0~lucid0                     active password cracking tool - character se
ii  libebox                              2.0.13                                   Zentyal - Common Library
ii  p3scan                               2:2.3.2-7ubuntu1+ebox1                   transparent POP3-proxy with virus- and spam-
ii  roundcube                            0.3.1-3ubuntu1~ebox2~lucid1              skinnable AJAX based webmail solution for IM
ii  roundcube-core                       0.3.1-3ubuntu1~ebox2~lucid1              skinnable AJAX based webmail solution for IM
ii  roundcube-pgsql                      0.3.1-3ubuntu1~ebox2~lucid1              metapackage providing PostgreSQL dependencie
ii  roundcube-sqlite                     0.3.1-3ubuntu1~ebox2~lucid1              metapackage providing sqlite dependencies fo
ii  samba-vscan                          0.3.6cbeta5ebox4-2                       Samba virus scanning VFS module

Please let me know what additional information you require.

Thanks!

[jamor@…]

Hello JGiles and thanks for your report.

I had set up a Zentyal with spam accounts in two domains and both of them receive mail correctly.

In your case the postfix message says that the account does not exists and I cannot figure the cause of that.

Can you make mail accounts in the same domain for regular users and they could receive mail correctly?.

What is the output of the comamnd 'slapcat | grep spam' ?

Cheers,

Javier

[jgiles@…]

Replying to jamor@…:

Hello JGiles and thanks for your report.

I had set up a Zentyal with spam accounts in two domains and both of them receive mail correctly.

In your case the postfix message says that the account does not exists and I cannot figure the cause of that.

Can you make mail accounts in the same domain for regular users and they could receive mail correctly?.

What is the output of the comamnd 'slapcat | grep spam' ?

Cheers,

Javier

Hi Javier,

slapcat | grep spam did not return anything.

I created a spam and ham account.

Here is the out put from slapcat for the ham and spam account:

slapcat | grep spam
memberUid: spam
description: HAM spam account
dn: uid=spam,ou=Users,dc=wolfserver2,dc=domain,dc=com
cn: spam account
uid: spam
homeDirectory: /home/spam
givenName: spam
mail: spam@joeman1.com
mailbox: joeman1.com/spam/
sambaHomePath: \\wolfserver\homes\spam

slapcat | grep ham
memberUid: ham
dn: uid=ham,ou=Users,dc=wolfserver2,dc=domain,dc=com
cn: ham account
uid: ham
homeDirectory: /home/ham
givenName: ham
mail: ham@joeman1.com
mailbox: joeman1.com/ham/
sambaHomePath: \\wolfserver\homes\ham

Is that the way its supposed to work?

Just out of curiosity; is there a way to change the names of those accounts as anyone on the internet can send a mail to ham@… and make their spam messages train on a mail server?

Thanks! Joe

[jgiles@…]

Just to let you know, I had a spam message in my inbox that I wanted to pass as ham, so I forwarded the e-mail to my ham@… account and it sent this time.

Couple of questions:

1. Do I have to manage these accounts going forward. For example, do I have to clean out their inbox from time to time, or does the spam filter take care of that for me?

2. Is there a way to change the account names to something more personalized so my ham account doesn't get spam and the filter things its ham and passes it? I imagine that every spam sender in the world knows of the ham/spam account and exploits it.

3. How often does the spam filter parse these accounts so I know when to clean them out in the event that the filter doesn't already do this?

Thanks again! Joe

[jgiles@…]

Hi Javier,

Just wanted to let you know that I just set up Zentyal 2.2 in a VM and enabled the ham and spam accounts and slapcat |grep spam returned nothing.

Is there some other setting I might be forgetting?

Thanks! Joe

[jamor@…]

(In [23435]) Fixed typo in code which prevented the creation of spam and ham users closes #3331

[jamor@…]

Hello Joe,

I have found the error, however it was in the first configuration of the module, so to fix your installation better run this command:

/usr/share/zentyal-mailfilter/mailfilter-ldap update

Then you surely we need to enable/disable the spam accounts in the mail domains and save changes again to put them in place.

[jamor@…]

Now I will answer your questions about this feature:

1 and 3- No need to clear anything, mail to these account is feed to the learning engine and it is not stored

2- This is a problem of this feature. However this mainly applies to the ham account, spammers throwing spam to the spam account could be a good thing. Of course a malicious individual could throw ham to the spam account but a spammer will not sweat so much for a single system.

You could use a private email domain (the spam/hma database is the same for all domains) to minimize this problem. Maybe we should remove this feature in the future. Remember you can also feed directly the learning engine through the Zentyal database or use directly the command line.

Regards,

Javier

[jgiles@…]

Hi Javier,

Thanks so much for looking into this!

I think its a great feature, but might need to be honed up some is all for security. I will take a closer look at how it work and see if I can come up with a contribution of some king.

In the past, I have use Dovevot/Postfix? but used user imap and not virtual domains and that was easy to set up using a simple script and user cron :).

Anyway, thanks again for helping with this!

Joe

[snoopy_22@…]

Hi all,

thanks to this ticket I managed creating the spam/ham accounts with the mailfilter-ldap command...'slapcat | grep spam' gives some output. I disabled/enabled the spam/ham accounts in the virtual domain and saved the configuration, too

BUT

trying to forward to "spam@domain" doesn't still work, now the mail server answers:

postfix/lmtp[19458]: 985B2CA03DD: to=<spam@domain>, relay=127.0.0.1[127.0.0.1]:2003, delay=0.17, delays=0.07/0.01/0.01/0.07, dsn=5.1.1, status=bounced (host 127.0.0.1[127.0.0.1] said: 503 5.1.1 User does not exist (in reply to RCPT TO command))

I think it has something to do when Zarafa as Groupware is enabled (as it is in my case)... I think, in this case the Mail accounts in Zarafa are not created...

slapcat sais:

dn: uid=spam,ou=Users,dc=domain
cn: Spam spam
uid: spam
sn: spam
loginShell: /usr/sbin/nologin
uidNumber: 1901
gidNumber: 1901
homeDirectory: /home/spam
userPassword:
quota: 0
objectClass: inetOrgPerson
objectClass: posixAccount
objectClass: passwordHolder
objectClass: systemQuotas
objectClass: CourierMailAccount
objectClass: usereboxmail
objectClass: fetchmailUser
structuralObjectClass: inetOrgPerson
entryUUID: 721aea00-a279-1030-85a1-b769c75b602a
creatorsName: cn=ebox,dc=domain
createTimestamp: 20111113192844Z
givenName: Spam
mail: spam@domain
mailbox: domain/spam/
userMaildirSize: 0
mailquota: 0
mailHomeDirectory: /var/vmail/
entryCSN: 20111113193421.016951Z#000000#000#000000
modifiersName: cn=ebox,dc=domain
modifyTimestamp: 20111113193421Z

Is the entry "mailHomeDirectory: /var/vmail" correct, despite Zarafa is installed? And what about the "objectClass: CourierMailAccount?" ?

Thanks in advance and greetings

Markus

[jamor@…]

Hello Markus,

I have tested the spam account and it received correctly the mail. I had tested both with the mail domain selected as Zarafa domain and with unselected. Also I tested to remove the spam option, save changes, re-enable it and save again. In all cases it worked.

Maybe you remember which steps you took to arrive to this situation. Other accounts from the domain receive their mail correctly?.

As for the /var/vmail directory as home; is correct because spam/ham account don't store anything, they just forward it to the training scriot.

[snoopy_22@…]

Hi javier,

ok, I try to remember:

1.) Install zentyal from burned iso 2.) Installed Zentyal Components like DNS,DHCP, Users and Groups, Groupware, Mail a.s.o. 3.) Configured Modules to my needs, everything works fine 4.) Activated the spam/ham learning accounts in Zentyal 5.) Sent a spam mail to ham@domain, getted the first error in this ticket 6.) Used slapcat to verify that ham/spam are were not created in openLDAP Directory 7.) Generated them with the commandline above 8.) deactivated / activated Ham/Spam? accounts with saves in between and after 9.) Sent spam mail to ham@domain 10.) Getted the error message I sent

I think that have to be the steps I took..

Mail to other accounts in the domain works fine (but these are Groupware / Zarafa accounts...) How does Postfix know that a receipient

user@domain

is a zarafa one

and

ham@domain

is a /var/vmail - directory one?

Can I assist u with sending some sort of config output?

[snoopy_22@…]

Hi again,

I did a little more of research and found out this:

I used local 'mail' command on the server to send testmails. The first mail to a normal user / mail receipient in Zarafa works fine as expected.

After that I sent a mail to "ham@domain" and after that analysed /var/log/syslog, here is the excerpt:

Nov 17 14:36:20 server postfix/pickup[8145]: 47586CA04C4: uid=0 from=<root>
Nov 17 14:36:20 server postfix/cleanup[16802]: 47586CA04C4: message-id=<20111117133620.47586CA04C4@server.domain>
Nov 17 14:36:20 server postfix/qmgr[12498]: 47586CA04C4: from=<root@server.domain>, size=350, nrcpt=1 (queue active)
Nov 17 14:36:20 server amavis[11146]: (11146-06) ESMTP::10024 /var/lib/amavis/amavis-20111117T102641-11146: <root@server.domain> -> <ham@domain> SIZE=350 Received: from server.domain ([127.0.0.1]) by localhost (server.domain [127.0.0.1]) (amavisd-new, port 10024) with ESMTP for <ham@domain>; Thu, 17 Nov 2011 14:36:20 +0100 (CET)
Nov 17 14:36:20 server amavis[11146]: (11146-06) Checking: 6C2vhBd8WH47 <root@server.domain> -> <ham@domain>
Nov 17 14:36:20 server postfix/smtpd[16808]: connect from localhost[127.0.0.1]
Nov 17 14:36:20 server postfix/smtpd[16808]: B0131CA054C: client=localhost[127.0.0.1]
Nov 17 14:36:20 server postfix/cleanup[16802]: B0131CA054C: message-id=<20111117133620.47586CA04C4@server.domain>
Nov 17 14:36:20 server postfix/smtpd[16808]: disconnect from localhost[127.0.0.1]
Nov 17 14:36:20 server postfix/qmgr[12498]: B0131CA054C: from=<root@server.domain>, size=957, nrcpt=1 (queue active)
Nov 17 14:36:20 server amavis[11146]: (11146-06) FWD via SMTP: <root@server.domain> -> <ham@domain>,BODY=7BIT 250 2.0.0 Ok, id=11146-06, from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: queued as B0131CA054C
Nov 17 14:36:20 server amavis[11146]: (11146-06) Passed, <root@server.domain> -> <ham@domain>, quarantine 6C2vhBd8WH47, Message-ID: <20111117133620.47586CA04C4@server.domain>,
Nov 17 14:36:20 server amavis[11146]: (11146-06) Hits: -0.001
Nov 17 14:36:20 server amavis[11146]: (11146-06) Passed CLEAN, <root@server.domain> -> <ham@domain>, Hits: -0.001, tag=0, tag2=5, kill=5, queued_as: B0131CA054C, L/Y/0/0
Nov 17 14:36:20 server postfix/smtp[16804]: 47586CA04C4: to=<ham@domain>, relay=127.0.0.1[127.0.0.1]:10024, delay=0.62, delays=0.17/0/0/0.45, dsn=2.0.0, status=sent (250 2.0.0 Ok, id=11146-06, from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: queued as B0131CA054C)
Nov 17 14:36:20 server postfix/qmgr[12498]: 47586CA04C4: removed
Nov 17 14:36:20 server postfix/lmtp[16809]: B0131CA054C: to=<ham@domain>, relay=127.0.0.1[127.0.0.1]:2003, delay=0.22, delays=0.1/0/0.01/0.11, dsn=5.1.1, status=bounced (host 127.0.0.1[127.0.0.1] said: 503 5.1.1 User does not exist (in reply to RCPT TO command))

As one can see, the mail went normally through virus scanner and spamassassin but in the last step, postfix wants to sent it to port 2003, which is zarafa-dagent:

root@server:~# netstat -anp|grep 2003
tcp        0      0 127.0.0.1:2003          0.0.0.0:*               LISTEN      19097/zarafa-dagent

Is that correct?

The mail should hit /var/vmail/domain/ham.... as u mentioned, right? But Zarafa uses his own mail store/mail users as far as I understood....

[jamor@…]

Sorry for the delay, Markus.

No, the mail should not hit either /var/vmail/domain/ham or /var/vmaild/domain/spam, the mail is piped to the filter training program and not stored in any mailbox.

I think the crux of the matter is you continue to do not have the account, see the '503 5.1.1 User does not exist' in your last line.

Could your run a 'slapcat' program and past the part where the spam and ham users appear?

[snoopy_22@…]

Hi Javier,

here is the ouput of slapcat regarding Spam / Ham users (I already sent the Spam - User part in my re-opening message above ;) )

dn: uid=spam,ou=Users,dc=domain
cn: Spam spam
uid: spam
sn: spam
loginShell: /usr/sbin/nologin
uidNumber: 1901
gidNumber: 1901
homeDirectory: /home/spam
userPassword:
quota: 0
objectClass: inetOrgPerson
objectClass: posixAccount
objectClass: passwordHolder
objectClass: systemQuotas
objectClass: CourierMailAccount
objectClass: usereboxmail
objectClass: fetchmailUser
structuralObjectClass: inetOrgPerson
entryUUID: 721aea00-a279-1030-85a1-b769c75b602a
creatorsName: cn=ebox,dc=domain
createTimestamp: 20111113192844Z
givenName: Spam
mail: spam@domain
mailbox: domain/spam/
userMaildirSize: 0
mailquota: 0
mailHomeDirectory: /var/vmail/
entryCSN: 20111113193421.016951Z#000000#000#000000
modifiersName: cn=ebox,dc=domain
modifyTimestamp: 20111113193421Z

dn: uid=ham,ou=Users,dc=domain
cn: Ham ham
uid: ham
sn: ham
loginShell: /usr/sbin/nologin
uidNumber: 1901
gidNumber: 1901
homeDirectory: /home/ham
userPassword:
quota: 0
objectClass: inetOrgPerson
objectClass: posixAccount
objectClass: passwordHolder
objectClass: systemQuotas
objectClass: CourierMailAccount
objectClass: usereboxmail
objectClass: fetchmailUser
structuralObjectClass: inetOrgPerson
entryUUID: 7228ac26-a279-1030-85a2-b769c75b602a
creatorsName: cn=ebox,dc=domain
createTimestamp: 20111113192844Z
givenName: Ham
mail: ham@domain
mailbox: domain/ham/
userMaildirSize: 0
mailquota: 0
mailHomeDirectory: /var/vmail/
entryCSN: 20111113193420.912392Z#000000#000#000000
modifiersName: cn=ebox,dc=domain
modifyTimestamp: 20111113193420Z

getent also sees the users:

root@server:~# getent passwd
...
spam:*:1901:1901:Spam spam:/home/spam:/usr/sbin/nologin
ham:*:1901:1901:Ham ham:/home/ham:/usr/sbin/nologin
...

[jamor@…]

(In [23760]) mailfilter autolearn accoutns are excluded from Zarafa transport closes #3331

[jamor@…]

Hello Markus,

I was able finally to reproduce and fix it.

You could apply the diff to hotfix your server but because it spans to multiple files maybe it would be easier to create a temporal domain from the ham/spam accounts.

Ah.. and the spam and the ham accounts have mail directories, their mail is periodically picekd up, feed to the learn engine and then deleted.

Regards,

Javier

[anonymous]

thanks