Index  Page 1  Page 2
Amavisd-new lookups
Install OpenProtect SARE rules - Optional
Install DCC (dccifd) - Optional
Install Postgrey - Optional
Install policyd-weight - Optional
Install SaneSecurity and MSRBL ClamAV signatures - Optional
Install Botnet SA plugin - Optional
Install p0f - Optional
Upgrade amavisd-new - Optional
Upgrade SpamAssassin - Optional
Install FuzzyOCR - Optional
Install AIDE - Optional
Install Altermime - Optional
Odds and ends

Amavisd-new lookups

Index Top Bottom
The amavisd-new README.lookups page may be somewhat difficult to understand. However, for our needs the two major concepts are reasonably simple. First let's talk about SQL and static lookups. A lookup is performed using an email address as the key (usually the envelope recipient). SQL lookups are performed before static hash lookups, hash lookups are performed before defaulting to constants. To be more precise, from the README: "default sequence of lookups: SQL, LDAP, hash, ACL, regexp, constant." (you can find examples of hash, ACL, regexp and constant settings in amavisd.conf-sample). For example: if a matching recipient email address (or part of an address, or catchall) exists in the SQL database (in the users table), and has a policy (in the policy table) with the field 'spam_tag2_level' set at 6.0 and there is also a static constant setting '$sa_tag2_level_deflt' (in the 50-user file) set to 6.31; then 6.0 is used (because SQL is checked first and first match wins). The search (for that particular piece of data) stops at the first definitive answer (not NULL or undef). If the SQL field is NULL however (which evaluates to undef), the search continues on. In this example there is no hash table, no ACL list and no regexp list, so the search ends at the constant (which does exist - and matches any recipient); the net result: 6.31 is used. That is the first important concept.

The second important concept is priority within the SQL and static hash lookups. Full email addresses are broken down into pieces and there is an attempt to match each piece with the data in the SQL, hash or other table/map. In static hash lookups the most specific comparisons are made first (full email address). The most general comparisons are made last (catchall "."). Hash lookups (e.g. for user+foo@example.com) are performed in the following order:
user+foo@example.com
user@example.com
user+foo@
user@
example.com
.example.com
.com
.
Note that the format is a little different for SQL:
user+foo@example.com
user@example.com
user+foo
user
@example.com
@.example.com
@.com
@.
Because of the order of searches and the fact that the search stops at the first match, in hash (and other) tables it only makes sense to place more specific patterns before the more general ones. For example, in this hash postmaster@example.net is placed before .example.net:
$sa_tag2_level_deflt = 6.31;

@spam_tag2_level_maps = (
  { 'postmaster@example.net' => 99.0,
    '.example.net' => 7.0 },
    \$sa_tag2_level_deflt,   # catchall default
);
In SQL however, the order in which the lookups are performed is determined by the SQL SELECT statement and the users.priority field. The supplied SQL SELECT statement orders by the priority field in DESCending order. This means you have to create the same 'most specific to most general' order by setting the proper priority. In other words, you could assign a priority of 7 to all users with a full email address in the email field (which we do), and you could add a domain wide user "@example.net" that would have a lower priority. If user@example.net had spam_kill_level set to 6.0 and @example.net had spam_kill_level set to 7.0, then 6.0 would be used because user@example.net has a higher priority. If user@example.net had spam_kill_level set to NULL, then the lookup would fall through to the @example.net user (and it would find 7.0). If @example.net had a higher priority than user@example.net, then 7.0 would be used. For a given recipient this is what the SQL SELECT statement that amavisd-new creates might look like:

SELECT *,users.id FROM users LEFT JOIN policy ON users.policy_id=policy.id
 WHERE users.email IN ("user@example.net", "user", "@example.net", "@.example.net", "@.com", "@.")
 ORDER BY users.priority DESC;


Here is an example from a level 5 log entry where user@example.net has a policy with spam_tag2_level NULL (which is translated to undef) but a static hash exists (@spam_tag2_level_maps as shown above). The lookup falls through to the static hash, where a match is found:
lookup_sql_field(spam_tag2_level) "user@example.net" result=undef
query_keys: user@example.net, user@, example.net, .example.net, .net, .
lookup_hash(user@example.net) matches key ".example.net", result=7
lookup (spam_tag2_level) => true,  "user@example.net" matches, result="7", matching_key=".example.net"
In this next example the same user still has spam_tag2_level set to NULL, but there is a lower priority (domain wide) user '@example.net' that has spam_tag2_level set to 9999:
lookup_sql_field(spam_tag2_level) "user@example.net" result=undef
lookup_sql_field(spam_tag2_level) "user@example.net" result=9999
lookup (spam_tag2_level) => true,  "user@example.net" matches, result="9999", matching_key="/cached/"
"/cached/" simply means it is using results found during the initial query. Here is an example where the user does not exist in SQL and is not found in the hash, therefore the lookup falls through to the scalar variable:
lookup_sql_field(spam_tag2_level), "user@example.org" no matching records
query_keys: user@example.org, user@, example.org, .example.org, .org, .
lookup_hash(user@example.org), no matches
lookup: (scalar) matches, result="6.31"
lookup (spam_tag2_level) => true,  "user@example.org" matches, result="6.31",
        matching_key="(constant:6.31)"
Now hopefully it will make more sense when you read README.lookups. When I initially created the policies, I left all the data fields NULL with the exception of spam_tag2_level, spam_kill_level and occasionally spam_quarantine_cutoff_level. This way all the remaining settings are controlled by entries in the 50-user file. When new policies are created in the SquirrelMail interface, only those two or three fields have non-NULL values.


Install OpenProtect SARE rules - Optional

Index Top Bottom
SARE (SpamAssassin Rules Emporium) offers additional SpamAssassin rule sets. OpenProtect gathers together the safest sets of SARE rules. During this process we will replace the current sa-update.sh script with a script that includes the OpenProtect SARE rules:
cd /etc/spamassassin
wget http://saupdates.openprotect.com/pub.gpg
sa-update --import pub.gpg
cd /usr/sbin
wget http://www200.pair.com/mecham/spam/sa-update.sh.txt
mv sa-update.sh.txt sa-update.sh
chmod +x sa-update.sh
sa-update.sh


Personally I vi /usr/sbin/sa-update.sh and have it delete a few things that it downloads I don't want. If you install a new version of SpamAssassin you would have to remember to update this:
[...]
if [[ $code2 > 1 ]]; then
  echo "problem with openprotect update"
fi

rm -f /var/lib/spamassassin/3.002003/saupdates_openprotect_com/70_sare_whitelist_spf.cf
rm -f /var/lib/spamassassin/3.002003/saupdates_openprotect_com/70_sare_stocks.cf
rm -f /var/lib/spamassassin/3.002003/saupdates_openprotect_com/loadplugins.pre
if [[ `expr $code1 + $code2` < 2 ]]; then
[...]



Install DCC (dccifd) - Optional

Index Top Bottom
DCC detects messages that are bulk mailed. A hit results in 1.4 or 2.17 SpamAssassin points. Personally, I set this to 1.75 points ('score DCC_CHECK 1.75' in local.cf). The way I understand the license for DCC is if you are not a reseller and you receive less than somewhere around 100,000 messages per day you can use the DCC client by itself. If you run a larger site then you need to run a DCC server and flood your data to public DCC servers. If you are a reseller you need to pay for a license. I only outline installing a client. Be patient when compiling the program - it may appear to hang at one point:
cd /usr/local/src
wget http://www.rhyolite.com/anti-spam/dcc/source/old/dcc-1.3.92.tar.Z
tar xzf dcc-1.3.92.tar.Z
cd dcc-1.3.92
./configure --with-uid=amavis && make && make install

ln -s /var/dcc/libexec/cron-dccd /usr/bin/cron-dccd
chown -R amavis:amavis /var/dcc
cdcc info


You should see entries with 'requests ok'. If not, then you might be having an issue with a proxy server or firewall. http://www.dcc-servers.net/dcc/firewall.html gives an example of a Cisco router access list entry. Keep in mind that you will likely not run a DCC server, only the DCC client. If the tests fail you won't be able to test again until some period of time has passed. The program does this to prevent broken clients from trying too often.
crontab -e

and insert:
43 11 * * * /usr/bin/cron-dccd

cd
echo "dcc_home /var/dcc" >> /etc/spamassassin/local.cf
sed -i 's/DCCIFD_ENABLE=off/DCCIFD_ENABLE=on/' /var/dcc/dcc_conf
sed -i 's/DBCLEAN_LOGDAYS=14/DBCLEAN_LOGDAYS=1/' /var/dcc/dcc_conf
sed -i 's/DCCIFD_LOGDIR/#DCCIFD_LOGDIR/' /var/dcc/dcc_conf
cp /var/dcc/libexec/rcDCC /etc/init.d/adcc
update-rc.d adcc defaults
/etc/init.d/adcc start
sed -i 's/#loadplugin Mail::SpamAssassin::Plugin::DCC/loadplugin Mail::SpamAssassin::Plugin::DCC/' /etc/spamassassin/v310.pre
amavisd-new reload


You will get errors because I chose to turn of logging. I wouldn't worry about them:
find: /etc/rc.d: No such file or directory
Jun 15 19:33:12 sfa dccifd[7459]: log thresholds set with -t but no -l directory
Jun 15 19:33:12 sfa dccifd[7459]: no -l directory prevents per-user logging with -U

Test with spamassassin:
su amavis -c 'spamassassin -D dcc <sample-spam.txt'

You should get something like this in the output:
[7615] dbg: dcc: dccifd is available: /var/dcc/dccifd
[7615] dbg: dcc: dccifd got response: X-DCC--Metrics: sfa 1049; Body=many Fuz1=many Fuz2=many


We need to suppress a few logcheck messages:
echo "stat\(log directory /var/dcc/log\): No such file or directory" >> /etc/logcheck/ignore.d.server/dcc
echo ": missing message body" >> /etc/logcheck/ignore.d.server/dcc
echo "detected 0 spam, ignored for 0, rejected for 0," >> /etc/logcheck/violations.ignore.d/dcc


Read the LICENSE:
cat /usr/local/src/dcc-1.3.92/LICENSE


Install Postgrey - Optional

Index Top Bottom
Of course Greylisting will reject a ton of spam - but at a cost. Some mail will be delayed. To help out I cut the retry time from 6 minutes to 29 seconds. We will use 'selective greylisting' with the idea being we will attempt to only target clients that appear to come from dial-up and dynamic IP addresses. If you decide to implement this I also recommend you whitelist clients you trust. The whitelists are located in the /etc/postgrey/ directory. We can also whitelist senders, clients or networks in Postfix.
apt-get install postgrey

sed -i 's/--inet=127.0.0.1:60000/--inet=127.0.0.1:60000 --delay=29/' /etc/default/postgrey
/etc/init.d/postgrey restart
ps aux | grep postgrey | grep -v grep

cd /etc/postfix
wget http://www200.pair.com/mecham/spam/check_client_fqdn
cp -ip main.cf main.cf-before-grey

postconf -e "smtpd_restriction_classes = check_greylist"
postconf -e "check_greylist = check_policy_service inet:127.0.0.1:60000"
cp access greylist_sender_exceptions
postmap greylist_sender_exceptions
touch cidr_greylist_network_exceptions


In main.cf, our smtpd_recipient_restrictions should already look pretty like the sample below. Read the beginning (half dozen lines or so) of http://www.postfix.org/postconf.5.html. Now we need to add greylisting and a sender whitelist for greylisting. Either vi /etc/postfix/main.cf or use the WinSCP editor and edit smtpd_recipient_restrictions, adding the items in red as usual:
smtpd_recipient_restrictions =
     permit_mynetworks,
     permit_sasl_authenticated,
     reject_unauth_destination,
     reject_unlisted_recipient,
     check_recipient_access hash:/etc/postfix/reject_over_quota,
     check_sender_access hash:/etc/postfix/rbl_sender_exceptions,
     check_client_access hash:/etc/postfix/rbl_client_exceptions,
     check_recipient_access hash:/etc/postfix/rbl_recipient_exceptions,
     reject_rbl_client sbl-xbl.spamhaus.org,
     check_recipient_access hash:/etc/postfix/greylist_sender_exceptions,
     check_client_access cidr:/etc/postfix/cidr_greylist_network_exceptions,
     check_client_access pcre:/etc/postfix/check_client_fqdn
Once the file is saved:
postfix reload
tail -f /var/log/mail.log


Set your MUA to deliver normally to port 25 (no authentication) and then attempt to send a message through and see if your client gets greylisted like mine did. If I wait 30 seconds I should be able to send the message. Note that my client IP is not included in mynetworks in main.cf (if it were, everything in smtpd_recipient_restrictions would be bypassed):
Jun 15 21:12:19 sfa postfix/smtpd[10095]: NOQUEUE: reject: RCPT from
 unknown[192.168.1.41]: 450 4.7.1 <unknown[192.168.1.41]>: Client host rejected:
  Greylisted, see http://isg.ee.ethz.ch/tools/postgrey/help/example.com.html;
   from=<garyv@example.org> to=<garyv@example.com> proto=ESMTP helo=<nobody>


I am going to whitelist 192.168.1.0 so clients on my internal network are not subjected to greylisting. Note that typically you would simply add your internal network to the mynetworks setting in main.cf. Cidr type tables do not need to be postmapped. http://www.postfix.org/cidr_table.5.html. I will place the whitelist entry in vi /etc/postfix/cidr_greylist_network_exceptions
192.168.0.0/16 OK

postfix reload

I am going to wipe out the entire Postgrey database in order to prove to myself that my change did in fact whitelist my network. You don't want to do this once the system is up and running (unless the database becomes corrupt). Then I am going to try to send another message.

ls -l /var/lib/postgrey/
/etc/init.d/postgrey stop
rm /var/lib/postgrey/*
/etc/init.d/postgrey start

ls -l /var/lib/postgrey/


tail -f /var/log/mail.log

And it worked:
Jun 15 21:36:51 sfa postfix/smtpd[10167]: connect from unknown[192.168.1.41]
Jun 15 21:36:52 sfa postfix/smtpd[10167]: 1EDD624218: client=unknown[192.168.1.41]


Note however that typically local networks are placed in $mynetworks, which means Postgrey would typically be bypassed for clients in $mynetworks.

Install policyd-weight - Optional

Index Top Bottom
Note that Policyd-weight is no longer maintained. As with any program that uses RBLs, certain RBLs may not work in the future. Depending on your volume, they may cost you money to subscribe to them (zen.spamhaus.org for example). So with this in mind, be aware that stuff that works today may not work in the future. Policyd-weight will block a client if it hits a few RBLs or appears to be poorly configured (or not configured) as a mail server. The $REJECTLEVEL setting is what will make the biggest difference as far as how much mail is rejected. I set it to a conservative 4.8 in /etc/policyd-weight.conf in order to give idiots running real (but poorly configured) mail servers a fighting chance. The RBL checks in policyd-weight may cause a slight delay when a client connects.
apt-get install policyd-weight

LINUX2
cd /usr/sbin
mv policyd-weight policyd-weight-etch
wget http://www.policyd-weight.org/policyd-weight
chmod +x policyd-weight

cd /etc
wget http://www200.pair.com/mecham/policyd-weight.conf
/etc/init.d/policyd-weight stop
/etc/init.d/policyd-weight start
cd /etc/postfix
cp main.cf main.cf-before-policyd-weight
postconf -e "smtpd_helo_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_invalid_hostname"
test -e greylist_sender_exceptions || cp -i access greylist_sender_exceptions
postmap greylist_sender_exceptions
touch cidr_greylist_network_exceptions


Now vi /etc/postfix/main.cf or use the WinSCP editor to edit main.cf. If you installed Postgrey, insert policyd-weight ahead of check_client_fqdn:
smtpd_recipient_restrictions =
     permit_mynetworks,
     permit_sasl_authenticated,
     reject_unauth_destination,
     reject_unlisted_recipient,
     check_recipient_access hash:/etc/postfix/reject_over_quota,	 
     check_sender_access hash:/etc/postfix/rbl_sender_exceptions,
     check_client_access hash:/etc/postfix/rbl_client_exceptions,
     check_recipient_access hash:/etc/postfix/rbl_recipient_exceptions,
     reject_rbl_client sbl-xbl.spamhaus.org,
     check_recipient_access hash:/etc/postfix/greylist_sender_exceptions,
     check_client_access cidr:/etc/postfix/cidr_greylist_network_exceptions,
     check_policy_service inet:127.0.0.1:12525,
     check_client_access pcre:/etc/postfix/check_client_fqdn
otherwise, add a few lines to the end:
     [...]
     reject_rbl_client sbl-xbl.spamhaus.org,
     check_recipient_access hash:/etc/postfix/greylist_sender_exceptions,
     check_client_access cidr:/etc/postfix/cidr_greylist_network_exceptions,
     check_policy_service inet:127.0.0.1:12525
Then:
postfix reload
tail -f /var/log/mail.log


Set your MUA to deliver normally to port 25 (no authentication) and then attempt to send a message through and see if you get a hit like mine did (you may have to temporarily remove your network from /etc/postfix/cidr_greylist_network_exceptions and reload postfix). Note that my client IP is not included in mynetworks in main.cf (if it were, everything in smtpd_recipient_restrictions would be bypassed). If you installed postgrey and entered your local network in cidr_greylist_network_exceptions you will have to remove it and reload postfix in order to test this. After doing so, I got:
Jul 20 20:39:42 sfa postfix/policyd-weight[4575]: weighted check:  NOT_IN_SBL_XBL_SPAMHAUS=-1.5
 NOT_IN_SPAMCOP=-1.5 NOT_IN_BL_NJABL=-1.5 CL_IP_NE_HELO=1.5 RESOLVED_IP_IS_NOT_HELO=1.5 (check from:
 .example. - helo: .myclient. - helo-domain: .myclient.)  FROM_NOT_FAILED_HELO(DOMAIN)=3 IN_AHBL=2
 IN_DSN_RFCI=3 IN_PM_RFCI=0.1 IN_ABUSE_RFCI=0.1; <client=192.168.1.41> <helo=myclient> 
 <from=test@example.com> <to=test@example.com>; rate: 6.7

Jul 20 20:39:42 sfa postfix/policyd-weight[4575]: decided action=550 Mail appeared to be SPAM or
 forged. Ask your Mail/DNS-Administrator to correct HELO and DNS MX settings or to get removed
 from DNSBLs; MTA helo: myclient, MTA hostname: unknown[192.168.1.41] (helo/hostname mismatch);
 in rhsbl.ahbl.org; in dsn.rfc-ignorant.org; in postmaster.rfc-ignorant.org; in
 abuse.rfc-ignorant.org; <client=192.168.1.41> <helo=myclient> <from=test@example.com>
 <to=test@example.com>; delay: 2s
In order to prevent internal clients from getting scanned (and possibly rejected) by policyd-weight (and postgrey), vi /etc/postfix/main.cf and place your internal network in mynetworks or whitelist the network by adding it to /etc/postfix/cidr_greylist_network_exceptions as mentioned above in the postgrey instructions.

Make a couple changes to logcheck (this is four lines)
echo "^\w{3} [ :0-9]{11} [._[:alnum:]-]+ postfix/policyd-weight\[[[:digit:]]+\]: decided action=" >>/etc/logcheck/ignore.d.server/postfix
echo "^\w{3} [ :0-9]{11} [._[:alnum:]-]+ postfix/policyd-weight\[[[:digit:]]+\]: weighted check:" >>/etc/logcheck/ignore.d.server/postfix
echo "^\w{3} [ :0-9]{11} [._[:alnum:]-]+ postfix/policyd-weight\[[[:digit:]]+\]: child: spawned" >>/etc/logcheck/ignore.d.server/postfix
tail -2 /etc/logcheck/ignore.d.server/postfix



Install SaneSecurity and MSRBL ClamAV signatures - Optional

Index Top Bottom
sanesecurity.com and msrbl.com author additional ClamAV signatures for scam, phishing, image spam and spam.
apt-get install curl rsync

cd /usr/sbin
wget http://www200.pair.com/mecham/spam/UpdateSaneSecurity.sh.txt
mv UpdateSaneSecurity.sh.txt UpdateSaneSecurity.sh
chmod +x UpdateSaneSecurity.sh
UpdateSaneSecurity.sh
ls -l /var/lib/clamav


You will notice the data has been downloaded:
drwxr-xr-x 2 clamav clamav    4096 2007-06-16 19:48 daily.inc
-rw-r--r-- 1 clamav clamav 9351789 2007-06-10 21:16 main.cvd
-rw------- 1 clamav clamav     260 2007-06-16 19:14 mirrors.dat
-rw-r--r-- 1 clamav clamav  347982 2007-06-16 19:25 MSRBL-Images.hdb
-rw-r--r-- 1 clamav clamav  228232 2007-06-08 04:33 MSRBL-SPAM.ndb
-rw-r--r-- 1 clamav clamav 1033688 2007-06-16 19:48 phish.ndb
-rw-r--r-- 1 clamav clamav  174338 2007-06-15 02:55 phish.ndb.gz
-rw-r--r-- 1 clamav clamav  516182 2007-06-16 19:48 scam.ndb
-rw-r--r-- 1 clamav clamav  102738 2007-06-15 02:55 scam.ndb.gz
Now we add a crontab entry with download attempts performed every 4th hour:
crontab -e

Insert this entry. Replace MM (minutes) below with a number between 1 and 59:
MM */4 * * * /usr/sbin/UpdateSaneSecurity.sh

Add an entry to suppress logcheck messages:
echo "UpdateSaneSecurity" >>/etc/logcheck/ignore.d.server/amavisd-new

I have heard of an occasional false positive. Amavisd-new 2.5.2 or newer can treat these as spam instead of viruses. Unfortunately the latest Debian release of amavisd-new is 2.4.2. Logs of the last download are located in /var/tmp/clamdb/


Install Botnet SA plugin - Optional

Index Top Bottom
This plugin is designed to detect botnet (zombie) clients. I trim the score down in case of false positives (which do happen). Running this check will increase scan times.
cd /usr/local/src
mkdir Botnet-0.8
cd Botnet-0.8
wget http://people.ucsc.edu/~jrudd/spamassassin/Botnet-0.8.tar
tar -xf Botnet-0.8.tar
wget http://www200.pair.com/mecham/spam/botnet8patch.txt
cp Botnet.pm Botnet.pm~
patch -p0 <botnet8patch.txt
cp Botnet.pm /etc/spamassassin/
cp Botnet.cf /etc/spamassassin/
cd /etc/spamassassin/
sed -i 's/5.0/2.0/' Botnet.cf
amavisd-new reload
spamassassin --lint


You will have to wait for some mail from distant servers to test this. Look for a hit to the BOTNET rule in the X-Spam-Status: header. This will only happen occasionally as much of this stuff may be blocked before SpamAssassin sees it.

Install p0f - Optional

Index Top Bottom
Passive OS Fingerprinting attempts to discover what type of system a message is sent from. If mail comes from a Windows XP machine (for example) one might assume the machine is a zombie.
apt-get install p0f

cd /usr/sbin
wget http://www200.pair.com/mecham/spam/p0f-analyzer.txt
mv p0f-analyzer.txt p0f-analyzer.pl
chmod +x p0f-analyzer.pl
cd /etc/init.d
wget http://www200.pair.com/mecham/spam/p0f
chmod +x p0f
update-rc.d p0f defaults
cd /etc/spamassassin
wget http://www200.pair.com/mecham/spam/p0f.cf
/etc/init.d/p0f start


vi /etc/amavis/conf.d/50-user and make these changes. Notice where $ is used, and where it is not used. You will have to jump around a little to find where to insert these three items (shown in red):
$penpals_threshold_low = 1;
$penpals_threshold_high = 18;


# Use passive OS fingerprinting
$os_fingerprint_method = 'p0f:127.0.0.1:2345';


$policy_bank{'MYNETS'} = { # mail originating from @mynetworks
  spam_admin_maps => ["postmaster\@$mydomain"], # alert of internal spam
  os_fingerprint_method => undef,
  final_spam_destiny => D_BOUNCE, # so the sender knows they are a spammer

$policy_bank{'TRUSTED'} = { # mail originating from trusted senders
  spam_admin_maps => ["postmaster\@$mydomain"], # alert of internal spam
  os_fingerprint_method => undef,
  final_spam_destiny => D_BOUNCE, # so the sender knows they are a spammer

Once the edits are finished:
amavisd-new reload

You will have to wait for some mail from distant servers to test this. Look for hits to P0F rules in the X-Spam-Status: header. If this machine is behind a NAT or proxy server that proxies the TCP sessions, P0F may not work. If you have both p0f and Botnet installed, you may find this post relevant http://marc.info/?l=spamassassin-users&m=118272807918926

Upgrade amavisd-new - Optional

Index Top Bottom
The Debian etch package of amavisd-new is somewhat outdated. Personally I find it much easier to stay current with amavisd-new if I migrate away from the Debian package. With the instructions provided we will still continue to use the Debian style configuration files. Once this is set up, we no longer have to rely on the Debian package maintainers to produce updated versions of amavisd-new. Amavisd-new is actively developed by Mark Martinec and new versions come out often. Log in as root. Before we remove amavisd-new we make backup copies of our important files. You should only perform this operation once (to avoid overwriting the original backup). When the amavisd-new package is removed, the /var/lib/amavis and /etc/amavis directories should not be removed (and I do not make a backup copy of /var/lib/amavis). If you have items in /var/lib/amavis that are important to you, you may want to consider making a copy of that directory (and subdirectories):
mkdir /usr/ambackup
mkdir -p /usr/ambackup/etc/init.d/
cp /etc/init.d/amavis /usr/ambackup/etc/init.d/amavis
cp -r /etc/amavis /usr/ambackup
mkdir /usr/ambackup/etc/cron.daily
cp /etc/cron.daily/amavisd-new /usr/ambackup/etc/cron.daily/amavisd-new
mkdir /usr/ambackup/etc/cron.d
cp /etc/cron.d/amavisd-new /usr/ambackup/etc/cron.d/amavisd-new
mkdir -p /usr/ambackup/usr/sbin
cp /usr/sbin/amavisd-new-cronjob /usr/ambackup/usr/sbin/amavisd-new-cronjob
mkdir -p /usr/ambackup/usr/share/amavis
cp -r /usr/share/amavis/conf.d /usr/ambackup/usr/share/amavis
cp /usr/sbin/amavisd-new /usr/ambackup/usr/sbin/amavisd-new
mkdir /usr/ambackup/usr/share/perl5
cp /usr/share/perl5/JpegTester.pm /usr/ambackup/usr/share/perl5/JpegTester.pm


Of the files above, JpegTester.pm, /usr/sbin/amavisd-new and /usr/share/amavis/conf.d/ should get deleted when amavisd-new is removed. We will need to restore those from the backups. There may be other files elsewhere that get deleted but they do not affect how amavisd-new functions so their absence will not hurt us. The documentation that we may loose will be provided with the original source code. Test that when you remove amavisd-new, only amavisd-new will be removed:
apt-get -s remove amavisd-new

If it is the only thing that will be removed, continue on:
/etc/init.d/amavis stop
apt-get remove amavisd-new


At this point we are temporarily going to put the same version of amavisd-new back together again so we can continue to process mail. The first command below should all fit on one line:
if ! grep -q amavis /etc/passwd; then adduser --group --system --home /var/lib/amavis --shell /bin/sh amavis; fi

cd /etc/init.d
wget http://www200.pair.com/mecham/debian/amavis-init-20030616
mv amavis-init-20030616 amavis
chmod +x amavis
mkdir -p /usr/share/amavis/conf.d
cp /usr/ambackup/usr/share/amavis/conf.d/* /usr/share/amavis/conf.d
cp /usr/ambackup/usr/sbin/amavisd-new /usr/sbin
cp /usr/ambackup/usr/sbin/amavisd-new /usr/sbin/amavisd-new-debian
cp /usr/ambackup/usr/share/perl5/JpegTester.pm /usr/share/perl5
cp /usr/ambackup/etc/cron.daily/amavisd-new /etc/cron.daily
cp /usr/ambackup/etc/cron.d/amavisd-new /etc/cron.d
cp /usr/ambackup/usr/sbin/amavisd-new-cronjob /usr/sbin
/etc/init.d/amavis start


The Debian amavisd-new package has been removed but you are still using most of the original components of the Debian package. The main difference is the /etc/init.d/amavis init script. Now it should be easy to upgrade to amavisd-new 2.5.4. We will install an amavisd.conf that simply loads in all the Debian conf files:
cd /etc/amavis
test -e amavisd.conf && mv amavisd.conf amavisd.conf-before-migration
wget http://www200.pair.com/mecham/spam/amavisd.conf-etch2.txt
mv amavisd.conf-etch2.txt amavisd.conf
wget http://www200.pair.com/mecham/amavisd/2.5.4/amavisd.conf-modified
wget http://www200.pair.com/mecham/amavisd/2.5.4/amavisd.conf-sample
ln -s /etc/amavis/amavisd.conf /etc/amavisd.conf

cd /usr/local/src
wget http://www.ijs.si/software/amavisd/amavisd-new-2.5.4.tar.gz
tar xzf amavisd-new-2.5.4.tar.gz
cd amavisd-new-2.5.4
cp amavisd /usr/sbin/amavisd-new-2.5.4
cp amavisd /usr/sbin/amavisd-new
cp p0f-analyzer.pl /usr/sbin
test -e /usr/sbin/p0f-analyzer && rm /usr/sbin/p0f-analyzer
ln -s /usr/sbin/p0f-analyzer.pl /usr/sbin/p0f-analyzer
cp amavisd-nanny /usr/sbin/
cp amavisd-release /usr/sbin/
sed -i 's|/var/amavis/amavisd.sock|/var/run/amavis/amavisd.sock|' /usr/sbin/amavisd-release
sed -i 's|/var/amavis/db|/var/lib/amavis/db|' /usr/sbin/amavisd-nanny
amavisd-new reload
cd /etc/amavis
amavisd-new -V


The amavisd.conf-modified file is an example of the amavisd.conf supplied with the source code but it has been modified for use with Debian. I suggest you read /etc/amavis/amavisd.conf.

This version of amavisd-new can treat sanesecurity and MSRBL 'virusus' as spam. If you are using the sanesecurity ClamAV signatures, download a rule set to score these as spam:
cd /etc/spamassassin
wget http://www200.pair.com/mecham/spam/amavis-sanesecurity_v2.cf
spamassassin --lint


If you are currently using external notification templates (you probably are since they are enabled by default in /etc/amavis/conf.d/30-template_localization), you should get the templates that match amavisd-new 2.5.4 and if necessary modify them in the same manner you have modified the currently installed 2.4.2 templates. The only templates available are English templates. If you don't normally customize your templates you should comment out:
read_l10n_templates('en_US', '/etc/amavis');

in the 30-template_localization file.

cp -r /etc/amavis/en_US /etc/amavis/en_US-backup
cd /etc/amavis/en_US
rm charset
rm template-dsn.txt
rm template-spam-admin.txt
rm template-spam-sender.txt
rm template-virus-admin.txt
rm template-virus-recipient.txt
rm template-virus-sender.txt
wget http://www200.pair.com/mecham/amavisd/2.5.4/en_US/charset
wget http://www200.pair.com/mecham/amavisd/2.5.4/en_US/template-dsn.txt
wget http://www200.pair.com/mecham/amavisd/2.5.4/en_US/template-spam-admin.txt
wget http://www200.pair.com/mecham/amavisd/2.5.4/en_US/template-spam-sender.txt
wget http://www200.pair.com/mecham/amavisd/2.5.4/en_US/template-virus-admin.txt
wget http://www200.pair.com/mecham/amavisd/2.5.4/en_US/template-virus-recipient.txt
wget http://www200.pair.com/mecham/amavisd/2.5.4/en_US/template-virus-sender.txt
amavisd-new reload


If for some crazy reason you should want to reinstall amavisd-new from the Debian package:
apt-get update

/etc/init.d/amavis stop
mv /etc/amavis/amavisd.conf /etc/amavis/amavisd.conf-mybackup
rm /etc/amavisd.conf
apt-get install amavisd-new


If you added configuration items that are only valid for newer versions of amavisd-new then amavisd-new should complain and you will have to remove them before it will start.

Now that you are running a newer version of amavisd-new we can use a function that was not available in older versions. Temporarily set $log_level to 1 and reload amavisd-new:
vi /etc/amavis/conf.d/50-user

and change:
$log_level = 0;
to:
$log_level = 1;

Then reload amavisd-new:
amavisd-new reload

Now send a message through (it needs to pass through SpamAssassin) and look in the log for a line similar to this:

Jun 23 10:00:05 msa amavis[5359]: (05359-01) extra modules loaded: /etc/spamassassin/Botnet.pm

We want to vi /etc/amavis/amavisd.conf again and take each module(s) listed and place it (them) in @additional_perl_modules (via a list of quoted words). You need to remove any commas that may separate items (and comments are not allowed within the qw() function). You could place this near the bottom of amavisd.conf in the area I have set aside for placing configuration items. The result for the log output above would be something like:
@additional_perl_modules = qw(
 /etc/spamassassin/Botnet.pm
);
There is a new setting 'spam_dsn_cutoff_level_bysender_maps'. If our users send spam we want to bounce it back to them. We now need to set this in addition to 'spam_dsn_cutoff_level_maps' in order to continue to do so. There is also a new 'originating' policy bank key available. Setting this to a '1' (true) in a policy bank says to amavisd-new that the client is a local client of ours. This is similar to a client that is included in @mynetworks and you are using the MYNETS policy bank. We will make use of this in the TRUSTED policy bank (and explicitly set it in MYNETS too), so vi /etc/amavis/conf.d/50-user and insert these red items in our two policy banks:

$policy_bank{'MYNETS'} = { # mail originating from @mynetworks
  originating => 1,
  spam_admin_maps => ["postmaster\@$mydomain"], # alert of internal spam
  final_spam_destiny => D_BOUNCE, # so the sender knows they are a spammer
  spam_kill_level_maps => [10.0],
  spam_dsn_cutoff_level_maps => [9999],

  spam_dsn_cutoff_level_bysender_maps => [9999],

  banned_filename_maps => ['BLOCK_EXE'],
};


$policy_bank{'TRUSTED'} = { # mail originating from trusted senders
  originating => 1,
  spam_admin_maps => ["postmaster\@$mydomain"], # alert of internal spam
  final_spam_destiny => D_BOUNCE, # so the sender knows they are a spammer
  spam_kill_level_maps => [10.0],
  spam_dsn_cutoff_level_maps => [9999],

  spam_dsn_cutoff_level_bysender_maps => [9999],

  banned_filename_maps => ['BLOCK_EXE'],
};


Then reload amavisd-new:
amavisd-new reload

If you were to send another test message through (at $log_level = 1; or greater), the "extra modules loaded" log entry should be suppressed. After testing it you may want to set the $log_level to 0 (or leave it alone until after we are finished with the installation.


Upgrade SpamAssassin - Optional

Index Top Bottom
It is highly recommended that you are using amavisd-new 2.5.4 or newer before you upgrade SpamAssassin using these instructions. I have only tested this with 2.5.4. At this time, the safest way to upgrade SpamAssassin is to install it using the backports.org sources. First check that SA 3.2.3 lints clean. If it does not, you should fix it before continuing:
spamassassin --lint

Make a backup copy of your spamassassin config:
test -e /usr/ambackup || mkdir /usr/ambackup
cp -r /etc/spamassassin /usr/ambackup


While SA version 3.2.3 supplied with etch works well, there will be some who want a newer version of SpamAssassin. The 3.2.x versions allows you to compile some rules using a new 'sa-compile' command (you are not required to do so, but it may help with the speed issue a little bit). This command requires the re2c program (from regular expression to c). Once you have compiled these rules, you use them by enabling the Rule2XSBody plugin in v320.pre. If you were using SpamAssassin 3.1.x, you may or may not have installed the ImageInfo plugin, but if you have, you need to remove it because it is now included in SA 3.2.x. Also, we must remove the loopback interface from internal_networks and trusted_networks (if they are there) because it is now automatically included. We begin by adding the backports source to /etc/apt/sources.list and configure apt pinning:
vi /etc/apt/sources.list

and insert the backports source, similar to what I have done here:
deb http://mirrors.kernel.org/debian/ etch main non-free contrib
deb-src http://mirrors.kernel.org/debian/ etch main

deb http://security.debian.org/ etch/updates main non-free contrib
deb-src http://security.debian.org/ etch/updates main

deb http://volatile.debian.net/debian-volatile etch/volatile main


deb http://www.backports.org/debian etch-backports main contrib non-free


Since we now have a few different sources it's probably a good idea to increase the default apt-cache limit. Assuming you have not already done this at some point in the past:
echo 'APT::Cache-Limit "25165824";' >> /etc/apt/apt.conf

I also suggest adding gnupg keys for a few of the apt sources to apt:
gpg --keyserver subkeys.pgp.net --recv-key BBE55AB3
gpg --armor --export BBE55AB3 | apt-key add -
gpg --keyserver subkeys.pgp.net --recv-key 6070D3A1
gpg --armor --export 6070D3A1 | apt-key add -
gpg --keyserver subkeys.pgp.net --recv-key 16BA136C
gpg --armor --export 16BA136C | apt-key add -
gpg --keyserver subkeys.pgp.net --recv-key 276981F4
gpg --armor --export 276981F4 | apt-key add -


Since we are running etch, etch currently has a default pin-priority of 500. We need to make certain that the backports source has a lower priority:
vi /etc/apt/preferences

and insert (if you don't already have something similar):
Package: *
Pin: release a=etch-backports
Pin-Priority: 200


We must install re2c if we want to use the new sa-compile command:
apt-get update

Now we install a re2c and few other programs that will support Mail::SPF, Mail::DKIM and Image::Info:
apt-get install re2c libversion-perl liberror-perl libimage-info-perl

Version 3.2.x prefers we use Mail::SPF instead of Mail::SPF::Query. We will install this from backports. We will also install :
apt-get -t etch-backports install libmodule-build-perl

apt-get -t etch-backports install libnetaddr-ip-perl

apt-get -t etch-backports install libnet-dns-resolver-programmable-perl

apt-get -t etch-backports install libmail-spf-perl

apt-get -t etch-backports install libmail-dkim-perl


If you are running SA version 3.1.x (use "apt-cache policy spamassassin" to see if you are) we need to remove the manually installed 3rd party ImageInfo plugin if installed:
test -e /usr/share/perl5/Mail/SpamAssassin/Plugin/ImageInfo.pm && rm /usr/share/perl5/Mail/SpamAssassin/Plugin/ImageInfo.pm

test -e /etc/spamassassin/imageinfo.cf && rm /etc/spamassassin/imageinfo.cf
sed -i 's/loadplugin Mail::SpamAssassin::Plugin::ImageInfo//' /etc/spamassassin/v310.pre
grep -r :ImageInfo /etc/spamassassin/


If you are running SA version 3.1.x, that grep statement should show that the ImageInfo plugin is not set to load. If you are running 3.2.x, it will show that it is set to load (which is fine). Now we need to make another change:
vi /etc/spamassassin/local.cf

and remove the loopback interface from internal_networks and trusted_networks:
# explicitly set our internal_networks (might be the same or similar to mynetworks)
clear_internal_networks
internal_networks 127/8
internal_networks 222.222.222.222/24
internal_networks 10.10.10.10/24
# add the same to trusted_networks, and possibly other computers/networks whose mail we trust
clear_trusted_networks
trusted_networks 127/8
trusted_networks 222.222.222.222/24
trusted_networks 10.10.10.10/24


Now see what version you are going to get:
apt-cache policy spamassassin

Today my machine shows:
spamassassin:
  Installed: 3.2.3-0.volatile1
  Candidate: 3.2.3-0.volatile1
  Version table:
     3.2.5-1 0
        200 http://www.backports.org etch-backports/main Packages
 *** 3.2.3-0.volatile1 0
        500 http://volatile.debian.net etch/volatile/main Packages
        100 /var/lib/dpkg/status
     3.1.7-2 0
        500 http://mirrors.kernel.org etch/main Packages
Looks fine, so I will remove 3.2.3 and install 3.2.5. You may be prompted whether to overwrite v310.pre or not. We can keep our modified file, so you can answer with the default 'N':
amavisd-new stop
apt-get -s remove spamassassin


I assume it will only remove spamassassin. If this is true, we can proceed to remove it and then install the new version (not that your configuration files in /etc/spamassassin should not be removed as a result of removing spamassassin):
apt-get remove spamassassin

apt-get -t etch-backports install spamassassin

spamassassin --lint

sa-update


Reimport the openprotect GPG key:
cd /etc/spamassassin
wget http://saupdates.openprotect.com/pub.gpg
sa-update --import pub.gpg


If there were show stopping problems with the --lint, you need to fix them. If you reinstall version 3.1.7, please delete v320.pre when you do. If things get real bad you might have to '--purge remove' and then reinstall. You might have to reimport the OpenProtect gpg key. You need soon need to get your mail flowing again:
amavisd-new start
tail -f /var/log/mail.log


You should vi /usr/sbin/sa-update.sh and see if you entered in any custom commands that may need to be updated to the new directory /var/lib/spamassassin/3.002005. Then we should update the openprotect rules by manually running the script:
sa-update.sh

Continue to tail the mail.log in the current PuTTY session. Open a second PuTTY session and work from there. If you are up and running on 3.2.x, you should consider running sa-compile:
sa-compile

If that looks good, you can enable the compiled body rules (this command is on one line):
sed -i 's/# loadplugin Mail::SpamAssassin::Plugin::Rule2XSBody/loadplugin Mail::SpamAssassin::Plugin::Rule2XSBody/' /etc/spamassassin/v320.pre

Make sure that loadplugin setting is no longer commented out and make sure lint is clean.
grep Rule2XSBody /etc/spamassassin/v320.pre
spamassassin --lint


Now grab a script that will run sa-compile once a week:
cd /etc/cron.weekly
wget http://www200.pair.com/mecham/spam/sa-compile.sh.txt
mv sa-compile.sh.txt sa-compile-weekly
chmod +x sa-compile-weekly


Run the script. If it is working correctly, it should take a while; there should be no output and amavisd-new should get reloaded. The script does require that sa-compile has ran at least once before. If you are still tailing the log in the other session, you should eventually see amavisd-new reload:
./sa-compile-weekly

You can close the second PuTTY session if you like. You should be good to go. Send a test message through to make sure you can receive mail. Don't be surprised if messages take longer to get scanned and SA puts more load on you CPU(s).

If you upgraded amavisd-new, you set a $log_level at 1 or above and you enabled the Rule2XSBody plugin; if you were to grep modules /var/log/mail.log you will notice:
Dec 6 10:42:46 msa amavis[7290]: extra modules loaded after daemonizing:
Mail/SpamAssassin/CompiledRegexps/body_0.pm, Mail/SpamAssassin/CompiledRegexps/body_500.pm


Don't worry about these (do not add Mail/SpamAssassin/CompiledRegexps/body_500.pm to @additional_perl_modules), but other modules may be found (remember, you must ge at $log_level = 1; to see these), for example:
Dec 6 10:43:52 msa amavis[7297]: (07297-01) extra modules loaded:
unicore/lib/gc_sc/Digit.pl, unicore/lib/gc_sc/SpacePer.pl


So, vi /etc/amavis/amavisd.conf and update @additional_perl_modules as needed (this is only an example):
@additional_perl_modules = qw(
  <...other stuff...>

  unicore/lib/gc_sc/Digit.pl
  unicore/lib/gc_sc/SpacePer.pl

);


You can also vi /etc/amavis/amavisd.conf and optionally include these settings. However, this is SA version specific and will need to be edited if you upgrade SpamAssassin:
my($sa_instdir) = '/var/lib/spamassassin/compiled/3.002005';
unshift(@INC, $sa_instdir, $sa_instdir.'/auto');


Then, of course:
amavisd-new reload

and send a test message through.

Since we should have all the pieces in place, we can optionally enable the DKIM plugin:
sed -i 's/#loadplugin Mail::SpamAssassin::Plugin::DKIM/loadplugin Mail::SpamAssassin::Plugin::DKIM/' /etc/spamassassin/v312.pre

Then:
spamassassin --lint

If all is well:
amavisd-new reload
and watch for errors once again. Remember to set $log_level back to 0.



Install FuzzyOCR - Optional

Index Top Bottom
FuzzyOCR attempts to detect image spam. It is of somewhat limited value at this time. It can add to scan times. It works best if you spend time understanding how to use debug output to help with tweaking scan sets and key words. Spammers are especially wise to FuzzyOCR and intentionally obscure image spam to work around it. FuzzyOCR can at times be CPU intensive.
apt-get install lsb-base libice6 libx11-6 xlibs-data libsm6 x11-common gifsicle ocrad

apt-get install netpbm bzip2 libmldbm-perl libstring-approx-perl libmldbm-sync-perl

apt-get install liblog-agent-perl libdbi-perl libdbd-mysql-perl libtie-cache-perl

cd /usr/local/src
wget http://internap.dl.sourceforge.net/sourceforge/giflib/libungif-4.1.4.tar.gz

tar xzvf libungif-4.1.4.tar.gz
cd libungif-4.1.4/util
wget http://users.own-hero.net/~decoder/fuzzyocr/giftext-segfault.patch
patch giftext.c < giftext-segfault.patch
cd ..
./configure --prefix=/usr && make && make install

cd /usr/local/src
wget http://www-e.uni-magdeburg.de/jschulen/ocr/gocr-0.43.tar.gz
tar xzvf gocr-0.43.tar.gz
cd gocr-0.43
./configure --with-netpbm=/usr/lib --prefix=/usr && make && make install

If you are using SpamAssassin 3.2.x, perform these next green colored statements:
cd /usr/local/src
svn -r 131 co https://svn.own-hero.net/fuzzyocr/trunk/devel

mv devel FuzzyOCR-svn-131
cd FuzzyOCR-svn-131
cp FuzzyOcr.preps FuzzyOcr.preps~
cp FuzzyOcr.scansets FuzzyOcr.scansets~
cp FuzzyOcr.cf FuzzyOcr.cf~


If you are using SpamAssassin 3.1.7, perform these next purple colored statements:
cd /usr/local/src
wget http://users.own-hero.net/~decoder/fuzzyocr/fuzzyocr-3.5.1-devel.tar.gz
tar xzf fuzzyocr-3.5.1-devel.tar.gz
cd FuzzyOcr-3.5.1
cp FuzzyOcr.preps FuzzyOcr.preps~
cp FuzzyOcr.scansets FuzzyOcr.scansets~
cp FuzzyOcr.cf FuzzyOcr.cf~


The rest can be used with either version:
wget http://www200.pair.com/mecham/spam/FuzzyOcr-3.5.0-rc1.netpbm_less_than_10.34.patch
wget http://www200.pair.com/mecham/spam/FuzzyOcr-3.5.0-rc1.netpbm_less_than_10.34.patch2
wget http://www200.pair.com/mecham/spam/FuzzyOcr-3.5.0-rc1.netpbm_less_than_10.34.p3
wget http://www200.pair.com/mecham/spam/capFuzzy.txt
patch -p0 < FuzzyOcr-3.5.0-rc1.netpbm_less_than_10.34.patch
patch -p0 < FuzzyOcr-3.5.0-rc1.netpbm_less_than_10.34.patch2
patch -p0 < FuzzyOcr-3.5.0-rc1.netpbm_less_than_10.34.p3
patch -p0 < capFuzzy.txt

wget http://www200.pair.com/mecham/spam/gary.3.5.0-rc1.old.netpbm.patch1
wget http://www200.pair.com/mecham/spam/gary.3.5.0-rc1.old.netpbm.patch2
wget http://www200.pair.com/mecham/spam/gary.3.5.0-rc1.old.netpbm.patch3
patch -p0 < gary.3.5.0-rc1.old.netpbm.patch1
patch -p0 < gary.3.5.0-rc1.old.netpbm.patch2
patch -p0 < gary.3.5.0-rc1.old.netpbm.patch3

cp -r FuzzyOcr /etc/mail/spamassassin
cp FuzzyOcr.cf /etc/mail/spamassassin
cp FuzzyOcr.pm /etc/mail/spamassassin
cp FuzzyOcr.preps /etc/mail/spamassassin
cp FuzzyOcr.scansets /etc/mail/spamassassin
cp FuzzyOcr.words /etc/mail/spamassassin

cd /etc/logrotate.d/
touch /var/log/FuzzyOcr.log
chown amavis:amavis /var/log/FuzzyOcr.log
wget http://www200.pair.com/mecham/spam/FuzzyRotate.txt
mv FuzzyRotate.txt FuzzyOcr

cd /etc/spamassassin
sed -i 's/#focr_verbose 3/focr_verbose 0/' FuzzyOcr.cf
sed -i 's|#focr_logfile /tmp/FuzzyOcr.log|focr_logfile /var/log/FuzzyOcr.log|' FuzzyOcr.cf
sed -i 's/#focr_threshold 0.20/focr_threshold 0.21/' FuzzyOcr.cf
sed -i 's/#focr_base_score 5/focr_base_score 2/' FuzzyOcr.cf
sed -i 's/#focr_add_score 0.375/focr_add_score 0.375/' FuzzyOcr.cf
sed -i 's/#focr_corrupt_score 2.5/focr_corrupt_score 0.1/' FuzzyOcr.cf
sed -i 's/#focr_corrupt_unfixable_score 5/focr_corrupt_unfixable_score 2/' FuzzyOcr.cf

Make sure this comes up clean:
spamassassin --lint

Run a test (you should get FUZZY_OCR rules hitting):

If running SA 3.2.x:
cd /usr/local/src/FuzzyOCR-svn-131/samples
su amavis -c 'spamassassin -tD < ocr-animated.eml'


If running SA 3.1.7:
cd /usr/local/src/FuzzyOCR-3.5.1/samples
su amavis -c 'spamassassin -tD < ocr-animated.eml'


Other tests are also available:
su amavis -c 'spamassassin -tD < ocr-gif.eml'

su amavis -c 'spamassassin -tD < ocr-jpg.eml'

su amavis -c 'spamassassin -tD < ocr-multi.eml'

su amavis -c 'spamassassin -tD < ocr-obfuscated.eml'

su amavis -c 'spamassassin -tD < ocr-png.eml'

su amavis -c 'spamassassin -tD < ocr-wrongext.eml'

If you want to see details, increase focr_verbose in /etc/spamassassin/FuzzyOcr.cf and look in /var/log/FuzzyOcr.log (but set it back to 0 when finished testing). If everything looks OK so far:
amavisd-new stop
amavisd-new debug-sa


Browse to http://www200.pair.com/mecham/spam/host.gif and save it to your computer. Send another test message into the mailserver with this gif attached and watch the debug output for signs of errors. In my output I found where FuzzyOcr added 5.188 points:
[23165] info: FuzzyOcr: Message is spam, score = 5.188
Also read the headers of the message that was received. If everything looks Ok, stop debugging with [Ctrl]+c, then:
amavisd-new start

vi /etc/amavis/conf.d/50-user and set $log_level = 1; and then amavisd-new reload . tail -f /var/log/mail.log and send a test message through and once again look for:
Jul 26 16:07:08 msa amavis[24971]: (24971-01) extra modules loaded: 
 /etc/spamassassin/FuzzyOcr.pm, /usr/lib/perl/5.8/auto/Storable/autosplit.ix,
 /usr/share/perl5/auto/Log/Agent/Priorities/autosplit.ix, 
 /usr/share/perl5/auto/Log/Agent/autosplit.ix, FuzzyOcr/Config.pm, FuzzyOcr/Deanimate.pm,
 FuzzyOcr/Hashing.pm, FuzzyOcr/Logging.pm, FuzzyOcr/Misc.pm, FuzzyOcr/Preprocessor.pm,
 FuzzyOcr/Scanset.pm, FuzzyOcr/Scoring.pm, Log/Agent.pm, Log/Agent/Formatting.pm,
 Log/Agent/Message.pm, Log/Agent/Priorities.pm, MLDBM.pm, MLDBM/Sync.pm,
 MLDBM/Sync/SDBM_File.pm, SDBM_File.pm, Storable.pm, String/Approx.pm, Tie/Cache.pm
Each of these modules (less body_500.pm and body_0.pm) should be added to @extra_perl_modules (without the commas). The net result may look like:
@additional_perl_modules = qw(
 /etc/spamassassin/Botnet.pm
 /etc/spamassassin/FuzzyOcr.pm
 /usr/lib/perl/5.8/auto/Storable/autosplit.ix
 /usr/share/perl5/auto/Log/Agent/Priorities/autosplit.ix
 /usr/share/perl5/auto/Log/Agent/autosplit.ix
 FuzzyOcr/Config.pm FuzzyOcr/Deanimate.pm FuzzyOcr/Hashing.pm
 FuzzyOcr/Logging.pm FuzzyOcr/Misc.pm FuzzyOcr/Preprocessor.pm
 FuzzyOcr/Scanset.pm FuzzyOcr/Scoring.pm Log/Agent.pm
 Log/Agent/Formatting.pm Log/Agent/Message.pm Log/Agent/Priorities.pm
 MLDBM.pm MLDBM/Sync.pm MLDBM/Sync/SDBM_File.pm SDBM_File.pm
 Storable.pm String/Approx.pm Tie/Cache.pm
);
So, vi /etc/amavis/amavisd.conf and do your stuff, then of course:
amavisd-new reload

Send one more test message through - if all is well you can set the $log_level back to 0 and reload amavisd-new again.

Install AIDE - Optional

Index Top Bottom
The concept with AIDE is to take a snapshot of some of the characteristics of critical files on our system, and then store that information in a database on read-only media; a write protected floppy for example. Then run a program that compares the current state of those files on our system with the database and report any differences. We should be able to tell if a hacker has compromised our system if the characteristics of the files on our system differ from the files in the database.

Place a good writable floppy in the floppy drive, then:
mkdir /floppy
test -e /dev/fd0u1722 || mknod /dev/fd0u1722 b 2 60
chmod 660 /dev/fd0u1722
chown root:floppy /dev/fd0u1722
fdformat /dev/fd0u1722


If the format fails, try again. If it fails again, try a different floppy. If it continues to fail, try a different brand of floppy or a different floppy drive. If the format goes well:
mke2fs /dev/fd0u1722

cd /etc/init.d
wget http://www200.pair.com/mecham/spam/startflop.txt
mv startflop.txt startflop
chmod +x startflop
update-rc.d startflop defaults 80
ln -s /etc/init.d/startflop /usr/bin/floppy


Now you can mount the floppy with floppy start and unmount the floppy with floppy stop (as long as the floppy is formatted as /dev/fd0u1722). The floppy will be left in the machine at all times and will automatically get mounted each time the system starts. You should reboot the computer and see if it starts back up. This is not a bootable floppy but some systems may hang if the BIOS is set to boot from the floppy drive. You may have to change the boot order in the BIOS so your server boots from the (first) hard drive before it boots from the floppy. Now install aide (choose the default responses to questions):
cd /usr/local/src
wget http://www200.pair.com/mecham/spam/aide_0.10-4_i386.deb
dpkg -i aide_0.10-4_i386.deb


Daily reports are mailed to root by default. [Ok]
Initialize aide database? [No]
Before AIDE can be used, you will have to initialize a database. [Ok]


We are going to obscure aide somewhat, use my configuration file and report and then initialize the database (which will take some time):
floppy start
mkdir /root/aide
cd /root/aide
cp /usr/bin/aide /root/aide
cp /usr/share/man/man1/aide.1.gz /root/aide
cp /usr/share/man/man5/aide.conf.5.gz /root/aide
cp /usr/share/man/man8/aideinit.8.gz /root/aide
cp /etc/cron.daily/aide /root/aide/report
cp /etc/cron.daily/aide /root/aide/report~
cp /etc/aide/aide.conf /root/aide/aide.conf
cp /etc/aide/aide.conf /root/aide/aide.conf~
wget http://www200.pair.com/mecham/spam/aide.conf.patch.txt
patch -p0 <aide.conf.patch.txt
wget http://www200.pair.com/mecham/spam/report.patch.txt
patch -p0 <report.patch.txt
chmod +x report

apt-get remove aide

mkdir /var/log/aide
cp /root/aide/aide.1.gz /usr/share/man/man1/
cp /root/aide/aide.conf.5.gz /usr/share/man/man5/
cp /root/aide/aideinit.8.gz /usr/share/man/man8/
cp /root/aide/aide /floppy
cp /root/aide/aide.conf /floppy
cp /root/aide/report /floppy
/floppy/aide --config=/floppy/aide.conf --init


Wait for the initial database to be created (/root/aide/aide.db.new) then copy it to the floppy:
cp /root/aide/aide.db.new /floppy/aide.db

Once the file copies over (wait about 30 seconds), run the report for the first time (this will take a while). Once the report completes, check the mailbox for the user that gets root's mail:
/floppy/report

Each time you run the report, it reads the database /floppy/aide.db but it also creates a new aide.db.new database. So why do we care about the aide.db.new? It's a matter of convenience. If the report is getting kind of long, and there appears to be nothing in it you would not expect; the aide.db.new database reflects the current state of the system and could be used to replace aide.db. You replace your aide.db on the floppy with aide.db.new, just as you did when you first initialized the system. If you run the report again, it should be clean, unless files have changed since it was created. You need to understand the risks of relying on the quality of the aide.db.new database if you use the copy that was created during the cron job. The cron job could have been hacked. A hacker could manipulate the file. Only rely on the aide.db.new database if you run   /floppy/report   from the command line and you inspect the report just prior to copying it over to the floppy. If you left the write protection off the floppy drive for any length of time, it's possible you can no longer rely on the database. Be paranoid.

So here's the gist of maintenance:
  • Run the report.
  • Read the report.
  • If it looks normal, and it has not grown to the point it is annoying, you're done for today.
  • If the report lists any critical system files you know you did not change or update, or you notice other suspicious activity,
    I hope you have a cloned hard drive ready to swap out because you may have been hacked. Study the report carefully.
  • If everything looks normal and you want it cleaned up, unmount the floppy, flip the write protect tab off, mount the floppy,
    copy /root/aide/aide.db.new to /floppy/aide.db, unmount the floppy, flip the write protect tab on, mount the floppy, done.
I have a script that can help with this last item. It's a lot more convenient if you are at the console when you run this (but it's not required). Please download it:
cd
wget http://www200.pair.com/mecham/spam/go.txt
mv -i go.txt go
chmod +x go


Then run it to see how it works:
./go

You should now have the program and data on the write protected floppy. Set up a cron job to run the report:
crontab -e

and insert something like this:
25 7 * * * /floppy/report

It is typical that the file '/dev/.udev/uevent_seqnum' will be on every report.


Install altermime - Optional

Index Top Bottom
If you feel the need to add a disclaimer to outgoing mail you can use altermime to accomplish this. I am going to assume you have installed amavisd-new 2.5.2 or newer. Version 2.4.2 does not have the necessary hook to call altermime. If you are still using 2.4.2 then you can use some of my preliminary notes if you like, but I abandoned that project when altermime could be called from amavisd-new. We are going to use the latest devel release.
cd /etc
wget http://www200.pair.com/mecham/spam/disclaimer.txt
cd /usr/local/src
wget http://pldaniels.com/altermime/altermime-0.3-dev.tar.gz
tar xzf altermime-0.3-dev.tar.gz
cd altermime-0.3-dev
make && make install


Now vi /etc/amavis/amavisd.conf and add this:
# altermime disclaimers
$altermime = '/usr/local/bin/altermime'; # a path to the program
@altermime_args_disclaimer = qw(--disclaimer=/etc/disclaimer.txt);
$defang_maps_by_ccat{+CC_CLEAN} = [ 'disclaimer' ];
$defang_maps_by_ccat{+CC_BADH} = [ 'disclaimer' ];


Now we want to enable disclaimers for outgoing mail. This is done through the MYNETS and/or TRUSTED policy banks. You need to insure that the IP address of clients sending mail from the internal network are included in @mynetworks - or your clients are authenticated and therefore using the TRUSTED policy bank. Now vi /etc/amavis/conf.d/50-user and add the red items as shown:
$policy_bank{'MYNETS'} = { # mail originating from @mynetworks
  originating => 1,

  allow_disclaimers => 1,
  spam_admin_maps => ["postmaster\@$mydomain"], # alert of internal spam


$policy_bank{'TRUSTED'} = { # mail originating from trusted senders
  originating => 1,
  allow_disclaimers => 1,
  spam_admin_maps => ["postmaster\@$mydomain"], # alert of internal spam


Then of course:
amavisd-new reload

Log into SquirrelMail and send a message. If all is well your disclaimer will be added. Of course you have to customize it:
vi /etc/disclaimer.txt

For the moment there is one limitation: there can only be one manger in effect at a time, it is not currently possible to both defang and to append a disclaimer, e.g. for internal-to-internal mail inserting a disclaimer takes precedence. Also, if you need different disclaimers for different domains, search for @disclaimer_options_bysender_maps in http://www.ijs.si/software/amavisd/release-notes.txt.


Odds and ends

Index Top Bottom
If you wish to give clients within your network the ability to relay mail to domains other than yours then you should place your network (or individual IP addresses) in mynetworks in main.cf. You can also exclude certain hosts in your network by preceding the IP address with an exclamation point. Excluded addresses need to be listed before included addresses. You should also place the network in @mynetworks in 50-user in order for the clients to take advantage of the MYNETS policy bank.

You should build another server dedicated to holding a nightly backup of this server. The minimal Debian installation should be fine. It should have a very large hard drive, but I imagine it need not be a powerful machine. I used http://www.howtoforge.com/linux_rdiff_backup and can say it works well. Make sure iptables allows SSH from the backup server. I like the way it explains how to connect via SSH without using a password. Also how to limit what the connected party can run. I did have a problem with the "ssh-copy-id" command - read the comments at the bottom of the first page for a solution.

I suggest reading:
http://www200.pair.com/mecham/spam/amavisd-settings.html

You should read this:
http://www200.pair.com/mecham/spam/bypassing.html

I suggest reading:
http://jaqque.sbih.org/kplug/apt-pinning.html

This can help determine how long SpamAssassin spends on certain items:
http://marc.info/?l=amavis-user&m=117874388132138

I archive copies of messages. Here are some hints on how I do it:
http://www.freespamfilter.org/forum/viewtopic.php?t=526

You can redefine %banned_rules (make sure you include 'DEFAULT') in order to set per-recipient banned rules sets (in SQL).
http://marc.info/?l=amavis-user&m=111995411713191

I think you should consider building a secondary MX server that can spool mail if this machine goes down. It also reduces load on this server because spammers often target the secondary MX. You will need to populate relay_recipients on that machine. FYI you can gather all the local recipients into a file with commands such as these:
mysql -upostfix -ppfix_password postfix -B -N -e "select concat(username, ' OK') from mailbox" >file1
mysql -upostfix -ppfix_password postfix -B -N -e "select concat(address, ' OK') from alias" >file2
mysql -upostfix -ppfix_password postfix -B -N -e "select concat(goto, ' OK') from alias" >file3
cat file1 file2 file3 >file4
cat file4 | egrep -v ^root@ | sort | uniq >file5


Here are links to most of the stuff you get from me (in the order downloaded):
my-medium.cnf.patch.txt
virtual.index.html
database_mysql.patch.txt
postfixadmin.autocreate.patch.txt
postfixadmin.lowercase.patch.txt
postfixadmin.multipatch.txt
admin-edit-mailbox.patch.txt
edit-mailbox.patch.txt
maildirmake.sh.txt
quotachange.sh.txt
mysql_virtual_alias_maps.cf
mysql_virtual_domains_maps.cf
mysql_virtual_mailbox_maps.cf
maildroprc.txt
maildrop.logrotate.txt
authmysqlrc.txt
maildircheck.txt
rmmailboxspam.txt
sa-update1.sh.txt
amavisd-new-trim-whitespace.patch.txt
amavisd-new-trim-whitespace.patch2.txt
amavis-sqmail.sql.txt
domain.patch.txt
trim-amavis.sql.txt
trim-amavis.txt
rmvirusquar.txt
gv-bayes-awl.sql.txt
local.cf-bayes-awl.txt
sample-spam.txt
trim-amavis-msgs.txt
trim-awl.sql.txt
trim-awl.txt
trim-sql-awl-weekly.txt
amavisnewsql.patch1.txt
pflogsumm.sh
gv-vac.txt
postfixadmin_forward.patch.txt
mzcpatch.txt
mailzu.logrotate.txt
mysql_relay_domains_maps.cf
mysql_virtual_domains_maps.cf.v2
sa-update.sh.txt
check_client_fqdn
policyd-weight.conf
UpdateSaneSecurity.sh.txt
p0f-analyzer.txt
p0f
p0f.cf
amavis-init-20030616
amavisd.conf-etch2.txt
amavisd.conf-modified
amavisd.conf-sample
charset
template-dsn.txt
template-spam-admin.txt
template-spam-sender.txt
template-virus-admin.txt
template-virus-recipient.txt
template-virus-sender.txt
ImageInfo.pm.txt
imageinfo.cf.txt
sa-compile.sh.txt
FuzzyOcr-3.5.0-rc1.netpbm_less_than_10.34.patch
FuzzyOcr-3.5.0-rc1.netpbm_less_than_10.34.patch2
FuzzyOcr-3.5.0-rc1.netpbm_less_than_10.34.p3
capFuzzy.txt
gary.3.5.0-rc1.old.netpbm.patch1
gary.3.5.0-rc1.old.netpbm.patch2
gary.3.5.0-rc1.old.netpbm.patch3
FuzzyRotate.txt
startflop.txt
aide.conf.patch.txt
report.patch.txt
go.txt

Here are general links:
Postfix
MySQL
Apache http 1.3
Courier IMAP docs
phpMyAdmin
amavisd-new
SpamAssassin
PostfixAdmin
SquirrelMail docs
Courier Maildrop
Vupil's Razor
Pyzor
pflogsumm
Bind
MailZu
MailGraph
mysql-zrm
OpenProtect SA updates
SARE
SARE Plugins
DCC
PostGrey
policyd-weight
sanesecurity
MSRBL
BotNet
p0f
FuzzyOCR

Apache and Courier are not tuned to your particular level of traffic. I leave it up to you to discover how many processes (and other tweaks) you might need.

Something to consider. When a user creates an alias that forwards to another address, depending on spam settings, there is a potential for that address to be rewritten to user+spam@example.com and the final destination server may not accept mail in that format. In this case another alias would need to be created to change the address back to user@example.com

You need to subscribe to:
http://lists.debian.org/debian-security-announce/
http://lists.clamav.net/mailman/listinfo/clamav-announce

You may like this:
http://www.mikecappella.com/logwatch/

You may also want to install webalizer:
apt-get install webalizer
Then vi /etc/webalizer/webalizer.conf and change the LogFile so it points to /var/log/apache-ssl/access.log. Run webalizer and then browse to https://your_home_page/webalizer

Some people (like me) do not like SpamAssassin AWL (auto whitelisting - which is actually a score averaging system). To turn it off vi /etc/spamassassin/local.cf and insert use_auto_whitelist 0 (don't disable the plugin in v310.pre) then:
spamassassin --lint
amavisd-new reload


Every six months (or so), it would be a good idea to optimize your MySQL tables. I think it's a good idea (possibly even necessary) to stop MySQL activity during this period. This means you should shut down postfix and amavisd-new and possibly even disconnect the ethernet cable. There should be no MySQL maintenance scripts running (like /etc/cron.daily/ztrim-amavis which should start running at 06:25). Log in to mysql (as root) and optimize the tables:
USE amavis;
OPTIMIZE TABLE maddr, mailaddr, msg, msgowner, msgrcpt, msgs, policy, quarantine, users, wblist;
USE postfix;
OPTIMIZE TABLE admin, alias, domain, domain_admins, log, mailbox, vacation;
USE sa_bayes;
OPTIMIZE TABLE awl, bayes_expire, bayes_global_vars, bayes_seen, bayes_token, bayes_vars;


If you change your NIC card, udev will call your new NIC something other than eth0. In this case you need to
vi /etc/udev/rules.d/z25_persistent-net.rules and correct the situation. I reboot afterwards.

You may want to edit some of the banned rules in the $banned_filename_re section of /etc/amavis/conf.d/20-debian_defaults. To start with, I usually comment out the "# banned extension - basic" line and uncomment the four lines below it that comprise the "# banned ext - long" section.

In /etc/postfix/master.cf I add " -o smtpd_client_connection_count_limit=21" to the smtpd daemon to limit the number of simultaneous connections a single client can make (within one minute's time). The default is 50 (half of the default maxproc). This lowered limit is not appropriate if you get mail from another relay server - you would not want to lower the limit for a server you trust, but otherwise it may be a good idea. It may help with curtailing dictionary attacks:
smtp      inet  n       -       -       -       -       smtpd
    -o smtpd_client_connection_count_limit=21
Ok, start customizing. BTW, It would not surprise me if there are vulnerabilities in some of the pages of the various web interfaces.


Index Top

Gary V
mr88talent at yahoo dot com
DEC 6 2008