This article is for server administration, not for single clients. This is because as a single client, most of the damage is done before you can see enough of the mail to filter it, and there just is no effective solution. The solution needs to be at the mail server, so spam is not just thrown away, but the spammer can receive an SMTP rejection of some sort.
RBLs vs Content Verification.
So do you block known bad IPs or do you check the contents of the mail for signatures that indicate spam. I've personally never been a fan of the false dichotomy so I won't linger on this for very long, but the answer is clearly "both". But you check the IP first. Because the IP address can be checked against a string of reverse DNS RBLs at the connection level, way before the message arrives. Hell, if everything is working nicely, before you even get the header. HELO, Shaadup! EHLO, Shaadup! easy.
Then, if the email doesn't flag anything on the RBLs, look at the header, and then look at the message.
RBLs and Content checking are just the two most common methods for checking spam. They are, for the most part, quite easy to understand. RBLs are very simple, Content checking has several variants and several commercial interpretations. Some newer systems are beginning to include image content checking as well, but this is likely to take a while to filter through to those of us not running supercomputers for mail servers. But searching for more effective anti-spam solutions has opened my eyes to many other options. Here are a few of the tricks I've come across, or use and think need more air-time.
Header Checking:
Just checking the formatting of smtp headers excludes a nice chunk of spam. Sad but true. Spammers often go to great lengths to hide their IPs, and as such include lots of spoofed header which is usually not quite formatted correctly.
Check that the supposed sender's email domain exists. ie, if the mail comes from spammer@spam.co.uk, check that spam.co.uk actually has an MX record. Sure, this isn't going to kill off 100% of your spam, but when you're looking at a multi-layered defense, this sort of additional check is useful. The secondary check is to check that the IP that sent the HELO(EHLO) has a reverse DNS A record. I personally have this one turned off, because too many legit people have a slightly screwy mail set-up, either because their ISP has stuffed them around or they just didn't know what they were doing. Potentially useful though.
One additional check I'd like to see, but haven't yet seen in any spam filter in the wild, would be checking that the MX record for the reply-to or from address match at least within the network address of the sending IP.
Delaying:
This is simply responding to all unknown incoming emails with a non-fatal error. When that particular email is retried, pass it through the delayer code. What this means is that unless the email is coming from a mail server with a queue, then the mail will probably just fail. No prizes for guessing what type of email that usually means. The problems arrise with the other sources of non-queued email. Web-forms often send mail direct with no queue. And some mail servers do not respond nicely to being delayed. Thankfully some people (http://projects.puremagic.com/greylisting/) have built lists of these misbehaving servers.
Penalty boxing:
One of my favourites. Any IP with more than a specified badness gets blocked. I weight most spammy type stuff very lowly to add to this badness, but weight bad email addresses fairly highly. What this does for me is fairly quickly penalize spammers that just try every possible combo of letters and numbers for usernames at a domain. You would be surprised how often that sort of thing happens. The penalty box can act like a local RBL, and reject connections before they become bandwidth. Happyness. One of the good points with Penalty boxing is that in most cases, spam will be stopped by one of either penalty boxing or delaying. If the spammer tries moving IPs and sending from different addresses, then the delaying will kill off the email, and if they sit still and retry to bypass the delayer, then the penalty box will shut them down.
The Other Side
Then there's the whitelisting. What most systems have is a manually entered list of email addresses to always accept email from. The system I use has an outgoing mail proxy that automatically adds all the outgoing emails' destination into the whitelist. This basically means that most (~60-80%) of legit email is whitelisted. I hear you thinking, "so what?". The good thing about this is it means you can be that little bit more agressive with non-whitelisted emails.