Tuesday, July 19, 2005

More stupidity about spam filtering

I am not a lawyer and don't play one on TV. I do, however, know a lot about deploying large scale email solutions for ISPs. Therefore I know a lot about spam prevention at large providers, like Comcast. As a disclaimer, I do not know all the details about how Comcast is filtering at the moment. I know a fair bit and I won't reveal the details that I do know because the less the bad guys know about a particular provider the better.

The Brad Blog apparently knows next to nothing about spam filtering. I suspect I would have nothing but disdain for the content from afterdowningstreet.com that is/was getting filtered but I have seen overreactions like this from groups that I support as well. This is not a first amendment issue. Trust me, the big carriers have better things to worry about than political speech that they may or may not agree with.

The BRAD BLOG has learned that cable company and Internet service provider, Comcast Corporation has been automatically deleting email sent to Comcast customers with the text "www.afterdowningstreet.org" in the body of the email.

So far so good here. I did some checking and this appears to be accurate.

The discovery that email was being secretly filtered was made after an investigation conducted by ADS co-founder, David Swanson who reported that many coalition members did not seem to be receiving email alerts and others messages being sent by the group.

Bold emphasis mine and where the wheels start to come off.

In a statement released by People-Link.org, the Internet host for the AfterDowningStreet.org domain, the directors of the "progressive" firm charge that the filtering is politically motivated and both they and Swanson have requested people contact Comcast to complain. (Contact information at the bottom of this article.)...

The wheels are off. It's a conspiracy!!!!! No it isn't. It is a mistake in a filter somewhere. The filters are modified many times per day. Some of it is automated, some of it is manual based on statistical tracking. There are a lot of possibilities. There is some VERY small possibility that one right wing wacko with the right access did it but if he did he risked his job for it.

While it is not uncommon for Internet Service Providers, or ISP's, to filter out messages sent to their users by domains which are known to send large amounts of unrequested junkmail, or "spam", Comcast acknowledged to Swanson that AfterDowningStreet.org was not on their list of domains alleged to have sent such email.

And your point would be. In the article itself they point out that changing the referral URL in the body of the message was sufficient to get the mail through. ERGO... it is being hit by a body string match filter. These filters are often created automatically by scanning mail as it comes through and looking for things like identical referral URLs. All large ISPs now use a combination of IP and domain blocking, volume control or throttling, pattern matching in the subject line, sender id matching, body text pattern matching and a host of other technical methods to block spams and frustrate spammers. This one is clearly a body pattern match rule.

My best guess: There were a large number of emails being sent out in a wave that contained the same body. Some automated filter creation program detected the pattern and created a filter which blocked the content. Believe it or not this kind of thing happens pretty frequently. Remember that in excess of 60% of all messages being sent to any large US ISP is spam. In many cases that number exceeds 90%. There is no such thing as a spam filtering system which is 90% or more effective that can guarantee no false positives. We all wish there were, but there aren't.

Happy emailing and please double check before assuming the service providers do anything that is politically motivated unless they are getting paid to do it (advertising, etc). Their only politics is making money.


Post a Comment

<< Home