Dropped Emails: Yahoo’s Gain?Friday, February 15th, 2008
As spam levels continue rising, all of the big players (Yahoo, Hotmail, AOL and Gmail) have been ramping up their spam detection and becoming more aggressive in their rejection of emails. Undoubtedly people have benefited from receiving fewer spam emails, but it seems at least in Yahoo’s case that valid emails are also dropped. Unbelievably, Yahoo stands to gain from overly-strict email verification.
Background: aggressive email verification
There are a number of competing email verification systems out there. Omar’s blog details what Yahoo, Google and Microsoft are up to. Yahoo instead has chosen to use a different verification system, called Domainkeys. The systems are not mutually exclusive, and none is supposed to be absolutely required for an email provider to accept an email as non-spam. In time one verification system may be accepted as the standard, but this is still early days for email verification systems.
The problem with delivery of emails to Yahoo-provided email seems to have begun for many around the end of October 2007. However, more recently Yahoo has continued to step up their spam defences. In this blog, Yahoo “Spam Czar”, Mark R., writes
“…at Yahoo! we’re tightening up on our spam controls in ‘08, and in fact we’ve already begun rolling out a significant new defense system (and it’s only January 4th!)”
This sounds like good news for Yahoo users: less spam is good news for everyone. But read on!
Results of the changes
My attention was drawn to Yahoo’s system changes when a client queried the failure of an email to arrive at Yahoo and BTinternet emails (Yahoo provides email service for BT). In recent months emails were consistently failing to arrive and what’s more, my client explained:
“…people we have been emailing with regularly for months are now saying they are no longer receiving our messages.”
By examining server logs, you can trace an email’s status: in this case and others I have now seen, it was clear that Yahoo’s servers were not accepting the email, but were deferring it:
2008-01-16 17:48:28.129627500 info msg 14206764: bytes 22222 from <firstname.lastname@example.org> qp 28514 uid 504
2008-01-16 17:48:28.129644500 starting delivery 40718: msg 14206764 to remote email@example.com
2008-01-16 17:48:28.381553500 delivery 40718: deferral: Connected_to_18.104.22.168_but_greeting_failed./Remote_host_said:_421_Message_from_(22.214.171.124)_temporarily_deferred_-_4.16.50._Please_refer_to_http://help.yahoo.com/help/us/mail/defer/defer-06.html/
2008-01-16 17:55:09.427873500 starting delivery 40932: msg 14206764 to remote firstname.lastname@example.org
2008-01-16 17:55:10.426180500 delivery 40932: deferral: 126.96.36.199_failed_after_I_sent_the_message./Remote_host_said:_421_Message_temporarily_deferred_-_4.16.51._Please_refer_to_http://help.yahoo.com/help/us/mail/defer/defer-06.html/
2008-01-16 18:15:08.176657500 starting delivery 41304: msg 14206764 to remote email@example.com
2008-01-16 18:15:12.860120500 delivery 41304: deferral: 188.8.131.52_failed_after_I_sent_the_message./Remote_host_said:_451_Message_temporarily_deferred_-_/
2008-01-16 18:48:29.365991500 starting delivery 41811: msg 14206764 to remote firstname.lastname@example.org
2008-01-16 18:48:29.599861500 delivery 41811: deferral: Connected_to_184.108.40.206_but_greeting_failed./Remote_host_said:_421_Message_from_(220.127.116.11)_temporarily_deferred_-_4.16.50._Please_refer_to_http://help.yahoo.com/help/us/mail/defer/defer-06.html/
2008-01-16 19:35:09.055033500 starting delivery 42669: msg 14206764 to remote email@example.com
2008-01-16 19:35:09.380619500 delivery 42669: deferral: Connected_to_18.104.22.168_but_greeting_failed./Remote_host_said:_421_Message_from_(22.214.171.124)_temporarily_deferred_-_4.16.50._Please_refer_to_http://help.yahoo.com/help/us/mail/defer/defer-06.html/
This deferral requires that the sending mail server resend the email, and this cycle continues usually until a point (often 48 hours later) when the sending server admits failure and bounces the email back to the sender, or the receiving server (in this case Yahoo) accepts the email.
Prioritization of emails
Strangely, in the light of the above, Yahoo deny that they use a process called “greylisting”, which involves rejecting an email and not accepting it until it is resent, on the assumption that spammers will not resend emails. The language they use to describe greylisting is this:
“[greylisting is] where an SMTP server will reject every message the first time it is attempted”
Well, maybe they do not reject every message - for example, Yahoo to Yahoo emails go through quickly - but there are consistent delays from some other domains. So consistent, that it seems like some messages from non-priority servers are always delayed. In fact one user was told by Yahoo that:
“…all shared hosting IPs are instantly flagged as suspect by them.”
Some have suggested that Yahoo servers have been overloaded with emails, which may have caused Yahoo to defer emails. However, if this were the case, then all emails would stand an equal chance of deferral. In reality, we see emails from some domains or mail servers that arrive almost instantly, whereas other emails are be placed in a cycle of deferrals. There is obviously some prioritization going on here Yahoo, why not admit to it?
Emails sent to spam folders
Even when these emails get through, usually hours or sometimes days later, the emails are often directed by Yahoo’s SpamGuard into spam folders. This is obviously no welcome for an email that has spent days trying to be accepted. These “false positives” might not be so frustrating if SpamGuard did not let through so many junk emails. Why SpamGuard singles out some legitimate emails for their spam folder and lets through obvious spam really needs to be addressed.
Still more worrying, there are whispers and rumours of Yahoo accepting emails, and then “silently dropping” them. This refers to the email being “lost” somewhere between Yahoo’s servers accepting it and the user: the email does not go to their inbox, but neither does it make it to their spam mail folder. It seems that Yahoo is simply deleting emails, without the email even entering the spam mail folder. Has anyone had any experience with this?
Who is affected?
- Obviously, businesses that have domain names that are not prioritized suffer. There is nothing surer than losing a customer when that customer thinks you have not replied to them.
- Yahoo users
- There are many domains that send relevant, useful email to Yahoo users. These people are not receiving emails and may know nothing about it.
How Yahoo gains by not delivering emails
Anyone would think that Yahoo’s business would also suffer if it did not perform the service of effective email delivery. However, because of their market position, as the biggest email provider, they are in a strong position. Businesses cannot afford to ignore users that have Yahoo email accounts. What is the best way to ensure your business email arrives at Yahoo email inboxes? Make sure your email is sent by Yahoo mail servers! So you can either open a Yahoo email account for your business or you can move web hosts to Yahoo’s web hosting! Sounds like good news for Yahoo…
This could be speculation at Machiavellian levels, and hopefully Yahoo do not think they would be able to go through with this without damaging their reputation. However, Microsoft’s recent bid to buy out Yahoo indicates that business at Yahoo is far from healthy. Hopefully, this is not a last desperate attempt to save business.
How to make sure your email gets through
So, if we do not want to put our proverbial tails between our legs, what can we do to avoid migrating to Yahoo web hosting? Not an easy one, I am afraid. However, here are some suggestions:
- Make sure you know how other mail servers see your email. There is a very good service to check what verification your email has as it is sent out, provided by Port25. All you have to do is send a test email addressed to firstname.lastname@example.org and you are sent back a reply indicating the verification that your mail server provides and also a spam-likelihood score based on the open source mail filter SpamAssassin. This can be very helpful in identifying any faults in your mail server verification and the analysis provided by SpamAssassin can also be helpful in seeing whether your emails are innately “spammy” or note. Amusingly, this is a domain that seems to suffer delays when sending to Yahoo mail servers: try sending an email from a Yahoo account.
- Send your email as plain text rather than HTML. Spam is less likely to be plain text and therefore spam filters are more likely to let plain text messages go through. You will lose some functionality in your emails though, such as no images, and font formatting.
- Watch the content of your emails and try and reduce their “spamminess”. This should help you at least with mail filtering, and keep your email out of the spam folder. As well as the Port25 email test above, you can test your email’s spamminess by emailing it to email@example.com (make sure you include the word “TEST” in the subject. Again, Yahoo users, be prepared to wait.
- Make sure you keep your domain out of any spam filter blacklist by ensuring any bulk emails have an option to leave the mailing list, and make it clear that you will not be passing on the recipient’s email address. Most people have now heard that spammers trick people into trying to opt out, but that they use this to identify real email addresses. So it is all too easy for someone to send your bulk newsletter to their spam folder to get rid of it, if they do not trust you to opt them out.
- This is more for server managers, but you could try and persuade your host to see if they can get get your domain whitelisted on Yahoo
- Again for server managers, install Yahoo’s email verification system, Domainkeys, although the trouble involved in adding this functionality to the server, added to the possibility that it will not make a difference, may be off-putting for most mail server managers.
- Installing one or more of the other competing email verification systems (e.g. SPF, CSV, DKIM, and SenderID is in some cases, easier than DomainKeys, and may help delivery, but if using DomainKeys will not guarantee it, then this is even less likely to ensure delivery.
- Goodmail Systems will certify your email (assuming you manage to pass their stringent accreditation checks), allowing you to receive a delivery confirmation:
CertifiedEmail, but your business headquarters will have to be in the US, Canada or UK.
- Perhaps one of the best solutions, is to use registered email, which gives you legal proof that your email has been sent, received and read. However, you will have to pay a price for this piece of mind (approximately $0.60 per email at time of writing): RPost. This will not guarantee your email will arrive, or that it will avoid being deferred. However, it will allow you to know for sure whether it has or has not arrived.
Any other ideas?
Having said all this, I admit to being a long-term satisfied Yahoo mail user, and I think the service beats Hotmail hands down and has held its own against Gmail. The real bad guys here are the spammers: if it were not for them…
Furthermore, it could be argued that a free email service owes nothing to its users, although a counterargument might centre around the monetization of this service through advertising. What is clear is that this recent turn of events is worrying, and may mean my dependence on my Yahoo email account goes into decline…