Sending Emails by Queue in Magento 1.9.1

January 17, 2015

My last blog post was over 6 months ago, so I think I’m technically way past due for a new post. In this post I wanted to take a tour through the new functionality in Magento 1.9.1 with regards to sending emails from your store.

The astute among you will have noticed there were some changes to email in Magento 1.9.1 from the Magento 1.9.1 release notes. Two things jump out as requiring further investigation:

  1. “all Magento e-mails (including order confirmation and transactional) are now queued and sent according to your configured cron schedule”
  2. “boasts responsive default email templates so customers can read your order confirmation emails and newsletters on any device”

The first change sounds worrying to me, after years of emails from merchants struggling to get their Magento store to send emails – anything which makes it _harder_ seems like a bad idea to me. However, we’ll look through the changes and see what’s new.

The second change sounds really good – I was tinkering on a responsive email extension myself, so it’s pleasing to see it has already become core functionality. I’ll take a look through the template changes, and how they affect your store in the second part of this blog post series.

Using cron to send Magento Emails

Magento email sending all boils down to the Template class Mage_Core_Model_Email_Template, you can see below the change for queuing has been to introduce a check for an available queue in the template class, and if available to enqueue the message with all of it’s data and return immediately.

if ($this->hasQueue() && $this->getQueue() instanceof Mage_Core_Model_Email_Queue) {
    /** @var $emailQueue Mage_Core_Model_Email_Queue */
    $emailQueue = $this->getQueue();
    $emailQueue->setMessageBody($text);
    $emailQueue->setMessageParameters(array(
            'subject'           => $subject,
            'return_path_email' => $returnPathEmail,
            'is_plain'          => $this->isPlain(),
            'from_email'        => $this->getSenderEmail(),
            'from_name'         => $this->getSenderName(),
            'reply_to'          => $this->getMail()->getReplyTo(),
            'return_to'         => $this->getMail()->getReturnPath(),
        ))
        ->addRecipients($emails, $names, Mage_Core_Model_Email_Queue::EMAIL_TYPE_TO)
        ->addRecipients($this->_bccEmails, array(), Mage_Core_Model_Email_Queue::EMAIL_TYPE_BCC);
    $emailQueue->addMessageToQueue();
 
    return true;
}

If there is no queue, the send() function proceeds as it used to, builds the mail object and sends it. Extensions like SMTP Pro and MageSend that hook in to the Template class to change the mail transport, will inject their transport at this point.

Slight Tangent: Are all emails actually being queued?
You might be thinking, as I was at this point, Under what circumstances will there be no queue? Good question. According to the release notes, “all Magento e-mails” are now queued, so I expect that the queue must be being set somewhere for all outbound emails. Let’s dig into this and find out.

Firstly, where the queue object is instantiated with a getModel() or getSingleton() call it would be referenced by it’s handle core/email_queue. So we grep for that handle, to find the places where it is instantiated. Oddly, I only find two places where the queue is created and set on the Template object. Both are in Mage_Sales_Model_Order, functions queueNewOrderEmail() and queueOrderUpdateEmail(). So my gut feeling at this stage is I have missed some obvious place where the queue is being set for *all* other emails (ignoring the obvious question, if it’s being set for all emails, why set it explicitly here?).

So, let’s try and find calls to the setQueue() method, assuming the core developers have not used a setData() – they must have called setQueue somewhere, to ensure the queue object is being used. Strangely I only find reference to Template.php and Mailer.php, both setting the queue based on what was passed to it.

OK, so I’m not sure all emails are being queued, let’s ask the community to double check me and continue with our analysis.

When Emails are queued…
If the message is queued, then the email data is saved in the Magento database where it will be later fetched and an email message will be reconstructed from the data and sent. The data is stored in two new tables, one for the email data and one for the recipients:

<email_queue>
    <table>core_email_queue</table>
</email_queue>
<email_recipients>
    <table>core_email_queue_recipients</table>
</email_recipients>

When and how is the queue cleared? For that Magento have added two new cron jobs:

<core_email_queue_send_all>
    <schedule><cron_expr>*/1 * * * *</cron_expr></schedule>
    <run><model>core/email_queue::send</model></run>
</core_email_queue_send_all>
<core_email_queue_clean_up>
    <schedule><cron_expr>0 0 * * *</cron_expr></schedule>
    <run><model>core/email_queue::cleanQueue</model></run>
</core_email_queue_clean_up>

That means the send() function on the queue will be called every minute (why */1 and not just * I wonder?) and the clean up once per day at midnight 00:00.

The send() function will grab a chunk of size MESSAGES_LIMIT_PER_CRON_RUN (default 100) emails data out of the DB table and send the emails one by one that have been queued since the last run. The send itself is handled in the same way the Template class does in the pre 1.9.1 code.

$collection = Mage::getModel('core/email_queue')->getCollection()
    ->addOnlyForSendingFilter()
    ->setPageSize(self::MESSAGES_LIMIT_PER_CRON_RUN)
    ->setCurPage(1)
    ->load();

A couple of things I wish the core team had done differently here. 1) to allow configuration of the rate of email sending from either the XML file or via the admin panel. It’s important for most 3rd party email sending services to control the rate of email sending to avoid being flagged as spam.

And 2) to re-use the email sending functionality from the Template class – instead of basically copy-pasting the Template class sending code in the Queue. This has the effect that extensions like SMTP Pro or MageSend – or any that try to modify the passage of outbound emails – will now need an extra override, one that is totally unrelated to actual sending, just to set the transport before sending.

Lastly, 3) the email queue table could also act as an email log (if all emails are sent using it) and even allow users to easily resend the emails. All of these are items I will try to address as 3rd party functionality in the next releases of SMTP Pro and MageSend.

I also think it’s important to test whether cron is running as part of the self test/setup – as it’s one of the areas of Magento that often is not handled correctly. I wonder why Magento just doesn’t do an auto-cron on front end page views like WordPress, but that can be a post for another time.

I hope this little tour through the new functionality helps you to understand the new email sending queue functionality in Magento 1.9.1. In the next post in this series I’ll take an in-depth look at the new responsive email templates shipped as part of Magento 1.9.1 – how they work, how you can customize them and things I’d like to see improved/changed (if any).

Ashley

Posts

Hi! I'm Ashley Schroder, a Software Engineer from New Zealand - this is a collection of notes on my experiences with Ecommerce Web Development, particularly Magento Development.

Are Your Customers Getting The Magento Emails You Send?

MageSend Magento Email sending with Amazon SES

I have made a premium Magento Extension called MageSend. MageSend makes it simple and easy to send using Amazon's highly reliable SES email service.

The extension is $99, and comes with a complete money back guarantee, please check it out, I am confident it will resolve any email sending problems with your Magento store.

6 responses to Sending Emails by Queue in Magento 1.9.1

  1. I have updated Magento to 1.9.1 a few days back and I cannot find any tables beginning with core_email. This is already leading to issues (for example with paypal).

    From what I remember after skimming through your nice article, I suspect my database is messed up? Can anyone confirm this and tell me there SHOULD in fact be tables like that?

    Or even better, can someone tell me where I find the structure of those tables so that I can recreate them on my own in order to fix the issue the paypal module is running into? Thanks a lot in advance 😉

  2. So it’s basically using the database as a job queue, which from a scaling perspective is inviting trouble from all sorts of angles. From a performance point of view, this just screams out for being replaced by an external messaging or e-mail service. Shops slowing down due to non-essential tasks is never a good thing.

    AWS fans could consider SQS for the queueing and then SES for the sending. Perhaps an idea for MageSend to incorporate SQS job queueing so other systems can also use SES via a common system-in-the-middle.

  3. A recent problem i’m having with this new way of sending emails is that the scheduler only runs for either 10 / 20 mins (jobs) before stopping abruptly…

    If i truncate the cron_schedule table it starts again and merrily continues with its job.
    It will schedule ahead as per admin settings, but never regenerate past the initial generation of jobs.

    any ideas..?

  4. Hello Ashley,

    Thank you so much for such useful notes, reading this i do understand to some extent about how to handle new email setup in magento 1.9.1

    I found the database table as you mentioned all the emails are queued.. after this point i couldn’t follow the instruction about the magento two new cron jobs (core_email_queue_send_all, core_email_queue_clean_up) are this cron jobs already exists in cron tab of some xml file? or do we need to create this entry in some .xml file?

    If this were cron job’s already exists in some file, could you please share with us the location of the file please… ? so i can create a “command” to run this job from my hostgator cronjob manager…

    P.S: i also tested by removing the queue process from the template.php file inorder to send an instant email when order is placed by the customer and it worked well..:), but i don’t want to deviate from the magento way of sending new order email…

    Thank you so much and appreciate your valuable inputs..

    Helmy

  5. hello sir,

    i am using magento 1.9.0.1 version to develop a store. and i want to remove the sku from order email. but after trying all metheds which are mentioned in google search like (sales.xml and editing block and template ) .i am unable to remove it form email.
    please help……

  6. Which queuing mechanism is being used here? I personally use Redis based queuing which works rock solid but i am definitely going to to try this.