How Geogram Built A Free Group Email Service Using Yii For PHP With MySQL

This post was written by Jeff Reifman. Jeff is technology consultant and writer living in Seattle. He is also the CEO and lead developer of Geogram, a free group email service for local places. You can follow him at @reifman.

Geogram provides free group email service for your block, your neighborhood and your favorite places. Users discover groups based on their location (with a bit of HTML5 geolocation and Google Maps magic). Once they join or create groups on the Geogram website, most of their interaction with our service occurs via email.

In the past, building a scalable email service around processing and sending email was a huge undertaking. In addition to performance issues, managing your server reputation and spam require a great deal of time and specialized expertise (Geogram’s competitor has 55 employees right now). If you don’t think managing email is harder than it looks, read this (So, You’d Like to Send Some Email (through code)).

So in this tutorial, I’ll walk you step-by-step through how to build a similar service to send, receive, and track emails for hundreds or thousands of users. If you are already familiar with the basics of using Mailgun, feel try to skip ahead to the section that interests you most.

I. How Geogram is architected

II. Sending emails

III. Handling incoming emails

I. How Geogram is architected

Geogram uses the open source Yii Framework for PHP with MySQL. Yii is essentially an MVC framework like Ruby on Rails, but with all the simplicity and maturity of PHP. It’s fast, efficient and relatively straightforward. It also includes scaffolding/code generation, active record, transaction support, I18n localization, caching support et al. The documentation, community support and available plugins are also quite good.

In the past, it would have taken me a year to build something like Geogram and longer to properly mature it. Even though I was new to Yii, I was still was able to code Geogram in three months (including Mailgun integration) – almost entirely solo. I’m still maturing the service but Yii has been a huge part of my quick success. So, the integration examples provided use Yii, PHP and cURL. They should be understandable to most PHP developers but you may notice bits of Yii-specific code that’s new to you. The code examples hopefully are generic enough though that they should help you understand how you might integrate Mailgun into your own application. 

In addition to the excerpts below, I’ve created a public Github repository (geogram-mailgun-tutorial) with code samples from this tutorial for you to review in a more organized fashion. Keep in mind though, this tutorial does not provide you a full working codebase of your own list-based application. It’s just meant to help you understand how to work with Mailgun’s API, how you might approach common challenges to running email-based applications and to ease the process of getting started.

Configuring Mailgun

Before you begin, make sure your server has support for PHP cURL. On Ubuntu Linux, we do this:

sudo apt-get install php5-curl

Mailgun will provide you a secure API URL and API key in the control panel:

We place these in a PHP .ini configuration file outside of the Apache web directory:

mailgun_api_key="key-xx-xxxxxxxxxxxxxxxxxxxx" mailgun_api_url="https://api.mailgun.net/v2"

As part of its startup process, we ask Yii to load these keys into the Yii::app()->params array. Whenever we want to connect to Mailgun, we call this function:

public function configMailgun() { //set the project identifier based on GET input request variables $this->_api_key = Yii::app()->params['mailgun']['api_key']; $this->_api_url = Yii::app()->params['mailgun']['api_url']; }

We’ve also created a helper function for initiating the most common structure for preparing Mailgun cURL requests – (beware it varies for other API methods):

private function setup_curl($command = 'messages') { $ch = curl_init(); curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_BASIC); curl_setopt($ch, CURLOPT_USERPWD, 'api:'.Yii::app()->params['mailgun']['api_key']); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'POST'); curl_setopt($ch, CURLOPT_URL, Yii::app()->params['mailgun']['api_url'].'/yourdomain.com/'.$command); return $ch; } 

II. Sending emails

The following section outlines how to programmatically process and send application generated emails. Geogram allows users to send both group and private message which each require special functionality on the back end. Additionally, Geogram itself needs to send individual messages (e.g registrations), so I rewrote the PHP mail function to use Mailgun.

Delivering email to Geogram groups

When a user posts a message via the Geogram website to send to a group, there are a couple of things that we do in sequence which might seem unusual to you.

Rather than using the API to post every message to Mailgun in real time, the first thing we do is store the message to our own database. Not only do we know our database is always available but this also provides fast visual feedback for users.

As the number of messages passing between Geogram and Mailgun scales, having the ability to asynchronously manage posting and receiving messages will be critical.

Here’s a sample database schema migration for our Post table and an example of how Yii’s Active Record implementation saves form data to it.

We run two different background processes that operate on the sent messages. Yii supports running console daemons with full ActiveRecord support. Although, you can also use a CRON job or write a script in your favorite language e.g. Python, NodeJS.

Determining the Recipient list

The first process looks for pending messages and completes pre-processing. During this phase, we determine the exact recipient list.

Some group members have blocked users, others have requested to receive email via daily or weekly digests, some have asked to unsubscribe from a thread, while others have asked to receive no updates via email. Generating the recipient list requires some processing time – so it’s best done in the background. The final recipient list is stored in a second table.

This process takes pending messages in the Outbox and creates records in two related Outbound Mail tables. You can see an example of us processing messages from the initial outbox to the outbound messages table here.

Handing off messages to Mailgun for sending

The second background process manages the actual transfer of the messages from the Outbound Mail tables to Mailgun via cURL.

The lines above that build $rx_var are constructing a JSON string of Mailgun’s recipient variables. These allow us to send broadcast messages without individually listing each recipient on the To: line.

Here’s how we submit the messages to Mailgun for sending:

Sending private message

One of the most important values/features of Geogram is that we never share our users’ actual email address publicly. When we send messages, each sender is given a unique anonymized FROM address combined with the place name (or slug) they are sending to. e.g. Cynthia B. <u_51b3e170ef1e4+cedarplace@geogram.com>.

If a user replies to this address with !private in the first paragraph, we know the message is meant for private delivery to Cynthia B (user 51b3e170ef1e4). Without the !private command, we send the message to all recipients at cedarplace. Users can also send private messages to each other with just Cynthia B <u_51b3e170ef1e4@geogram.com> and we also allow private messaging through the web interface.

When we receive a request to send a private message, we do the following:

And, here’s the generic sendsimplemessage function we use to talk to Mailgun:

Sending individual messages

Sometimes you just want to send individual email messages immediately. When we send new users their activation link, we want to do this without any delay. Or, emails from our web-based contact form can just be forwarded to our support queue directly. Obviously, this is easy in Mailgun as well.

The Yii-User module extension uses the standard PHP mail function. We’ve replaced it with a version that sends via Mailgun:

and

III. Receiving incoming messages

One of Mailgun’s powerful value-added features is its processing and parsing of inbound email. Parsing inbound email successfully into its component parts requires sophisticated code written to specific standards (as well as properly handling inevitable non-standard messages). I’m much happier letting Mailgun write this code and have them post the parsed results to my website.

Furthermore, Mailgun’s spam filtering prevents my website from having to do the work of spam detection and filtering. For a group-oriented email service like Geogram which uses a catch-all mailbox, this is a great benefit. Their spam filtering prevents me from having to process a large volume of invalid requests.

Setting Up Your Route to forward emails to the app

Mailgun provides programmable routes for receiving different types of email:

For example, any email to support@geogram.com forwards to our Tender App inbox and any email to jeff@geogram.com can forward to my work address.

More importantly, we have a catch-all rule – any email to unknown mailboxes is forwarded to our website as a posted form. In this case, you set up your forwarding address as a URL for your production environment e.g.

http://yourdomain/messages/inbound

Mailgun will post the data from incoming messages to that URL.

Receiving and storing inbound posts

Again, for scalability reasons, when Mailgun posts inbound messages to Geogram, we first just store them unprocessed in the database. As mail traffic on Geogram increases, this point of contact must be always available and fast. We simply serialize the POSTed data from Mailgun and store it as a BLOB:

The verifyWebHook call ensures that the message was POSTed by Mailgun and is not malicious. Here is the code for verifying POSTs:

Another background process actually looks in our database and processes each message.

Processing inbound messages

Inbound messages to Geogram are often new messages or replies to existing messages. We post these to the outbox table and they get sent via the processing described earlier. We receive other kinds of messages which include email commands such as: !private (send a private reply), !leave (resign from this group/place) !favorite (mark this message as a favorite) !spam (mark this message as spam).

Our inbox processing is actually quite detailed but here are a few aspects of it to guide your own implementation.

First, we find all the unprocessed messages starting with the oldest first. This is the framework for our processing loop:

We’re unserializing the POSTed fields from the database, then verifying the sender. If the sender isn’t a registered user, we send them an error message.

Here’s how we look for commands at the top of the message body:

Here are the basics for how we match the incoming To: address to the known Places in our Geogram world. Basically, we’re matching the place part of the To: field to the slugs for Places in our database. We’re checking that the place exists and that the user is a member authorized to post:

Then, we just save the message to the database as if it had been posted from the website. It will get picked up by the background process that handles outbound messages.

Tracking reply threads

Replies to emails include a header with a unique message ID identifying the thread of the conversation. When we ask Mailgun to send a message, it returns this unique thread ID to us and we store in the database.

$result = $mg->send_broadcast_message($fromAddr,$message['subject'],$message['body'].$this->mail_footer.$place_url,$to_list,$rx_var,$i['post_id']); $mg->recordId($result,$i['id']);

Here’s where we record the ID in the database:

public function recordId($result,$id) { // update outbound_mail table with external id // returned from mailgun send $resp = json_decode($result); if (property_exists($resp,'id')) { OutboundMail::model()->updateAll(array( 'ext_id' => $resp->id ), 'id = '.$id ); }

Later, when we receive messages, we look to see if they are part of an earlier conversation. Here is some example code:

Advantages of Mailgun

I hope this post has given you an idea of how powerful Mailgun and its API are. In summary, here are a few reasons I’ve chosen Mailgun for Geogram:

• It provides a simple API that’s easy to integrate with common platforms

• It manages the complexity of inbound and outbound email for me

• It manages scaling most of the email-side of my application for me

• It manages my server’s email reputation and filters spam

• It provides super fast, reliable delivery – no more wondering if my messages will get through

• And finally, Mailgun provides outstanding technical support

Questions? Criticisms? Feedback?

I’ll be the first to admit that our code can be optimized greatly and that it’s not perfect – it’s a one person startup at this point. But, hopefully, you now have a bit more of a feel of how to work with Mailgun’s API via PHP.

If you have any questions, please feel free to post them in the comments and I’ll do my best to respond. You can also follow my blog or @reifman.

Kudos to the Mailgun & Rackspace folks (esp. Michael & Travis) who have been super helpful and supportive in the Geogram construction / launch phase.

What are you waiting for? Invite your neighbors to Geogram and then get coding your awesome Mailgun-integrated application.