How UserVoice Solved Their Incoming Email Problem

This post was written by Jonathan Novak, Head of Engineering at UserVoice, where he just celebrated his 3rd year at the company. UserVoice is a complete solution for managing user engagement, feedback, helpdesk requests, and knowledge bases. Over 120,000+ organizations in over 170 countries use it to listen to their users’ voices.

About two years ago, we ran UserVoice’s ticketing system on our own Postfix servers. We’ve since swapped to Mailgun for both incoming mail and outgoing mail, and are really happy about it. Here’s our story and problems that we dealt with in regards to incoming email.

Incoming Mail Before Mailgun

Our initial ticketing system used Postfix to handle incoming (and outgoing) email. Postfix is really solid software, but it’s a bit dated, and honestly, the documentation for it is a bit obtuse. This is compounded by the fact that we weren’t setting up Postfix in a ‘normal’ way. Most tutorials and examples show you how to configure discrete users and mailboxes within postfix, but we wanted to manage all of that within our Rails app.

After a lot of dorking around, we finally got it working. We got a shell script to execute for each incoming email, which added the email Redis via a Resque job.

Within the resque job, we then parsed the email using TMail. One issue we ran into was with encodings. Our Rails app is designed to work with UTF-8 and Ruby 1.9.3 encoding support, but TMail has legacy encoding support. We tried the new Mail gem, but it failed to parse a mail if it had anything ‘funny’ about it. Email is like the wild west: anything goes. Your parser should be extremely robust, and the Mail gem (~2 years ago) was not.

After we parse the email, we then need to parse the body to separate out the new portions of the email from the quoted thread/history of the email. Our first version was relatively naive, and basically just split on “On xx/xx/xxxx, Some One replied:”. We then considered the top part of that the new portion of the message. Our system did not handle ‘inline’ replies very well.

If there was ever an issue and we needed to find diagnostics, we had to ssh into our postfix box and grep our logs. This was ok, but our less-technical support staff couldn’t autonomously solve issues.

In terms of redundancy or scaling, we had one box. Luckily it never went down, and luckily we didn’t need to scale.

After Mailgun

Mailgun addresses all of these issues.

First, we no longer need to wrestle with Postfix. By configuring a simple “catch_all” rule in Mailgun, all emails addressed to *@*.uservoice.com are posted to our servers via HTTP.

These emails come pre-parsed. Everything posted to our servers arrives in UTF-8. That alone is worth its weight in gold.

Beyond that, Mailgun automatically separates out the new portions of the message from the thread. They’re included as a parameter called “stripped-text” in the webhook, which is exactly what we need when we show a message in UserVoice. We no longer have to maintain a messy collection of regular expressions. Plus, it automatically handles inline replies.

Our support team has full searchable logs of all incoming and outgoing messages. No more grepping around for messages, and the interface Mailgun provides is completely accessible to our Support Team.

Finally, we no longer have to worry about scaling our servers, failover, and things of that nature. Mailgun handles all of that for us.

The best part of Mailgun

Mailgun’s technical team and their support are top notch. They’re always responsive and helpful. They’ve come through and implemented key features when we needed them to close certain customers. They proactively let us know about problems (like spamming customers) and possible configuration issues. They also help us track down issues when our customers are struggling to forward email. As a company that is completely focused on making customer support better, we like that!