Your Guide To Webhooks

You already know that email isn’t a send-and-forget thing. When you’re sending a message, you want to know how to track it and respond to any of its issues, right? After all, knowing how your emails are performing helps you maintain your sender reputation and deliverability rates. Webhooks – user-defined HTTP callbacks – are a very easy way for developers to monitor their email campaigns and build programs to handle bounces, unsubscribes, spam reports, and more in real-time. 

By pointing a webhook to their preferred CRM, product and marketing teams can see relevant engagement metrics quickly and easily, and make decisions based on up-to-the-minute info.  But, before we dive into use cases, we should cover the basics. So what’s what when it comes to webhooks? Let’s get started to understand what they are, why they’re used, and how to use them.

What is a webhook?

Webhooks are user-defined HTTP callbacks typically triggered by an event, such as a successful email delivery or bounce notification – put simply, webhooks send event data to another application after the event occurs. For example, when you provide your banking app with your direct deposit information and phone number, and they send you an SMS message telling you that you’ve just received a deposit, a webhook is what makes it possible for your bank to send that text message with relevant updates.

 When the event occurs, the source site makes an HTTP POST request to a URL the developer has configured to receive the webhook. Users can configure them to cause events on one site to invoke behavior on another. Then, that webhook’s data can be sent to the URL as a webhook payload in JSON or XML formatting.  The best use for email webhooks is to get information about how effectively messages are delivered to recipients, and once they receive it, how favorably they respond via opens and clicks.

Webhooks are useful for many reasons. First, they’re far more flexible than the typical email service provider (ESP) dashboard. With webhooks, users aren’t limited to the reports and analytics offered by their ESP. Instead, developers receive data directly and can report it, analyze it, respond to it; whatever they want. Second, webhooks are more efficient than API calls. Think of the difference between a phone call and a text. When a user sends a call to an API, they’re essentially asking their ESP for data. Why not just use a webhook?

A webhook is more like a text message that the system sends automatically as soon as a triggering event occurs. Since webhooks are based on HTTP POST, they are easy to use. What’s more, webhook scripts can be written in just about any scripting language developers prefer, including curl, Ruby, Python, PHP, Java, C# and Go. Once the webhook’s data has been captured, it can be stored in a database and used to gauge the effectiveness of email campaigns or augment recipient profiles.

So, how does a webhook work? Let’s start by exploring the webhook POST request, which can be encoded as application/x-www-form-urlencoded for most messages, and as multipart/form-data if there’s an attachment included with the message. The POST request method is designed to request that a web server accept the data enclosed in the request message’s body for storage.

Here’s an example of an HTTP POST made by Mailgun to a URI at Runescope. Note that the Content Type header is set as “application/x-www-form-urlencoded”:

The body of the message contains parameters stored as key-value pairs. (We’ll go into greater detail about the data that is posted in the section below.) Since the data is encoded, it will simply appear as gibberish. Here’s what the decoded body might typically look like:

Webhook events and parameters 

As explained above, webhooks are triggered by specific events. In the email realm, these events include opens, clicks, unsubscribe requests, and other events resulting from attempted or successful email delivery. Here’s a complete breakdown of the events that can trigger a webhook:

• Open: This event occurs every time a recipient opens a message. Open tracking is enabled by using the O:TRACKING or O:TRACKING-OPENS parameters when sending a message. 

• Click: This event tracks every time a recipient clicks on links in an email message. Enable click tracking by using the O:TRACKING or O:TRACKING-CLICKS parameters when sending a message. As with opens, the appropriate CNAME records must be included in the user’s DNS. 

• Unsubscribe: This event occurs when a recipient clicks on the “unsubscribe” link in a message. Spam Complaint: Not every ISP supports Feedback Loop (“FBL”) notifications for spam complaints, but developers should make sure that they get data from all of the ones that do. 

• Bounce: An email message is said to “bounce” if it is rejected by the recipient SMTP server. These are often classified as hard or soft bounces as follows: 

• Hard bounces (permanent failure): Recipient is not found, and the recipient email server specifies the recipient does not exist. The app should stop attempting delivery to invalid recipients after one hard bounce. 

• Soft bounces (temporary failure): Email is not delivered because of a temporary issue, such as a full inbox. Apps can programmatically respond to soft bounces by reattempting a set number of times before removing the recipient address from the list. 

• Failure: Failures consist of both hard bounces and soft bounces. Depending on its capabilities, an ESP may notify users through a webhook when a message is dropped (i.e., stop retries) for any of several reasons.

• Delivery: A successful delivery occurs when the recipient email server responds that it has accepted the message. Depending on the event, webhooks can deliver a variety of parameters to help identify and describe the message in question. This data can then be parsed via scripts for analysis. Common parameters include:

Event, Recipient, Sending Domain, Message Headers, and Recipient Identifying Details, such as country, region, city, device, email client and OS. Depending on the ESP, other parameters may include custom variables, tags and campaign names and authentication or user IDs, among others. In addition, some events offer more detail, such as a URL clicked; a reason for/description of a negative type of event or special codes providing specific event details.

Creating webhooks with Mailgun

You can find complete information and code about creating and deleting different types of webhooks in our documentation, but we’ve included a few common examples below. Check them out!

You can use our API to create a new webhook:

Or update one:

Alternatively, you can easily delete an existing webhook:

Additionally, for information about your webhooks, you can get details about any webhook URL:

Of course, if we made examples of all the ways you can use our API to create and optimize your webhooks, we’d be here all day. But, the point is that you can do pretty much whatever you want (except maybe magic).

Securing webhooks

A receiving URL must be public, so webhooks should be secured with a signature timestamp and token to create a hashmap. This hashmap uses an API key to verify that the data is coming from the developer’s ESP. Users should program their application to check this hashmap and compare it to that of the ESP, and then allow the POST to be made only if it matches.

To verify the webhook is originating from their ESP, users should link the timestamp and token values, encode the resulting string with the HMAC algorithm (using the ESP’s supplied API Key as a key and SHA256 digest mode), and compare the resulting hexdigest to the signature. Also, users can cache the token value locally and refuse to honor any other requests with the same token. This will prevent hackers from using the token to repeat or misdirect actions.

Another level of security would be to check the timestamp to confirm that the POST attempt has been made within a certain timeframe.

Below is a Python code sample used to verify a webhook signature:

How to use webhooks

Now that we’ve described the uses for webhooks, let’s talk about how to use them for your email program. After that, we’ll walk through a couple of useful examples.

1. Choose the desired data. The first decision developers must make as they plan out their email tracking and response is exactly what data they’ll be looking for. For example, if one is only interested in knowing when sent emails are bouncing, the user URL could perform a script on incoming POSTs to capture and save the email address in a local database. The same script could be augmented to capture recipient name, subject or any other parameter provided by the webhook.

2. Attach data to messages. When sending an email, some ESPs permit users to attach data to their messages by passing custom data to the API or SMTP endpoints. The data will be represented as a header within the email, and is typically formatted in JSON. This custom data would then be included in any webhook events related to the email containing it. Several such headers may be included and their values will be combined.

Example:

X-MAILGUN-VARIABLES: {"FIRST_NAME": "JOHN", "LAST_NAME": "SMITH"}

X-MAILGUN-VARIABLES: {"MY_MESSAGE_ID": 123}

To add this header to a message:

USING API: PASS THE FOLLOWING PARAMETER, "V:MY-CUSTOM-DATA" => "{"MY_MESSAGE_ID": 123}".

USING SMTP: ADD THE FOLLOWING HEADER TO THE EMAIL, "X-MAILGUN-VARIABLES: {"MY_MESSAGE_ID": 123}".

 Sometimes it’s helpful to categorize outgoing email traffic based on some criteria, perhaps separate sign-up emails from password recovery emails or from user comments. The ESP may permit tagging each outgoing message with a custom value. The user can then access deliverability statistics aggregated by these tags. To attach a tag to a message, supply one or more O:TAG to it.

Tagging Code Sample:

3. Set up the URL. In order to receive the data from a webhook, users must give their ESP a URL to deliver requests to. This means that they also need to set up the URL in their app, so it is accessible from the public web and different IP addresses (hence the need for security). The ESP’s webhooks will then POST data to the URL as application/x-www-form-urlencoded or multipart/form-data.

4. Create scripts to capture data. The final step is to add scripts to the URL that capture the data provided by the webhooks, and process it in any way the developer sees fit. See the use cases below for specific examples of these scripts.

Webhook use case: tracking email bounces

A use case for a webhook might be to capture an attachment and store the file locally. For example, the following code uses a combination of microframework for Python, Flask and Requests HTTP library. Here’s a quick Flask App to capture a file from a Bounce webhook, keep the native filename and store it locally on your webserver.

To see it in action, run your application and paste the URL (For example: http://yourdomainhere.com:100/webhook) into the Bounce webhook and click to “Test Webhook.” You’ll get a file named “message.mime” with:

And then you’ll have your webhook completed!

Conclusion

While there are several methods for accessing the data generated by email delivery, including ESP dashboards and API calls, cut yourself some slack and choose wisely – the most flexible and efficient way to collect email message data is to use webhooks. Rather than pulling data from their ESP, developers can receive continuous push data related to email in real time. Then, teams can use this real-time information to make decisions for current and future email campaigns across different web applications and web services. Cool, right? It’s a winning situation with no forgetting in sight.

For more easy, step-by-step info on how to use webhooks with Mailgun, check out our documentation on webhooks.