GHOST Mitigated And Our Patching Methodology

This was originally posted on January 29, 2015.

Mailgun has completed patching all of our infrastructure against the recently announced GHOST security vulnerability.

GHOST is a security vulnerability in the gethostbyname*() family of functions of GNU C Library (glibc). This particular vulnerability allows an attacker to potentially take over a server via either a local or remote exploit.

We have monitored our logs and have no reason to believe an attacker exploited our infrastructure, this was an entirely a preventive measure. No customer data was lost or affected.

We strongly recommend all customers review and patch their infrastructure accordingly as well.

Our patching methodology

To provide a little insight into our security posture here at Mailgun, I’d like to share the following information about how we deal with security vulnerabilities.

• We try to minimize the effects on our customers. That means we typically do rolling updates to ensure that a part of Mailgun is always running. This takes longer, but it allows us to minimize downtime.

• For security vulnerabilities that come with either a proof of concept (POC) and/or are remote exploits for services we run, we apply these patches immediately. We also check our logs to ensure that no one was able to successfully exploit the vulnerability before we were able to patch it.

• For security vulnerabilities that do not come with a POC and/or are local exploits, we typically patch our infrastructure according to our patching schedule. We do this because it takes longer to go from bug to exploit than our patching period.

We are pretty diligent about applying security updates whenever they become available. While it may seem like a low payoff way to protect server infrastructure, it actually raises the bar for the attacker. Keeping your infrastructure patched makes automated tools ineffective and requires the attacker find a bug in either our server configuration or our application itself which raises the bar for the sophistication of the attacker significantly.