Mailgun’s Active Defense Against Log4j

On December 10, 2021 the world was rocked by a huge vulnerability that left millions of IT and security professionals scrambling. The vulnerability, dubbed Log4Shell, left assets vulnerable to a (ridiculously) simple exploit that led to Remote Code Execution (RCE). This blog post isn’t a technical article about the vulnerability itself– there are many great resources for that, just a Google search away. Instead, we want to share how Mailgun treated the vulnerability as an immediate incident and pivot into an active defense model within hours of recognizing our own exposure... 

Identification

As soon as we recognized the impact of this vulnerability, we mobilized our in-house incident response (IR) team. We generally handle new vulnerabilities through our standard vulnerability management process, which would have seen a high level of Common Vulnerabilities and Exposures (CVEs) patched within seven days or critical vulnerabilities patched within 48 hours. However, this was no “normal” vulnerability! 

Within two hours of notification, our IR team had reviewed our asset inventory and software bill of material. We found that no public assets for Mailgun were affected by this vulnerability. However, there were multiple internal assets that did have some aspects of the impacted software present. Some of these assets were protected due to the use of Java Security Manager. 

Another often missed component of such a critical vulnerability response is to understand your third-party exposure. During our initial identification phase, our compliance team kicked off a vendor review. Contacting all of your vendors is not a simple process, but is critical to understanding whether some of your third parties need to have limitations imposed against their access to data. 

After this initial review, the incident response team moved into a remediation and active defense phase. 

Remediation

Again, within hours of understanding our exposure, the engineering teams at Mailgun had rolled out the first patches for this vulnerability. Due to the critical nature of some of these assets, we had to work through a standard patching process, hitting our development and staging environments first. However, it is worth noting how critical those environments are. Many vulnerability management programs miss these “non-critical” environments in their efforts to achieve an agile stance. Attackers don’t care– they will still compromise those environments and abuse them to hurt you! 

Unfortunately, over the course of the next several days, more CVEs and exploits were let loose into the world, and we had to continually re-evaluate our assets and deploy new patches or mitigations. Our engineering teams were up to the challenge, though, and continued patching and reviewing reports throughout the following days. 

After the patching effort had begun and our engineering teams were engaged, the security team moved to active defense. 

Active Defense 

The next major phase in combating such a critical CVE is engaging in active defense. We consider active defense to consist of a few elements: preventions, detections, and honeypots. Luckily Mailgun’s chosen Web Application Firewall (WAF) began deploying detections for this exploit on day one. However, any security practitioner worth their salt will recognize that this is, and was not, good enough. The number of permutations against this vulnerability became nearly impossible to keep up with… 

This is where good logging and writing solid detection logic becomes critical in an incident. Mailgun invests heavily in logging requests sent to our infrastructure, and we were able to see just how targeted this vulnerability became. Overnight, we saw the string `${` skyrocket in our logs from attempts to exploit this vulnerability. Due to the scale of the attack, we recognized that basic IP blocking would not work. Coupled with the attack permutation and barrage of new CVEs, we recognized that we had to throw our efforts into completely mitigating this attack path and deploying ways to cut through the noise. On to the honeypots… 

The next stage in a mature active defense is deploying honeypots. The security community at large came through swimmingly here. Within days of the reported vulnerability, there were numerous published honeypots that could be deployed in minutes. After doing our due diligence against several of these, we chose to deploy a small bash monitoring script that could turn a server into a listening honeypot. Deployed within our internal networks and open to nearly all (inbound) servers in the environment, we would very quickly recognize if a bad actor was working inside of our private networks. 

Summary

At Mailgun, we embrace the idea of “when, not if”. We recognize that with this particular vulnerability, we were lucky not to have any exposed external assets or assets that would have been compromised through this exploit without internal access. Tomorrow, we may not be so lucky. But we can continue to prepare and work towards building more secure systems and better capabilities.