Devgun: Creating Development Environments With Kubernetes

When we first approached the problem of creating local development environments, we reached for common tools like vagrant. But as with most vagrant-built environments, build times are long. This means vagrant images have long lives and tend to drift overtime.

Our services and infrastructure change constantly, so the vagrant build was almost always broken. We wanted to avoid the traditional “new developer right of passage,” which easily entails days of work to get vagrant to successfully build a new development VM.

Ideally, new developers can be productive within hours of hire, not days or weeks.

All of this was compounded by the fact that we have a lot of micro services that need to run in the development environment and integrate seamlessly.

The goals of the project were simple:

• Setup and destroy a dev environment quickly and in a reproducible manner.

• Single step development environment setup and teardown.

• Start and stop services, including their dependencies, at will.

Evaluating orchestration platforms

Since our micro services were already running in containers, we knew we needed to choose an orchestration system.

Several devs on our team were interested in Mesos Marathon, so we tested that out first. Unfortunately, the memory and hardware requirements required to run marathon in a small local development environment were far too restrictive for our purposes. At the time we tested, minimesos was also unable to correctly network containers in Marathon to the local network – a major roadblock for our project. These issues made it clear that Mesos would be impractical for our needs.

Next, we evaluated Kubernetes and Minikube as we were attracted to the vibrant community around these projects. We found the memory footprint to be minimal, and networking worked perfectly. Most of the features that interested us worked out of the box, and getting our third-party applications running inside Minikube proved easier than expected.

So with Minikube as our chosen platform, we were able to move forward with the project quickly and spend most of our time developing what would become Devgun.

Building a CLI

To accomplish our project goals, we decided to build an all-in-one command-line interface written in Golang. We wanted a CLI that could be distributed as a single binary to our developers, so we created one called Devgun.

Devgun makes no assumptions about the development environment. It automatically downloads all the tools it requires, including Minikube, kubectl, and Mailgun-specific tools, into the /.devgun/bin directory. All you need to do is add your path after the initial setup has been completed. Installing everything under a single directory makes uninstall as simple as rm -rf ~/devgun.

We compiled all the kubernetes manifest files into the static binary using the go-bindata project, thus enabling a single file download and run to install a development environment.

Running Devgun looks like this:

In addition to creating a Kubernetes environment, Devgun also adds some network routes to the local box. For instance, on OS X, the following routes are added:

route -n add 172.17.0.0/16 <minikube ip> route -n add 10.0.0.0/24 <minikube ip>

This allows our developers to run and test applications on their local boxes with full connectivity to containers in the k8 cluster as if they were running in the cluster themselves. We also enable DNS lookup by utilizing the resolver functionality built into OS X. We create /etc/resolver/local with the content:

search cluster.local svc.cluster.local default.svc.cluster.local nameserver 172.17.0.3

Thus enabling DNS resolution of kubernetes services.

Managing custom and third-party services with a spec

Because Devgun has to manage a mix of custom and third-party services, we developed a custom yaml spec file we define for each service. The spec file looks something like this:

It defines a custom service called ‘scout’ which reads events from Kafka and records analytics about the events in Cassandra. The spec tells devgun which custom and third-party services it depends on and which hooks to run on different containers before starting the scout service.

The resources key references the kubernetes manifest files like config map, deployments, and service descriptions defined elsewhere in the spec file that devgun uses to start the containers in K8.

Where do we go from here… future work

When we designed Devgun, we started from the bottom up without consideration of the bigger development and deployment stories. Now that we’ve had time to use Devgun, we find ourselves wondering if we got the abstractions correct.

We started asking questions: What is the experience we want our developers to have when they sit down to create and deploy a new service? Where do the responsibilities of the developer end and operations begin? How much of operations can we automate and how much requires human interaction? How do we facilitate the passing of information between developer and operations?

We’ve started to define stories and personas that will shape how Devgun evolves going forward. In my next post, I’ll share some of our work and thoughts on how the developer experience could evolve as we continue to work with Kubernetes and Devgun. Be sure to subscribe to our blog so you don’t miss out.