Infrastructure is all the buzz these days. And there is a lot of buzzwords around it. Clusters, Kubernetes, serverless, on-premise, cloud, edge computing and tons of other more or less valuable terms. There are many different levels of progress along the axis of infrastructure, from those just becoming aware that infrastructure is something that exists to those that are already far in their journey and already have extra dope infrastructure. I have compiled a set of principles that will help you make sane decisions regarding how you think about infrastructure. To help making your infrastructure DOPIER.
Too much infrastructure is ad-hoc, or arbitrary. We are not quite sure how Server A is spec’ed, why it has 8 GB of RAM, or whether it has SSD or not. This is broken.
We can’t handle our uncertainty by ignoring it. This server has 2 CPUs because it is running Service X, and that is the requirement for performance.
Our infrastructure should be as it is, because we asked for it to be, not because of some arbitrary boxing of machines, or because it is what you use to order.
Most companies have some degree of virtualization, and it has never been easier to obtain a machine somewhere.
This extremely low barrier to entry makes it much easier to forget that the work of making that machine run properly has only just begun with ordering it. We forgot things like updating the operating system, or the service that is running there. Often we do not have the skill set to do so in a professional manner.
It is important that we are able to do operations work on our infrastructure, at a reasonable level of abstraction. For some teams this means having full control of the entire hardware stack, configuring networks, firewalls and kernels. For other teams this means that a machine magically is running somewhere and appears up-dated once in a while.
There is no one true level of abstraction. What matters is that the infrastructure capabilities are rightsourced so you can spend your effort on what provides value to your customers. I probably have an unhealthy attraction to higher levels of abstraction, so for me serverless computing does look very interesting.
Infrastructure is hard. It is hard to change, once deployed. But this is an anti-pattern. Infrastructure needs to be easy to change in sane ways. This does not mean that we should go completely anarchistic and wild west on our infrastructure. On the other hand this requires more guidance, better tools and is more difficult than simply having traditional IT operations. GitOps is one perspective to obtain plastic infrastructure, although it seems a bit buzzwordy. The goal is to make it easy to change in sane ways but difficult to break.
We are in a time of change, so make sure that you are able to adapt quickly, in a safe manner.
On the surface immutability conflicts with plasticity, but I use it on different levels. Infrastructure as a whole should be plastic, the atomic units that compose it should be immutable. In practice this means that if you are running a container somewhere, that container is uniquely versioned and traceable. There should be no doubt of exactly what code or version is running where.
Compose your plastic infrastructure from immutable atomic units!
“If it is not under version control, it does not exist”
Originally this quote was targeted at source code, but in these times I think we should extend it to also cover anything infrastructure. In our minds this should go from nice-to-have to a simply necessary fact. Not having things in version control is a risk that is both unnecessary and potentially catastrophic.
“If it is not monitored, it is not in production”
This is a corollary to the first quote, but it is the IT version of “Do you know where your child is, right now?”. Is your infrastructure happy, is it healthy? If you are unable to answer these questions easily you will at some point be caught off guard. Personally I prefer my surprises to be of the “You did not expect this party!” kind, rather than “You better call your spouse” kind. But if you prefer the latter, feel free not to monitor your services. The same way tests capture knowledge about your software your monitoring configurations captures knowledge about your IT landscape.
At this point it feels like beating a dead horse,but the point still holds. One-offs should no longer be a part of the vocabulary of an IT professional. We need to be able to reproduce our infrastructure. In production we need to be able to reproduce our infrastructure as a means of disaster recovery. In the steps before production readily available environments for stating and testing are key to the software delivery process. The value of having these environments, preferably scripted and automated, is hard to fathom before having lived it.
The best way to obtain a reproduceable infrastructure is to automate provisioning and have the scripts under version control.
Making your infrastructure DOPIER
So now we have established the properties of DOPIER infrastructure, how do we go about getting there? I think the most important thing is to start figuring out the world and being painfully honest. Enumerate the services that your team is running, but not really doing operations on. Figure out how to make those parts of your infrastructure DOPIER. That should start the process and the next steps will show themselves.