IaC Tooling: Build vs. Buy

Hey folks,

Building IaC tooling yourself is a super common pitfall that we see clients run into.

If clients have infrastructure as code (IaC), they might have written Jenkins pipelines or GitHub Action pipelines around that code because they thought "Why is this different than my application code? Why can't I just write a pipeline that makes the IaC code apply?"

I think a lot of people don't agree with this, but infrastructure as code from an operational perspective is complicated. Even though it seems deceptively simple, like it is just:

• give me a plan

• I'll review it

• then you apply

It gets complicated because if you're doing infrastructure as code at scale that doesn't fall over, then you need many root modules. You have sets of code that you apply and each has their own state file. You also need security checks and some level of policy.

You also need flexibility: "That's the dev environment, so let that autodeploy. In the prod environment, I really want to make sure that the security team confirms these changes look good".

So there are all these complexities in building an IaC pipeline yourself.

When we see clients who have done that, they're either:

• ignoring these complexities or

• they're going so much slower than they really think they are

They're building their own pipelines, they're adding functionality, and they're having an engineer build their own GitHub Action pipelines for all this infrastructure code. This effort is burning a hole in their pocket because that engineer, who's writing these pipelines all the time and maintaining them, is reinventing the wheel. They're just building a worse wheel.

So we are big fans of the various vendors that are out there. Even if you're going to use Terraform Cloud, that's better than building your own thing! (Even though I think Terraform Cloud's pricing is so outrageous that we wrote an article about migrating off of it.)

Beyond Terraform Cloud, there's a lot of good tooling out there. If you want to go the open-source route, there is Atlantis.

There is Terrateam, they have a light version of their product that's open source. It's written in OCaml. (I know. I think I'll always give those guys a little bit of flack for writing their tool in OCaml.)

But we're partnered with Spacelift for a reason. We really love that product. It has OPA wired everywhere. Every time we roll it out to a client, they eventually get to the point where they're just like, "Ah, this is really amazing. We are really shipping things quickly. We have the full control that we need and now we’re moving fast."

But there are plenty of good vendors out there. If you're in that decision space of "Do we build our own pipelines in Jenkins or do we go and talk to a vendor?", evaluate a few.

Sometimes people focus on big invoices or the day 1 experience. But it is the day 200, the day 300 experience that I think you need to live through to see how big of a maintenance burden homegrown IaC tooling actually is.

If you go with round numbers, a US-based software or platform engineer earns $100,000 in salary. If they're spending 50% of their year building pipelines for 10 teams, that's $50,000 they're spending (not counting other costs such as benefits or opportunity costs). They're building custom code. They're building a custom thing for this organization that needs to be maintained and updated.

If this engineer leaves, somebody else needs to come in and understand all that custom code. They still need to continue to maintain it.

The ROI does not usually match up.

When we've talked to really large organizations, I've said "Hey, if you want to put two full-time people on maintaining pipelines for your organization, they might be able to build something that's good enough on GitHub Actions." But you're talking about hundreds of thousands of dollars there.

It's smarter for you to adopt the vendor tool. They:

• are built and functional today

• enforce good practices

• give you the ability to immediately wire in security checks and role-based access control out the wazoo

• offer a good framework to extend

People don't want to hear that sometimes, but that's the truth. We've seen it across many clients and it's reality. Sometimes it's really smart to spend money with a vendor. IaC tooling is one of those times.

May your engineers never re-invent a wheel, poorly,

Matt @ Masterpoint

PS If you want to chat about how to evaluate a vendor for a possible buy decision, grab some time on my calendar here. If you’ve found this newsletter helpful, please forward it to a friend. Or if you want to share on your company slack, here’s the archive of all my newsletters.