When A Client Built Their Infra Automation Pipelines... In Jenkins

Hey folks,

A couple years ago, we were brought in to help a client standardize their multi-region AWS setup using infrastructure as code (IaC) and Terraform.

They had one region managed via Terraform. The others? They were ClickOps chaos: EC2 snowflakes with managed with SSH and hand-tuned configs. Oof.

We got to work making their infrastructure consistent across regions. We distilled what was actually different and replaced their ClickOps processes with something sustainable.

But we hit major issues with their custom infrastructure automation pipelines, written in Jenkins:

1. Upgrading Terraform? Broke.

2. Deploying to a new region? Broke.

3. Adding policy-as-code for important approvals? Broke.

They insisted on managing the Jenkins pipelines in-house. Our suggestions about better, more cost effective options fell on deaf ears.

Another team owned the pipelines. Changes to the pipelines took weeks and we had a tight timeline. What should’ve been straightforward infra work turned into a mess because this organization thought that their investment in their homegrown infra pipelines was the only way.

A classic sunk cost fallacy!

We sadly couldn't change their minds, and as a result we had to work harder to ship slower. Double oof.

It was a cultural problem for sure, but sometimes technology decisions, like "we must build custom pipelines for everything", further enhance bad cultural decisions.

They just weren't willing to invest enough into their tooling: either through putting more engineering work into their custom pipelines OR buying a better tool like a TACOS and managing their automation using it (our recommendation).

The main takeaway here? If your automation is so fragile and maintenance heavy that other teams can't rely on it, you don’t have a platform. You have a liability.

Unfortunately, we see this as a repeated pattern in the DevOps and Platform space. People think "Oh we have custom pipelines for our application, why wouldn't have the same for our IaC?".

The reality is that building a great infra pipeline is a harder problem to solve then people think. So they try and then they get it wrong. The amount of investment needed to build a good IaC automation platform to ship infra at scale on top of Jenkins (or any CI/CD system) is extremely high. It should be avoided.

Unfortunately many keep digging even after they discover they are in a hole. It's rough. Avoid it if you can.

On the other hand, if you are building your own pipelines for IaC, I’d love to hear why. Because honestly, I haven’t seen it go well at scale once!

May you never build your own IaC pipelines,

Matt @ Masterpoint

PS If you want to chat about how to move your infra pipes off of Jenkins or educate me on how you built your own scalable IaC pipeline, grab some time on my calendar here. Or if you’ve found this newsletter helpful, please forward it to a friend or share it in your company’s slack channel.