GitOps, IaC, And FRD (fear of resource deletion)

Hey folks,

Have you run across someone who doesn’t want to use GitOps and TF to manage their infrastructure because they are worried about automation inadvertently deleting business critical resources?

I have definitely have thoughts on this.

Where we see this come up a lot is people with their RDS databases (or other similar managed databases in other clouds).

People wonder:

• “How can we manage RDS using TF?”

• “How can we manage our whole data tier, which is critical to the operations of our business, using GitOps”?

• “How can we manage that critical functionality via infrastructure as code when it could so easily be destroyed?”

People get concerned about that!

I call it FRD: fear of resource deletion.

Luckily, there are protections at both the API and provider layer that make us at Masterpoint feel very comfortable managing data infrastructure using IaC.

For example, for RDS databases in AWS you have the ability to turn on delete protection. For S3 you have the ability to turn on Object Lock. Azure and Google have similar options.

With these options, I don't remember the exact steps taken to delete a resource, but you have to go through a special process to actually remove them. You have to be explicit, and that means if something gets deleted then it was intentional.

It's difficult to do from IaC and TF. Even from the console, you need to be very explicit and check a box that says something like "yes, this database needs to go, here is the name of the database so that you know that I’m sure."

This configuration should be turned on for all production data storage resources. When it is, and for us it always is, we feel comfortable using IaC to manage critical business infrastructure.

That’s data; now let’s talk application code.

When it comes to applications resources in production, like a cluster, our thinking changes a bit. With application code it's like, "hey, if we can delete it fast, we can bring it up pretty fast too."

If you are wary, I understand but would advise that you can bring application code back quickly if you're doing things right. That’s the beauty of immutable infrastructure and treating infra as cattle and not pets.

That’s the way Masterpoint thinks about the destructive risks of IaC.

May your deletion protection always be enabled,

Matt @ Masterpoint

PS Bought my tickets to AWS re:Invent. The first talk I registered for was DOP316: Unleashing AWS agility with Terraform, a chalk talk with Tyler Lynch. Looking forward to asking him some good questions. Reply to this email if you’ll be at re:Invent and want to catch up!