My cloud billing screw-up

Heya,

You know that one time that you screwed up and did something that ended up skyrocketing your cloud bill?

Yeah, we've all got that story. Here’s mine:

Back when I was a solo consultant, I was building a new greenfield AWS platform for a client.

Late on a Friday I pushed a code change to the customer's Dockerfile which changed the Entrypoint script. I got my code merged into main, everything built fine, the container followed the typical continuous deployment process. It was deployed to Dev.

It was late and I was done for the evening so I didn't validate the change. Whoops.

Come back on Monday to find my Entrypoint change had caused the container to fail on start within the ECS cluster. It would come up, fail to start, and then ECS would retry. And retry. And retry.

Over and over and over again all weekend.

The size of this Docker container was not massive, but due to the sheer number of times that container got pulled from within a private subnet to the open internet through a NAT Gateway, I had racked up close to $1000 in NAT Gateway data processing in just over a weekend.

Not my favorite conversation with my client when I had to explain that one to them.

Luckily, it ended up not being a big deal because I had an amazing client at the time and my ability to get AWS to give us a credit for those charges meant it didn't end up on our bill. Day saved!

What’s your cloud billing horror story?

May your cloud costs be controlled,

Matt @ Masterpoint

PS Speaking of cloud costs, running infra in test environments wastes money. We like efficiency and money, so we wrote a terraform module that removes everything in a test AWS account. Use carefully!