The DNS Bottleneck Hiding In Your Infra

Hey folks,

If you are using AWS and your Terraform/OpenTofu plans feel mysteriously slow, go look at how you're managing your DNS records. This bit one of our clients and it's really easy to miss.

The issue: Most teams throw their Route 53 record resources right in alongside the rest of their infra; this feels natural. But AWS puts a hard 5 requests/second limit on the Route 53 API for record requests. Once you've got Terraform doing data lookups across a bunch of hosted zones, you blow right through it.

Plans that should take seconds sit there grinding on the rate-limit backoff, and you're then multiplying that slowdown across every run.

We hit this with our client Cursor. Their DNS setup turned out to be one of the biggest bottlenecks slowing down how fast they could ship infra. It affected one of their critical root modules. Untangling that problem was one of the biggest speedups we found across the engagement.

Here’s how to spot it in your own infra:

• Turn on debug logging (TF_LOG=debug) on one of your slow plans.

• Check the logs and look for errors that read similar to Throttling: Rate exceeded (HTTP 400). These are easy to hit with Route 53 (but keep an eye out for other services as well).

• If you're on OpenTofu, flip on its OpenTelemetry tracing to see exactly where the time is going.

• If you hit the Route 53 rate limits and you see it in the logs, then pull your DNS records out into their own root module and statefile. This will prevent the rate limit from slowing down every plan.

This issue is cheap to look for and it could be the silent killer of your TF throughput.

May your Route 53 lookups never be rate limited,

Matt @ Masterpoint

PS If you want to chat about how we debug gnarly problems with TF, grab some time on my calendar here. If you’ve found this newsletter helpful, please forward it to a friend. Here’s the archive of all my newsletters to share on your company slack.