Inside terraform-provider-archive: A Memory Pattern From 2016 That Scales With Your Lambdas

Wait 5 sec.

A CI Runner That Shouldn't Have DiedIf you deploy AWS Lambdas through Terraform, you almost certainly use archive_file. With enough lambdas, a single terraform apply can kill the CI runner with OOM. The trickiest part is that you will not see any errors in Terraform output and have no clue what just happened.I noticed this when my lambdas started failing — every first terraform apply after a routine change. SIGKILL from the kernel OOM killer and nothing in Terraform logs. The strange part is that reapply sometimes worked — not always on the first try, but eventually it went through. I've named the ticket "Flaky CI," and two weeks of investigation was focused on the CI itself: runner memory, parallel jobs, Docker leaks. terraform apply was the last suspect — from my perspective, there was no way or reason for it to consume so much memory.