But let’s rewind a bit and look at how we got here. It all started with a hilariously tragic tale on Medium titled “How an Empty S3 Bucket Can Make Your AWS Bill Explode.” It’s the kind of story that starts with a routine day and ends with a cloud engineer crying into their coffee. You see, it wasn’t just a simple misconfiguration; it was a comedy of errors involving open-source tools, misdirected backups, and a hefty bill that could make even the most hardened DevOps pro break into a cold sweat.
The Tale of the Exploding Bucket
Imagine you create an empty, private AWS S3 bucket in a region of your choice. What will your AWS bill be the next morning? Our protagonist found out the hard way. While working on a proof of concept for a document indexing system, he created an S3 bucket and uploaded some files for testing. A couple of days later, he checked his AWS billing page, expecting everything to be well within the free-tier limits. Surprise! His bill was over $1,300, thanks to nearly 100,000,000 S3 PUT requests executed within just one day.
Where Were These Requests Coming From?
AWS doesn’t log requests against your S3 buckets by default, but after enabling CloudTrail logs, it became clear: thousands of write requests were bombarding his bucket from multiple accounts. It turned out that a popular open-source tool had a default configuration to store backups in S3, using the same name as our protagonist’s bucket. Every deployment of this tool with default values attempted to store its backups in his bucket!
S3 Charges for Unauthorized Requests
Adding insult to injury, AWS charges for unauthorized requests, even those that result in a 4xx error. So, if someone tries to write to your bucket and gets an AccessDenied error, you still get billed. And, if that weren’t enough, requests without a specified region default to us-east-1, which means additional costs for redirected requests.
The Security Angle
With all those misconfigured systems trying to back up their data into his S3 bucket, our hero decided to open it for public writes. Within less than 30 seconds, he collected over 10GB of data from various sources, revealing just how dangerous an innocent configuration oversight could be.
Lessons Learned
- Anyone Who Knows Your Bucket Name Can Ramp Up Your AWS Bill: There’s no foolproof way to protect your bucket from being accessed directly through the S3 API.
- Use Random Suffixes for Bucket Names: This reduces vulnerability to misconfigured systems and intentional attacks.
- Specify the AWS Region Explicitly: Avoid additional costs from S3 API redirects.
Aftermath
The open-source tool maintainers fixed the default configuration, AWS kindly canceled the massive bill (as a one-time exception), and our protagonist learned some valuable lessons.
AWS to the Rescue
Fast forward to today, AWS decided to ease the pain. As of now, Amazon S3 HTTP error codes will no longer cost you a penny. Whether it’s a 403, 404, or even the dreaded 500, those errors are now free of charge.
This change is a game-changer for businesses big and small. It means more predictable billing, fewer unpleasant surprises, and a bit more wiggle room to innovate and, yes, occasionally mess up without financial penalty. This is AWS saying, “Hey, we get it. Sometimes things go wrong. Let’s not add to your troubles.”
So, here’s to AWS for making life a bit easier and a lot less expensive for us all. Let’s raise a virtual toast to no more paying for HTTP error codes. Next time you see a 404, instead of a grimace, you can afford a little smile. After all, it’s on the house now.
Happy (error-free) clouding, everyone! 🌥️
13/05/ 2024
Blog
