https://tty.neveragain.de/2023/09/21/aws-cannot-escape-ipv4.html neveragain.de teletype AWS: Cannot Escape IPv4 2023-09-21 --------------------------------------------------------------------- This is the first part of a blog series. Other pieces will be linked here when published. --------------------------------------------------------------------- Introduction AWS has announced that they will charge for public IPv4 addresses soon. Surprisingly, most reactions to this announcement were welcoming, as it's believed to bring a big push for IPv6 adoption. The AWS blog had this to say: This change ... is also intended to encourage you to be a bit more frugal with your use of public IPv4 addresses and to think about accelerating your adoption of IPv6 as a modernization and conservation measure. I will explain why this charge will not affect IPv6 adoption in any meaningful way. I will also show that it is pretty bold for AWS to use IPv6 in trying to soften the blow of this charge: AWS IPv6 support has huge gaps. AWS makes it impossible for customers to move on from IPv4. But first, a quick summary, and some costly anti-patterns to watch out for. What's Going to Change Every dedicated public AWS IPv4 address^1 will be charged with $0.005 per hour (almost $4 per month / more than $40 per year), starting February 2024. Some examples of affected services: * Elastic Load Balancers (one address per availability zone, at least two - and IPv4 cannot be turned off) * EC2 instances & Elastic IPs * ECS Fargate^2 tasks configured with public IPs * Global Accelerator IPs * Site-to-site VPN IPs * RDS (nine in ten customers have public IP on RDS by accident) * Managed NAT Gateways So for a public Elastic Load Balancer configured in three availability zones (as recommended), this is a base price increase of over 50%. For Managed NAT Gateway, this is a base price increase of ~10%. I didn't think those could become more expensive, but here we are. By the way, this is a good time to remember that a NAT Gateway does not provide zonal redundancy - a NAT Gateway in every availability zone is required when outbound connections are critical. All shared IPv4 addresses, like those used for Cloudfront distributions or API Gateway endpoints, are not affected. Public IPv4 addresses used by non-VPC Lambda functions aren't affected either. Wasteful Patterns I have observed two patterns that waste quite a lot of addresses and are heavily impacted by the upcoming charges. The first pattern is having multiple Load Balancers (per VPC); this is often the result of using several readily available Cloudformation templates / Terraform modules, or somehow using Kubernetes ingress controllers that create a Load Balancer for every service. This is fixed by not doing that! A single Load Balancer can handle many URLs and services. The second pattern is intentionally configuring public IPv4 on EC2 instances and Fargate tasks just to avoid Managed NAT Gateway (commendable until now!). Ironically, this will still be cheaper until hitting around ten public IPv4 addresses per availability zone. How to Check The usage data for public IPv4 addresses is already being accounted, so it's easy to check. Usage is currently listed with $0 but will start to cost $0.005 per hour starting in February 2024. * In Billing Dashboard > Bills, usage is displayed as PublicIPv4:InUseAddress in the Virtual Private Cloud section * In Cost Explorer, it's easy to see usage hours by filtering to only include PublicIPv4:InUseAddress; this is nice because Cost Explorer shows usage per day/hour and it's easy to group e.g. by member account * VPC > VPC IP Address Manager (at the very bottom) > Public IP insights is a nice way to get an overview of public IPv4 address usage; this part of IPAM is free, but it shows data only per account and region, and it's incomplete data - it doesn't capture transient address usage. Accelerating Your IPv6 Adoption So it does sound pretty obvious: IPv4 addresses will soon cost a few bucks, while IPv6 addresses are free of charge. Even better: There is no concept of private addresses in IPv6, which means farewell to the Managed NAT Gateway and its magnificent pricing. It seems like the perfect time to finally adopt IPv6. There are two approaches to adopting IPv6. Either by going dual-stack, which means configuring IPv4 and IPv6 addresses on all systems. Or by going IPv6-only internally, with IPv4 connectivity only where facing the general public - for example at the very edge, on the CDN, which takes care of serving IPv4 clients. Running IPv6-only is the optimal solution for a future-proof network. Even the U.S. Government, of all places, recognizes this. A memorandum from the Executive Office of the President states: In recent years it has become clear that [running dual-stack] is overly complex to maintain and unnecessary. As a result, standards bodies and leading technology companies began migrating toward IPv6-only deployments, thereby eliminating complexity, operational cost, and threat vectors associated with operating two network protocols. That memorandum was issued three years ago. The underlying IPv6 support on AWS is excellent. It's possible to configure IPv6-only EC2 instances in IPv6-only subnets and everything works fine, including local DNS, time service, host configuration, security and all the stuff to be expected in an IPv6-enabled network. Components like VPN, Transit Gateway, Direct Connect and Network Firewall also support IPv6. Cannot Escape IPv4^3 The problem is actually using all the (other) Amazon Web Services. It's practically impossible to run IPv6-only on AWS. Trying to configure core services like API Gateway, Lambda, ECS or App Runner into IPv6-only subnets makes them either sternly refuse those subnets or throw comically sad error messages like Not enough IP space available (Elastic Load Balancer). And it gets worse: Not even running dual-stack internally helps. Most AWS services, when configured into a dual-stack subnet, happily ignore that the subnet has IPv6 configured; they can connect to IPv4 targets only. This actually hurts customers. Almost no AWS API can be used from a VPC without public IPv4 addresses^4. Running an innocent aws s3 ls from an EC2 instance will fail. Using Systems Manager - the recommended way to connect to and manage an instance - does not work. Accessing SQS from an ECS Task is impossible. And so on. This is because more than 90% of all AWS service API endpoints do not support IPv6. Other endpoints like SES SMTP or ECR repository endpoints^5 don't work either. Cloudfront cannot connect to IPv6 origins. And then AWS frames this new charge as an "encouragement" to adopt IPv6. This is what ticked me off: There is no escape, no way to avoid these charges - because AWS has significantly neglected IPv6 for years. Charges Without Changes For most customers, nothing will change. Which is exactly my point. For AWS customers of any relevant size, this IPv4 charge is a drop in the bucket; they won't even notice it on their bill. In my experience so far, this will usually be very well below 1%. Nobody will invest significant engineering effort for that alone. For many SMBs, hobbyists and startups though, this charge can easily amount to 10-30% of the bill. These customers certainly would adopt IPv6 to avoid such a price increase. So that's where we are: The larger customers do not care about this charge; and those that will feel the impact have no way to avoid it. Conclusion So Far I think the IPv4 charge by itself is fine - maybe even necessary, given the IPv4 acquisition costs lately. I would like AWS to be a role model in IPv6 adoption, like they are in many other areas. If the AWS service landscape had mature IPv6 support, many customers could indeed migrate to IPv6-only environments. Larger customers still would not care - but it would create a lot of first movers within the AWS community, setting examples to follow. But IPv6 is clearly not a first-class citizen on AWS today. Given those circumstances, this charge will do nothing to drive IPv6 adoption. It will stiffle innovation and make it even less attractive to have a personal AWS account - for example, for learning AWS, to study for certifications (to invest in AWS as a career path) or to support open-source work. AWS talks big about democratizing cloud; this isn't it. Upcoming Posts This is the first part of a blog series. Upcoming posts will take in-depth looks at options for ingress and intra-VPC traffic (IPv6 support in Cloudfront, API Gateway, App Runner etc.), egress traffic (avoiding public IPv4 addresses and NAT Gateways), application programming issues (accessing AWS service endpoints and peculiarities of the AWS CLI and SDKs in dealing with IPv6), other obscure details and a list of helpful resources. --------------------------------------------------------------------- Discuss and/or follow on Twitter! Comments on Hacker News --------------------------------------------------------------------- Links * Dual Stack and IPv6-only Amazon VPC Reference Architectures (PDF) * AWS Workshop: Get hands-on with IPv6 * AWS blog + AWS Public IPv4 Address Charge + Public IP Insights + Identify and optimize public IPv4 address usage on AWS + Dual-stack IPv6 architectures for AWS and hybrid networks Part 1 and Part 2 * AWS whitepaper + Designing an IPv6 AWS Cloud network + IPv6 on AWS * AWS VPC documentation + AWS services that support IPv6 (includes a list of ~30 services that have at least some IPv6 support; not listed at all are the ~200 other services) + Migrate your VPC from IPv4 to IPv6 + DNS64 and NAT64 --------------------------------------------------------------------- Update 2023-09-21 - added "Links" section Update 2023-09-21 - affected services: added RDS; made it clear that those are examples (list is not exhaustive) --------------------------------------------------------------------- 1. IPv4 address charges only apply to AWS-owned IP addresses. AWS allows to "bring your own IP" (BYOIP), so if a customer happens to "own" a routable block of public IPv4 addresses (many large organizations do), they can use those addresses for AWS resources. Public "BYOIP" addresses will not be charged. - 2. Interestingly, while EKS supports Fargate, it's not possible to configure public IP addresses. And Fargate's awsvpc networking mode is available on EC2-backed ECS as well, but doesn't support public IP addresses there either. ECS/Fargate seems to be the only combination to allow this. - 3. Thanks to u/slfyst for this reddit post that I shamelessly stole the title from. - 4. It is possible to connect to many (but not all) services via PrivateLink instead, which is a direct connection into a customer VPC and requires no public IP addresses. But it is charged per service and per availability zone, about $9 per month per endpoint - so in order to connect to, say, Secrets Manager, EC2, SSM and SSM-Messages (yes, that's separate) in three availability zones, that's already over $1,200 per year and VPC. Additional traffic charges apply. - 5. So it's not possible to launch ECS tasks from AWS Container Registry, but it is possible from Dockerhub, which does support IPv6. Dockerhub also does rate limiting on pulls. -