[HN Gopher] AWS's Egregious Egress
___________________________________________________________________
AWS's Egregious Egress
Author : jgrahamc
Score : 461 points
Date : 2021-07-23 13:00 UTC (10 hours ago)
(HTM) web link (blog.cloudflare.com)
(TXT) w3m dump (blog.cloudflare.com)
| inopinatus wrote:
| TL;DR: Cloudflare are upset that AWS won't peer with them in
| terms of Cloudflare's choosing, and this is the angry shitpost.
| agucova wrote:
| It does seem like Cloudflare are not the only ones upset about
| how AWS handles traffic
| mindfulplay wrote:
| Great post. Minor rant: However while Cloudflare does not charge
| for edge-egress, their pricing story is terrible, awful and dare
| I say terrible again.
|
| It looks like they took their internal evolving developer docs
| and turned it into pricing for customers.
|
| There is a tree of docs and pricing you have to get to the
| punchline which is that Cloudflare for the most part is several
| orders of magnitude cheaper than AWS or any other cloud provider.
| And each 'team' seems to have their own little pricing page with
| arcane details - exactly the definition of 'ship your org chart
| as products'. Just check out Workers pricing for instance. The
| product is great but the pricing story is silly.
|
| But it's almost like they want to drive customers directly to AWS
| with Amazon's simple pricing calculator and tools.
| agucova wrote:
| In what sense is it order of magnitudes cheaper? Egress
| traffic?
| eloff wrote:
| Thank you cloudflare for calling AWS out on their ridiculous
| transfer pricing. Anybody working in this space has been
| frustrated with it for over a decade. We all know it's a ripoff,
| and we all know it's priced in a way to lock you and others into
| AWS.
|
| I've used AWS at every job I've ever had. But that bandwidth
| pricing has been incredibly frustrating.
|
| Also where possible we now host on lightsail, because the
| bandwidth savings make the hardware completely free. It's not the
| best hardware, but it works. The web console is also a joy to use
| by comparison.
| Terretta wrote:
| Or, it's priced at market -- within 10% of Azure, GCP, and
| Alibaba, and not the highest priced among them.
| eloff wrote:
| Except it's not when it comes to peered bandwidth, like with
| cloudflare. Which is a big deal especially to cloudflare.
|
| If you're in Alibaba, Azure or Google Cloud you can cut your
| costs by putting cloudflare in front, even though that adds
| another service you pay for.
|
| And all of them have ridiculous bandwidth pricing compared to
| the entire rest of the hosting industry.
| edoceo wrote:
| Reads like CF has a bone to pick with AWS. Are they mad about AWS
| not in the peering group? Is the AWS price really out of line -
| the comparison to wholesale cost only seems to miss a lot of
| infrastructure that gets baked into the final retail prices -
| it's not simply (Retail-Wholesale=7000% margin).
|
| Feels like a weak "hit piece" to me.
| Daishiman wrote:
| I think it's quite fair. You're not paying anywhere near close
| to the price of the infra with AWS; you're paying for the
| convenience, but be mindful that the cost has a break-even
| point and many companies may be beyond that point.
|
| I don't have the experience to assess whether the total cost of
| ownership is favorable to most AWS clients or not.
| pkilgore wrote:
| They directly address these costs in the article as "rounding
| error". Do you have some evidence to dispute that? Did I
| misunderstand them?
| notsureaboutpg wrote:
| Cloudflare is ramping up it's enterprise sales and deals (and
| doing a good job of it too, anecdotally). So they are competing
| more and more with AWS (their Workers product is a good
| example), and why not point out things people dislike about
| their competitors?
| isbvhodnvemrwvn wrote:
| They are in the market for reducing that bandwidth by caching
| stuff at CloudFlare, so it's not exactly surprising.
| mikevm wrote:
| Speaking of egress... I've been thinking about hosting stuff on
| Hetzner and using Backblaze B2 for storage to get a cheap
| alternative to EC2+S3, but unfortunately they're not partners so
| I won't get free traffic, or low latency :(.
| erhk wrote:
| No one will migrate off baremetal if ingress were to be that
| expensive.
| StratusBen wrote:
| We consistently hear from our customers that egress and transit
| costs are one of the most difficult things to track from a cost
| perspective as it relates to AWS.
|
| If anyone is struggling to see egress as it relates to S3
| specifically, we put together this guide here which may be
| relevant for this conversation as well:
| https://www.vantage.sh/blog/how-to-see-s3-bucket-request-met...
| andrewguenther wrote:
| AWS Egress - $0.09/GB
|
| Azure Egress - $0.0875/GB
|
| GCP Egress - $0.11/GB
|
| Alibaba Egress - $0.123
|
| Interesting that no mention is made of the other cloud providers
| pricing here when AWS isn't even the most expensive egress...Oh!
| I wonder if it is because Cloudflare has data transfer agreements
| with GCP, Azure, and Alibaba! But that would mean this is a
| shoddy hit piece to strong arm AWS into an agreement with them.
| Nonsense, I'm sure Cloudflare would never do that...
|
| Edit: I've removed claims about Cloudfront's pricing since it
| isn't actually clear what they charge for egress and seems to
| vary on a number of factors. Thanks Dylan16807 for pointing that
| out.
| MisterPea wrote:
| /thread
|
| Thanks for this.
|
| I was trying to find a simple comparison of pricing on the
| article but only found an overly simple water analogy.
| fireant wrote:
| The link you have submitted is about Workers however, which is
| a Cloudflare's function-as-a-service solution. I could not find
| egress pricing for general bandwidth that does not go through
| Workers last time I tried. Could you please point me to a
| document that says CF general egress pricing is $0.045 now?
| andrewguenther wrote:
| The original version of my post called out that you cannot
| actually see Cloudflare's pricing without an account, that
| link was the best I could find. Funny to be calling out their
| competitors for pricing when they won't even publish
| theirs...
| Dylan16807 wrote:
| https://developers.cloudflare.com/workers/platform/pricing
|
| Edit: I was quoting a 100MB per request limit but that's
| actually ingress, egress is 512MB/unlimited so if you
| calculate that out it would be an even smaller fraction of
| a _penny per terabyte_ for bundled workers.
| partiallypro wrote:
| I don't know about GCP but Azure's egress costs go down as you
| use more and more dropping as low as .05/GB, and if you are
| using Cloudflare on top of it the costs go down even more. Then
| don't forget ingress fees, Azure charges nothing, while AWS
| charges ~.015/GB in. Since a lot of protocols rely on heavy
| ingress as well that starts to add up. AWS is notoriously
| expensive in bandwidth costs, that's nothing new.
| corlinp wrote:
| I took the article as pointing out that they specifically do
| not charge AWS for data transfer to CloudFlare and yet AWS
| charges full-price for bandwidth while other clouds do not.
| That's not a shoddy hit-piece, it's exposing some predatory
| business practices. Of course it'll benefit CloudFlare, but
| that doesn't mean it's a disingenuous article.
| laurencerowe wrote:
| If you're doing a lot of egress look into AWS Direct Connect (I
| think GCP has an equivalent too.) That gets you down to
| $0.02/GB
| Dylan16807 wrote:
| What are you on about? A big chunk of the article is about the
| bandwidth alliance, and your number for Cloudflare is not
| egress.
|
| Edit: And in particular, if you are using normal cloudflare
| workers, not the "workers unbound" that are "intended for
| applications that need long execution times", then the price is
| $0.50 per million requests and $0 for bandwidth.
| andrewguenther wrote:
| Thanks for pointing this out, I've gone ahead and removed
| mentions of Cloudflare's pricing from my comment since I
| can't find an authoritative source for their egress pricing
| in their docs. The point general point still stands, all
| cloud provider egress costs suck and Cloudflare is just
| trying to strongarm AWS into entering a bandwidth agreement
| with them.
| Dylan16807 wrote:
| Honestly I'd love it even if AWS just joined into the
| bandwidth alliance. Then I could store backups in glacier
| but have several options to retrieve it at a reasonable
| price.
| xtracto wrote:
| Oracle cloud egress goes from $0.0085 USD to $0.025 USD
| (depending on the zone) and the first 10 TB are free.
|
| It's quite good; too bad their cloud services are quite bad.
| art-vandelay wrote:
| Bad in what sense? Curious because I just signed up for OCI a
| few hours ago.
| jaytaylor wrote:
| My takeaway as a daily user of OCI and an AWS / GCP
| customer:
|
| Oracle cloud has proven to be refreshingly nice and easy to
| work with and develop + manage services on.
|
| AWS is fine but can be costly and has technical baggage.
|
| The cluster mess that is GCP is by far the most
| frustrating. For starters, the UI is indefensibly terrible.
|
| Curious to know what specific complaints or grievances
| about OCI folks have.
|
| _Disclaimer: I currently work at Oracle, but don 't speak
| on behalf of the company in any capacity. This is all my
| personal opinion and honest experience. YMMV and that's
| okay!_
|
| p.s. The OCI "free-forever" VMs are pretty generous-
|
| 2x instances with 1GB ram, much higher network and compute
| performance compared to GCP free-tier.
| dougmoscrop wrote:
| Honestly, looking at Oracle Cloud, I think the best thing
| they could do is spin off and just completely remove any
| trace of the name Oracle. You have some absolutely
| fantastic products that are being criminally neglected
| because people won't go anywhere near the name, and can
| you blame them?
| ksec wrote:
| I dont even think they need to Spin it off. They could
| reform it as subsidiary and have a different company
| name.
| ithkuil wrote:
| GCP has the worst UI, except for all the others
| pqdbr wrote:
| If you are in Brazil, AWS Egress is so expensive (and the BRL is
| so undervalued against the USD right now) that hosting in a bare
| metal with generous free bandwidth tiers (like 20TB you can find
| pretty much anywhere) is a huge competitive advantage against
| those that are hosting on AWS.
| soheil wrote:
| This is a bit shameful. Publicly ousting AWS because they
| basically don't want to join Cloudflare's little attempt at
| market domination named, in 1984 style, Bandwidth Alliance. Any
| attempt AWS makes or for that matter any company makes to build a
| moat around a business can be categorized as anti-consumer
| behavior. Of course, AWS should do their best to make sure they
| make it harder for their customer to leave on a dime while at the
| same time doing their best to keep them happy. Why wouldn't any
| business do that? It's not a charity. The entire cloud
| infrastructure providers could be seen the same way, you build
| your infra in the cloud and it'll become significantly more
| difficult to move away to your own datacenter at some point in
| the future. This is well know yet, that's not an issue raised
| here but asymmetric bandwidth cost is the axe Cloudflare chooses
| to grind?
|
| It seems to be an egregious and opportunistic move by Cloudflare
| to go after AWS right as their parent CEO steps down after a
| divorce.
| rojoroboto wrote:
| This blog post is pretty disappointing. While true that AWS's
| egress can be costly, it is interesting that this article makes
| no mention of the fact that "non-partnership egress" on GCP is
| actually more expensive than AWS and that Azure is roughly on
| par. This article misses the opportunity to talk about cloud
| providers in a broad context and instead attempts to publicly
| shame AWS for not cutting a partner discount with them on egress
| for customers. Don't get me wrong, I'd love an egress discount,
| but this is "partner channel propaganda" masked as an AWS call-
| out.
| [deleted]
| thejosh wrote:
| Azure discount for Cloudflare seems tiny though, but that won't
| fit into their recent marketing efforts Cloudflare has been
| doing.
| pumanoir wrote:
| I didn't see the amount of the discount in the article. Is
| there a place where one can see the pricing (i.e. 10TB azure ->
| CF)?
| donmcronald wrote:
| I couldn't find it when I searched. Some BA participants have
| fair usage limits too. Ex: I think Wasabi is about 1x your
| stored data per month.
| derefr wrote:
| > The only rationale we can reasonably come up with for AWS's
| egress pricing: locking customers into their cloud, and making it
| prohibitively expensive to get customer data back out. So much
| for being customer-first.
|
| Presuming the average AWS customer is running an API SaaS, said
| customer's API response load (= egress bandwidth) will tend to be
| a good measure of how much "useful work" said SaaS is doing for
| its own clients -- either in the form of API responses, requests
| to third parties, emails sent, notifications pushed, uploads to
| other providers, etc.
|
| As such, egress bandwidth tends to be a good measure of having
| product-market fit, and thereby of willingness-to-pay.
|
| Things that trigger mostly ingress load -- dropping attacks at a
| firewall, accepting uploads, receiving emails/notifications, etc.
| aren't as emblematic of product-market fit (they aren't things
| you would traditionally charge customers for "by volume"), but
| rather are usually activities done as loss-leaders; and so _do
| not_ correlate to willingness-to-pay.
| alvis wrote:
| Back in the old day, we were charged by the maximum bandwidth.
| Not quite scalable maybe, but it definitely won't bankrupt you if
| you're not careful.
| AgentK20 wrote:
| Working in realtime games that require high bandwidth usage, AWS
| and basically every other public cloud is fundamentally unusable
| for us because of this exact problem. Our bare metal
| infrastructure for the Hypixel Minecraft network uses 3-4PB per
| month, so we simply lease a 100gbps transit link billed at 95th
| percentile.
|
| Last I checked, AWS wanted ~$200k per MONTH, for _just_ bandwidth
| (no compute, memory, storage, or anything else). We 'd love to be
| able to use the cloud, but not at the cost of increasing our
| monthly expenses by an entire order of magnitude, so we just
| stick to bare metal colocation.
|
| I honestly believe that if you have the technical skill in-house,
| and are spending more than, say, $30k/mo on public cloud hosting,
| that you should seriously evaluate whether bare metal could
| significantly decrease your costs.
| zxcvbn4038 wrote:
| I've found that are generally three stages in a company's life.
| If you are extremely technical you can make bare metal beat
| cloud in the beginning. As the environment grows and IT roles
| diversify cloud starts to make more sense financially then bare
| metal so you do a cloud migration. If you are lucky then
| eventually you get to the point that it's cheaper to run your
| own hardware then rent someone else's and you end up migrating
| out of the cloud again.
|
| If there is a fourth stage then it's getting to the point that
| it's more profitable to run a side business renting out your
| unused resources to others and you become a cloud.
|
| I'm surprised there isn't a Walmart cloud or an Exxon cloud by
| now.
| Zababa wrote:
| I think a big driver for adoption of AWS and the cloud in
| general is tech salaries in the US. You'll often hear people on
| HN talking about how useless it is to try to save $Xk because
| the engineer cost will be even higher than that.
| [deleted]
| kelp wrote:
| This is true until you get to a certain scale and then you
| start looking at your margins and realize a ton of it is your
| wasteful use of AWS.
|
| I spent a big chunk of my last 2 jobs dealing with that
| issue.
| Zababa wrote:
| That's true, and on the opposite side you have companies
| that manage so much hardware that it starts being
| profitable for them to become a cloud operator.
| kazen44 wrote:
| also, the math for some cloud providers doesn't work that
| well if you are in regions which are not EU-west or the US.
|
| even in europe, AWS has "meh" regions compared to azure or
| many of the colo/dc locations.
| Zababa wrote:
| Do you mean compared to the cost of living in those
| regions?
| [deleted]
| jordo wrote:
| This is exactly why we are using DigitalOcean to host our
| gameservers... Our egress costs are basically 1/10th of what we
| would pay on the big three.
|
| DigitalOcean Egress - $0.01/GB
| fwsgonzo wrote:
| Is there any cacheable content? I'm working on high-performance
| compute on CDN software you can run yourself.
| AgentK20 wrote:
| Not when we don't control the game client. We're the largest
| Minecraft server in the world (220k peak concurrent users)
| yet we operate entirely as a third party with no involvement
| from Microsoft.
| Danieru wrote:
| This is super true, yet sadly much of the Japanese mobile
| gaming industry is paying through the nose for cloud hosting.
| There just is not a culture of optimizing the cost structure.
| Zababa wrote:
| Isn't this because they're basically printing money with
| slightly disguised gambling?
| ketzo wrote:
| Are they printing money with gacha games? Yes.
|
| Doesn't mean they couldn't do some optimization! But to
| your point, yeah, I imagine there's probably a little more
| pressure to develop new features/characters/etc. rather
| than spend months rigging up in-house network
| infrastructure.
| cblconfederate wrote:
| > We'd love to be able to use the cloud
|
| why?
| walrus01 wrote:
| For pricing reference you can get a 100GbE transit link (from a
| top-20 sized carrier by CAIDA ASrank size) at major IX points
| now for well under $6000 a month.
|
| And if you are present with your own bare metal infrastructure
| at such a place you almost certainly also have the opportunity
| to connect to a serious IX for settlement-free open peering,
| and to run PNIs to other major sources or sinks of your
| traffic. So by no means will all of your traffic be going
| through transit.
| throw_nbvc1234 wrote:
| How much would 1k 100GbE links cost from one of those
| providers? How much would 10k of those links cost? Does the
| price increase linearly or exponentially?
| walrus01 wrote:
| If you need multiple 100GbE transit connections from ISPs
| larger than yourself, in multiple locations, you most
| likely also are a fair sized ISP, so the situation is very
| different because you'll also be purchasing a variety of
| transport (point to point circuits) between cities at
| 100/200/400GbE capacity, various DWDM circuits, lighting
| your own dark fiber, etc. It's a whole other ball game.
| ianhawes wrote:
| Is *that* why Hytale hasn't been released yet?
| JMTQp8lwXL wrote:
| There's the opportunity cost to throwing away your
| organization's knowledge of AWS, though. Everyone will have to
| learn to do things the bespoke way on your bare metal. So
| developer productivity could decline.
| AgentK20 wrote:
| I would say the exact opposite in some ways for teams who are
| not using AWS yet. My team for example has been operating
| bare metal for 8 years now, and we know how to do that.
| Transitioning our team to the "bespoke way" that AWS does
| things has a huge opportunity cost, too.
| freedomben wrote:
| This is where Kubernetes I think helps a ton. I work with
| bare metal customers all the time that stand up OpenShift on
| their BM and can migrate k8s apps very easily. Depending on
| the apps you may need to throw an object storage solution in
| there too (such as OpenShift Container Storage). It does
| require a certain scale before this makes sense, but it's not
| nearly as high of a scale as most people think.
| surfer7837 wrote:
| You go on to AWS for their managed services like Fargate,
| DynamoDB, ECS, S3 etc. Have used OpenShift in the past and
| had endless problems with cluster stability (especially in
| 3.x), and weird inconsistencies.
|
| With AWS I could just spin up 10 Kubernetes clusters with
| pretty much unlimited resources, can't do that in OpenShift
| because you'd hit a resource quota or limit.
| dilyevsky wrote:
| Imo using cloud vendor apis is more bespoke than using
| kube/nomad or even old school andible/salt on baremetal
| cluster. With the exception of s3 all your knowledge will
| tabled if ever need to switch the vendors
| agucova wrote:
| How do you orchestrate or manage your servers currently? And do
| you employ a in-house solution for it as well?
| aclelland wrote:
| At $JOB, we have a similar issue with static content. We serve
| over 1PB a month and the price AWS would charge to use
| CloudFront is an order of magnitude larger than even a
| Cloudflare Enterprise plan (which comes with some nice bells
| and whistles that Cloudfront doesn't offer).
|
| Even with discounts from AWS it just doesn't make sense to use
| AWS to serve up the assets to users.
|
| We do use S3 as our static asset backend and the combination
| works really well. I would love to see Cloudflare release a S3
| compatible storage service though. I think we'd jump onto that
| in a heartbeat.
| HeavenFox wrote:
| Backblaze has S3-compatible API and they have free data
| transfer to Cloudflare
| aclelland wrote:
| The last time I looked at BB S3 API they didn't offer the
| lifecycle controls that AWS over. Mainly the ability to
| remove old versions of files after a period of time and we
| didn't want to roll our own expiry solution. Might be worth
| looking again though.
| treesknees wrote:
| Yep, B2 does offer this capability now. They've been
| working hard to add S3-compatible/similar features.
|
| "Lifecycle rules instruct the B2 service to automatically
| hide and/or delete old files. You can set up rules to do
| things like delete old versions of files 30 days after a
| newer version was uploaded."
|
| https://www.backblaze.com/b2/docs/lifecycle_rules.html
| dvaun wrote:
| What if your business has fluctuating loads?
|
| I can see how running game servers--with a (somewhat, maybe?)
| predictable load of players--can be done efficiently on
| baremetal and with colocating.
|
| For another business that has huge peaks of demand, such as
| analytics with dynamic queries, I fail to see how baremetal can
| compare to spinning up hundreds of instances on-demand.
|
| Perhaps it comes down to what services you offer your clients
| and how you implement them?
| benlivengood wrote:
| Just compare the spot/preemptible instance price or
| committed-use price to the on-demand price to see how it can
| be cheaper.
| jandrewrogers wrote:
| We always ran most load in our own data centers with the
| ability to transparently shed load into AWS for large peaks.
| It saved a lot of money.
| brianwawok wrote:
| The scale part of the cloud is often not as exciting as it
| seems on paper. Yes a cloud can auto-scale 100x. But can
| everything else support that? Like is your DB setup to handle
| 100x increase in load? (Sure, use dynamo DB, but that has
| other restrictions).
|
| If the cloud to bare metal price difference is 10x.. you
| could easily just buy bare metal = 2x your peak load, and
| still come out ahead..
| dvaun wrote:
| What you state makes sense. For me, I haven't worked in an
| environment (yet) that would need to handle fluctuating
| loads at scale--so my comment is my own speculation based
| on my experience working for smaller businesses with MUCH
| less data and bandwidth usage compared to those mentioned
| here.
|
| And that's why I come to HN, lobsters, etc :) so that I can
| read and learn from others' experiences...
| freedomben wrote:
| I've worked with dozens of different companies, and it's
| pretty rare to truly have such crazy variance in load.
| qeternity wrote:
| The promise of cloud scalability never really materialized in
| the way that people talk about. The value in cloud today is
| really in hosted services. Why have loads of exports for
| different supporting infra if you can just lease them from
| AWS through a hosted service.
|
| If you look at what it would cost to run bare metal with top
| shelf companies, it's often as cheap or cheaper than
| spot/preemptible instances at the cloud shops.
| mdasen wrote:
| On the Hotel California Effect
|
| If it's expensive to move data out of AWS, it's not just about
| making it hard for customers to leave AWS. It means that any
| third-party service that wants to sell to AWS customers must also
| use AWS.
|
| For example, Snowflake can't really run its own data centers.
| They need to rent from AWS. If you're a company on AWS, you won't
| want to spend $0.08/GB sending things to Snowflake.
|
| The high egress cost gives Amazon a lot of power to keep the
| third-party ecosystem within AWS and without as much negotiating
| power on things like EC2 costs. Snowflake can't say to Amazon,
| "if you don't give us a great discount, we're going to move our
| operations to DigitalOcean" because Amazon would just say, "no
| you're not. Your customers aren't going to pay $80/TB to load
| things into Snowflake when you're charging them $23-40/TB for
| storage. They'd be paying 2-3.5x your storage costs just to load
| the data into your system!" Yes, I'm sure that Snowflake does get
| a discount from AWS, but having less negotiating power can make
| discounts smaller.
|
| There are other reasons to want a third party to use AWS if
| you're on AWS. Still, the egress pricing seems to make it very
| hard for third-party tech providers not to use AWS if their
| customers are using it.
| dilyevsky wrote:
| There _is_ an option of peering at 2c /g but yeah it's not
| particularly cheap either when you compare it to transit costs
| in the real dc
| jfim wrote:
| These costs are not only on egress traffic to the internet, but
| also on other ways to get data out of AWS. For example, the
| snowball appliance that is basically a pile of hard drives that
| can be shipped to AWS has free ingress, but costs a few cents
| per GB for egress [0].
|
| [0] https://aws.amazon.com/snowball/pricing/
| nindalf wrote:
| Thank you for this. I'm surprised that CloudFlare didn't
| mention it in an otherwise thorough post. After all, the reason
| they're trying to pressure AWS to change is that they'd like to
| offer services to AWS customers without bandwidth markup.
| Essentially competing with existing AWS services on an even
| footing.
|
| This to me makes it unlikely that AWS will change their
| bandwidth pricing. If it was just about this revenue source
| disappearing, they might bite the bullet and hope that the
| lower prices would attract more customers in the long term. But
| bandwidth pricing is the moat around their castle. If it didn't
| exist, Snowflake and others would leave. But more than that,
| new AWS services benefit from the captive audience choosing
| them by default. Without the moat, each service has to compete
| on its merits. AWS Lambda, for example would need to be better
| than CloudFlare Workers or fly.io containers. It won't win
| customers like it does today just because it's "free"
| bandwidth.
|
| Removing these fees is a risk AWS is currently choosing not to
| take. And they won't change their pricing regardless of the
| number of blog posts that competitors write. The only thing
| that would force a change is an exodus of customers leaving AWS
| citing this as a reason.
| ksec wrote:
| And I always thought they will drop the price of their EC2
| some day to balance things out. But nope.
|
| I am wondering if Amazon could introduce a bandwidth cost
| rebate, where the bandwidth cost would be subtracted from
| your final bill at a maximum of 30% of your total VM bill.
|
| But it is like Apple, once you have a monopoly like market
| they just dont need to. Amazon is buying as many TSMC
| capacity as they could as demand for Graviton 2 is way above
| their projected expectation. It seems they are still growing
| at such a pace despite their size.
| Hokusai wrote:
| Amazon is really good at leveraging its semi monopolistic
| presence in different markets. Once you are big enough you only
| need to tweak a few parameters to dominate the market.
|
| It's interesting how AWS uses egress to this effect. Your
| comment is very insightful.
| stadium wrote:
| They employ teams of economists to model price elasticity of
| demand. There are likely many orders of magnitude more than a
| few parameters. And they look at how a change in one product
| feature or pricing model cannibalizes or compliments other
| features and prices.
| ignoramous wrote:
| AWS, may be, subsidizes the costs of its loss-leaders by (what
| seems like) charging disproportionately for egress. I reckon,
| as Cloudflare gears up to take on incumbent cloud leaders, it
| modus operandi will be to commoditize its competitor's
| advantages.
|
| Apart from CloudFront and Lambda@Edge, Cloudflare has offerings
| that compete with Lambda (Workers Unlimited), Web Application
| Firewall, AWS Elemental (Stream), and (app/network) load
| balancers (Argo, Spectrum, Magic Transit). Check out
| Cloudflare's bandwidth / usage pricing for some of those, and
| it isn't much different than AWS.
|
| Cloudflare, I'd predict, will eventually arrive at a _similar_
| pricing model for its IaaS outside of the _Bandwidth Alliance_
| (which, ironically, _is_ Hotel California with select though
| multiple providers at play).
| kelp wrote:
| And features like AWS PrivateLink play right into this
| strategy. If you're a SaaS provider built on top of AWS, the
| egress costs heavily incentivize you and your customer to
| connect over PrivateLink vs over the internet.
|
| This keeps both parties locked into the AWS ecosystem.
| noasaservice wrote:
| I've always referred to this as a roach motel model.
|
| Most proprietary vendors do various tactics under this to
| retain customers via lockin, versus actually good products.
|
| Software file formats serve the same for proprietary software.
|
| "Cloud" vendors provide easy API services that have no analog
| to on-prem, so that migrations are painful. And the Great
| Egress is a painful problem with AWS.
|
| Roach motel indeed. They check in, and they don't check out.
| IceHegel wrote:
| Something is very off. The post claims an 8000% markup but others
| have pointed out that AWS is in line with other cloud providers
| and the same math shows cloudflare with a ~3800% markup. From
| u/andrewguenther:
|
| AWS Egress - $0.09/GB
|
| Azure Egress - $0.0875/GB
|
| GCP Egress - $0.11/GB
|
| Alibaba Egress - $0.123
|
| Cloudflare - $0.09/GB pre-April 2021 $0.045 now
|
| 1. If AWS is charging me $6 per Mbps for something that costs
| them 8C/, why is Cloudflare charging me $3 for the same 8C/ good?
|
| 2. What is going on with the price of the underlying technologies
| that make up network capacity. Is it just fiber and switches? Are
| these costs going down?
| baskire wrote:
| Buying 1 Mibps means you can use up to 1Mibps. Paying $/GiB
| means you can use 100s of Gibps for a split second and only pay
| the $/GiB fee.
|
| In the past i did a TCO comparison for a large spikey workload
| where the $/GiB cost even at list was cheaper than allocating
| enough bandwidth for peak events.
|
| At small scale the $/Mibps likely comes out cheaper, but for
| any large company sitting on 100s of Gibps of excess capacity
| for an event that occurs once or twice a year, or a few hours
| per day is very costly and might eat up savings from p99/p95th
| billing.
| Dylan16807 wrote:
| > Something is very off. The post claims an 8000% markup but
| others have pointed out that AWS is in line with other cloud
| providers
|
| None of the biggest ones are trying to compete on egress.
| Especially because, why make it easy to run some of your
| services outside their cloud? Please move it all in to the same
| place and dedicate your budget to them.
|
| > Cloudflare - $0.09/GB pre-April 2021 $0.045 now
|
| That's not general egress, that's one random service optimized
| for using more CPU time. If you use the less-CPU-time version,
| bandwidth is free/negligible.
|
| > What is going on with the price of the underlying
| technologies that make up network capacity. Is it just fiber
| and switches? Are these costs going down?
|
| Better tech allows vastly increasing amounts of data to flow
| through the same wires using the same amount of equipment. The
| price per unit goes up, but the price per byte goes down.
| donmcronald wrote:
| It looks like Cloudflare cut egress pricing for workers unbound
| to $.0045 per GB, but that's still half of AWS or "40x markup"
| for me in Canada.
|
| The original workers model of pay per execution is incredible.
| IIRC I figured one run is the equivalent of 6KB of egress at AWS.
| Is there a point where Cloudflare will start charging egress on
| those if I transfer too much? I want to use it for something, but
| don't like the "free bandwidth until you hit the invisible limit"
| side of things Cloudflare has going.
|
| 1. https://blog.cloudflare.com/workers-unbound-ga/
| satyrnein wrote:
| There's a typo, it's $0.045 per GB (half of AWS, as mentioned).
| I got excited for a second!
| donmcronald wrote:
| Damn. I'm stupid today. I don't think I can edit anymore
| either :-(
| yepcfstyle wrote:
| Cloudflare argo pricing (basically stay on the CF network) is
| almost identical to GCP Premium Tier and AWS default tier. They
| claim it does some cool optimizations to traffic path, no
| different to BGP optimizers - it is simply ridiculous to
| presume GCP/AWS doesnt optimize their routes too on a continous
| basis - yet apples and oranges doesn't apply CF says. I applaud
| CF nudging AWS to reduce bandwidth fees and things like
| bandwidth alliance but they sure do a hold my beer on
| hypocrisy.
| Dylan16807 wrote:
| > The original workers model of pay per execution is
| incredible.
|
| It's still there!
| [deleted]
| torcete wrote:
| Base on the costs of transferring data into AWS being much
| cheaper, I wonder if a reverse transfer protocol could be
| designed with an adaptation of binary search.
|
| You divide your data in blocks of, let's say 100 Mb, the receiver
| creates a block of random data and sends it to a process running
| on AWS. The process has to answer, higher, lower or exact (and
| then both parties move to the next block).
|
| Probably it will be very slow, but it will lower a lot the
| bandwidth usage for egression.
| CodesInChaos wrote:
| Getting out more data than bandwidth allows is a information
| theoretical impossibility. Your proposal has the flaw that it
| needs one pass per bit in the block on average.
| cameronh90 wrote:
| "In other words, ingress (data sent to AWS) doesn't cost them any
| more or less than egress (data sent from AWS). And yet, they
| charge customers more to take data out than put it in. It's a
| head scratcher.
|
| We've tried to be charitable in trying to understand why AWS
| would charge this way. Disappointingly, there just doesn't seem
| to be an innocent explanation. "
|
| One consideration could be that you can't directly control data
| ingress. Charging for ingress opens an avenue for attackers to
| attempt to bankrupt you by sending unrequested packets to your
| network. At the very least, this could cause a customer service
| headache. Perhaps AWS decided they'd rather not deal with that?
| sitkack wrote:
| > One consideration could be that you can't directly control
| data ingress.
|
| Are you thinking UDP? Or for a DDOS? Connection setup overhead
| could be accounted for. I highly doubt this is the reason.
| blamethenetwork wrote:
| At a routing and peering level. Once you have an announcement
| for your netblock out there, traffic will start to head
| towards it. A lot of this is due to the BGP Path Selection
| Algorithm.
|
| You can try and influence how traffic arrives, by doing
| things like, AS prepends, but you are still going to get
| traffic.
|
| The main reason for this is that the other side that is
| egressing to you has their own egress policy that also
| follows path selection. Things like localpref and weight will
| force my traffic to leave via a path before it considers how
| a network has AS padded.
|
| As an example: Lets say I want to egress (company A) to a
| downstream company (company B). If I learn routes to Company
| B via multiple ways: peering fabric (low cost), paid peering
| (medium cost), transit1 (high cost, variable quality),
| transit2 (low cost variable quality), I can choose which way
| my traffic goes, via localpref, weight etc.
|
| Only when I view the paths equally (equal localpref, weight
| etc.) will I evaluate the shortest AS Path (which the
| receiving company has influence on).
|
| The only way to completely not get inbound traffic via a
| specific link, is to remove your BGP advertisement for your
| netblock from that link. (some providers also let you do this
| selectively via BGP communities).
|
| There are also some other tips/tricks - such as adding a more
| specific prefix to a certain link, to attract traffic, but
| care needs to be made to have a fallback route in case things
| go wonky.
| linuxftw wrote:
| If there's an IP exposed on the internet, you can just send
| it tcp payloads. The end destination will silently drop them,
| but it doesn't mean people can't send you gigs of useless
| data.
| otterley wrote:
| I think you mean UDP. TCP requires a 3-way handshake first.
| ithkuil wrote:
| Intermediate routers don't care about that; they only
| forward the IP packets; four target host/firewall will
| drop them (because they don't belong to a valid
| connection) but they will be still accounted for as
| ingress traffic.
| tw04 wrote:
| Coming from someone who believes that the AWS egress fees are
| highway robbery, I'd throw this out there. In order to maintain
| settlement free peering, you need to keep egress and ingress
| somewhat even with your peers.
|
| Amazon proper probably has 3x egress vs ingress between prime
| and twitch. AWS is an easy avenue to balance it out.
|
| All that being said: I would guess there's almost 0 chance
| that's what's happening here. Amazon is just using it as a form
| of lock-in IMO.
| hnpasf wrote:
| I agree that it is just lock-in. The price of egress is the
| same even when using devices like the Snowball where the data
| is loaded onto a server and physically shipped to you.
| res0nat0r wrote:
| The only service AWS hasn't consistently lowered pricing on
| (unlike almost everything else) is their egress cost. Everyone
| complains about vendor "lock in" for anything AWS related, and
| I've never thought so since they've always reduced prices
| across the board. But egress is the exception. They're keeping
| pricing high for this exact reason to prevent folks from moving
| their data elsewhere.
|
| I've got about 25TB of stuff I'd love to keep all in Glacier
| Deep Archive, but the restore costs are just too insane to
| justify due to the egress pricing still at .09c after all these
| years. Too bad.
| qeternity wrote:
| > Everyone complains about vendor "lock in" for anything AWS
| related, and I've never thought so since they've always
| reduced prices across the board.
|
| How would keeping prices high for these services keep people
| locked in? The whole point of vendor lock in is there is some
| other indirect cost that you bear by switching, which is why
| you don't switch.
|
| For AWS, this indirect cost is bandwidth. It _is_ the lock in
| mechanism.
| res0nat0r wrote:
| > How would keeping prices high for these services keep
| people locked in?
|
| By doing the opposite and _raising_ prices across the board
| randomly because they know folks can 't spend the time and
| money to migrate without a large support burden or
| time/money sunk cost.
|
| The only service not going down in price is egress and the
| only thing I've seen which has been like this, to directly
| discourage folks from migrating their data to say GCP when
| their offering(s) look more attractive.
| slownews45 wrote:
| If you have larger data needs you might look at snowcone ->
| snowmobile family of solutions. Generally 0.03/GB and scales
| to 50 petabyte + transfer volumes.
| res0nat0r wrote:
| I've actually looked at that before if I was wanting to a
| large restore if my home nas just totally crashed. .03c is
| still going to cost like $1000 which still is pretty steep.
| Mostly because you know these devices aren't being plugged
| in and the data extracted from s3 over the WAN, they are
| likely plugged in directly in the datacenter over some
| insane internal high-speed link.
| skuhn wrote:
| I view network egress as the primary lock-in mechanism for
| all of AWS.
|
| If you want to migrate to another provider, network egress
| costs mean that you'll spend multiples of your normal monthly
| operating costs to do so. That stifles competition.
|
| But well before that point, most services in AWS have already
| been built around avoiding network egress wherever possible.
| You're always going to prefer AWS APIs and services, even if
| they aren't the best for your use case, because services
| outside AWS have a network egress tariff placed on them. So
| you aren't always buying best-of-breed, you're buying the
| best of what AWS chooses to offer (or the selection of
| vendors that choose to build in AWS to remain competitive).
|
| And if that isn't good enough, your only other option is to
| migrate out (and pay those egress costs!).
| isbvhodnvemrwvn wrote:
| If you get DDoSed you also pay through the roof in fees (for
| whatever is in your VPC - e.g. ELB which are billed per
| throughput) unless you buy their anti-DDoS services (AWS Shield
| Advanced) for 3k$/mo - without that AWS won't waive the fees.
| RKearney wrote:
| > One consideration could be that you can't directly control
| data ingress. Charging for ingress opens an avenue for
| attackers to attempt to bankrupt you by sending unrequested
| packets to your network. At the very least, this could cause a
| customer service headache. Perhaps AWS decided they'd rather
| not deal with that?
|
| An equally likely reason is by making ingress free it helps
| balance out ingress and egress traffic flows which are usually
| desirable for settlement-free peering agreements.
| qeternity wrote:
| > One consideration could be that you can't directly control
| data ingress. Charging for ingress opens an avenue for
| attackers to attempt to bankrupt you by sending unrequested
| packets to your network.
|
| Most request/response models are very asymmetric. I can request
| a relatively large asset from S3 with a small request and
| maliciously generate fees as an attacker.
|
| AWS egress is vendor lock-in 100%.
| kevincox wrote:
| The simple explanation is that ingress is far, far lower than
| egress so the cost isn't relevant since links tend to be
| symmetrical. So the cost to them is determined based on egress,
| their ingress provisioning happens at no additional cost.
| eli wrote:
| It's also just a way to segment customers. The people doing
| tons of egress, on average, probably have bigger budgets for
| cloud computing. There's no particular reason AWS costs have to
| mirror how they charge.
| trinovantes wrote:
| Weird that Digital Ocean is not part of the bandwidth alliance.
|
| If they were, would that mean I _may_ not get charged bandwidth
| for traffic from my DO VPS to CF proxies?
| ksec wrote:
| Oh that is interesting because I remember DO and Linode were
| part of their partners during the initial announcement. And now
| its name is gone.
|
| https://www.cloudflare.com/en-gb/bandwidth-alliance/
| PaulHoule wrote:
| Does this make sense?
|
| He seems to be comparing peak bandwidth pricing (megabits per
| second) to average throughput pricing (megabits).
|
| Peak bandwidth pricing is really a lie (if you actually used it
| all the time you'd be looking for a new host)
| eastdakota wrote:
| Calculated assuming 20% utilization, which is the low end of
| industry average (20% - 40%).
| snicker7 wrote:
| I speculate that the reason why Amazon charges so much for egress
| (other than to nickel-and-dime its customers) is that it inhibits
| hybrid-cloud or multi-cloud architectures. It promotes lock-in.
|
| In literally every business domain, Amazon succeeds not because
| of "customer obsession", but because of their willingness to
| engage in anti-competitive or outright monopolistic practices.
| unethical_ban wrote:
| I wonder if net neutrality rules, combined with the absolute
| dominance of public cloud infrastructure in the US, could force
| providers to have more reasonable egress pricing.
| adventured wrote:
| > In literally every business domain, Amazon succeeds not
| because of "customer obsession", but because of their
| willingness to engage in anti-competitive or outright
| monopolistic practices.
|
| You're obviously casting a blanket that is far too wide. That's
| the hate Amazon train running out of control.
|
| They legitimately do a great job for retail customers, re
| customer obsession. Their retail customer service and
| willingness to make things right, is excellent.
|
| I've been buying from Amazon.com for ~23 years or so now.
| They've never failed to make something right. I've had maybe
| three bad purchases in those 23 years out of hundreds of orders
| (meaning something arrived that I didn't order, or it arrived
| damaged), all were made right with little effort on my part.
| Amazon almost never protests if something goes wrong with an
| order.
|
| Their retail customers by and large adore them for that reason.
|
| Their kindle ebook solution won because it was an excellent
| experience overall with a vast selection that was priced very
| much in the favor of consumers (and still is). They produced a
| great ebook at a great price. It's very easy to use and their
| pricing was so tremendous publishers sued them to try to force
| book prices higher on consumers.
|
| Their Prime membership program has been no more anti-
| competitive than Costco's membership program.
|
| Your scenario damns Amazon every direction no matter what they
| do, which reveals the plot. If they raise prices, they're a
| brutal monopolist taking advantage to gouge consumers. If they
| give consumers cheap prices, they're an anti-competitive
| monopolist undercutting on prices. And if by chance they
| perfectly align with everyone else on pricing, well they're
| obviously colluding to set prices, a clear anti-competitive
| practice.
| slownews45 wrote:
| Interesting to see them grow so big then - they used to be the
| minnow to the Walmarts / Targets etc.
|
| At least in my experience (retail product ordering) their
| service flat out _IS_ head and shoulders above much of the
| compeitition.
|
| I order off a Facebook ad - I get a drop shipment from china
| that takes forever, is effectively not returnable etc etc.
|
| I order from Amazon - delivery is insanely reliable, when I
| have an issue refunds are handled incredibly well. Even special
| cases (I had a weird issue with an iphone order that required a
| manual override and refund despite what looked like a failure
| to return - I got a real person and a refund promptly - they
| would have been relatively justified NOT to refund but I
| thought I'd ask).
|
| Even on AWS my support contract get's real responses. When I
| had a billing / customer service issue - a real person (sounded
| like native English speaking US call center) handled it very
| well.
|
| My brokers is constantly experiencing "unexpected call volume"
| (hour long waits). The IRS when I call has "unexpected call
| volume" and hour long waits. Calling southwest they have
| "unexpected call volume" and hour long waits. A ton of places
| make it virtually impossible to actually call and talk to
| anyone (much smaller than Amazon BTW) - you can search website
| all day long. Or you are speaking to call centers that can't do
| anything at all.
|
| I contrast AWS to the treatment we got as a paying google suite
| customer (endless different sales people calling us, but you
| don't have good paths in to trouble shoot weird state issues
| etc beyond basic customer service). And AWS devices support
| employees work calendars (via gsuite) while google devices DO
| NOT!
| zxcvbn4038 wrote:
| I've had very poor experiences with gsuite support. Once your
| issue gets past the front line support it disappears into an
| opaque void where nobody knows anything or can talk to
| anyone. Once I was asked to write a business case clarifying
| why answering my support question benefits Google. (Found a
| bug in gmail css handling, wanted a workaround).
|
| With AWS support I can get an answer to my toughest "found a
| bug" issues in hours or a couple days at worst.
|
| I'm pretty sure if I called AWS support on Thanksgiving and
| asked for help cooking a turkey they would send me a recipe.
| Their support is really amazing.
| vel0city wrote:
| > I contrast AWS to the treatment we got as a paying google
| suite customer
|
| Did you have an additional paid support contract with that
| Google Suite/Workspace account? What would your customer
| service have been with AWS if you didn't have that additional
| support contract? If you're paying for premium support on one
| service it seems a little unfair to compare that to the
| included base tier support of the other.
| clvx wrote:
| I wonder if this could be taken as IE on Windows by default.
| Would this be anti competitive knowing there are industry efforts
| to benefit customers on other big customers.
| kbuck wrote:
| I've assumed this was well-known... AWS has never been a good
| deal on the billing side. You're paying a huge premium for API
| access and the strength of the AWS brand.
|
| I was a bit confused as to why Cloudflare was blogging about it
| until halfway through the article, when The Bandwidth Alliance
| came in. Makes sense now: they just want to shame Amazon into
| offering free egress bandwidth to Cloudflare. This is basically a
| hit piece.
|
| Also, another aside: for hosting providers (AWS included),
| ingress bandwidth is typically free because it's also basically
| free for the provider. Outbound dominates so much that the
| inbound traffic is a rounding error if it's billed by their
| transit providers at all. But I agree that the high egress prices
| are a vendor lock-in strategy.
| [deleted]
| pkilgore wrote:
| Perhaps I misunderstood, but the article says Amazon already
| does not pay for egress to Cloudflare. But Amazon still charges
| their customers as if they did. Did I misunderstand something?
| inopinatus wrote:
| You sure did. Peering isn't free, or even cheap. Cloudflare
| are whining for special treatment for their bandwidth cartel,
| and AWS are large enough to tell them to get fucked.
|
| The key is realising that Cloudflare have always been about
| disingenuously leveraging other people's stuff, since they're
| basically just a middleman.
| api wrote:
| It's very much by design. They want ingress not egress to create
| "gravity." The end game is for virtually all compute to happen in
| the cloud and be accessed only by thin client. This pricing
| drives things in this direction by making companies that would
| use the cloud to just service local compute less profitable than
| companies that host everything in the cloud and provide only web
| or other thin client access.
|
| There are providers that don't do this, but they are much less
| batteries-included. The best bandwidth deals are via bare metal
| hosting which is basically just box-in-a-rack rental. You can get
| bandwidth there by capacity rather than transfer quota. It's many
| many orders of magnitude cheaper.
___________________________________________________________________
(page generated 2021-07-23 23:01 UTC)