[HN Gopher] An Uber-like CDN
___________________________________________________________________
An Uber-like CDN
Author : mranton
Score : 56 points
Date : 2023-03-15 06:13 UTC (16 hours ago)
(HTM) web link (medium.com)
(TXT) w3m dump (medium.com)
| emj wrote:
| Coral CDN 2004-2015 was ment to be p2p, why that can be a problem
| is mentioned in comments here and in this paper from Princeton.
| https://www.cs.princeton.edu/~mfreed/docs/coral-nsdi10.pdf
| jedberg wrote:
| This sounds a lot like CoralCDN. You should look them up, they
| had some good papers. We actually used CoralCDN on reddit for a
| while, where we'd send users to the CoralCDN link of the webpage
| they were trying to get to instead of the actual page. It sadly
| didn't work very well.
|
| Here is one of their papers:
| https://www.usenix.org/legacy/event/nsdi10/tech/full_papers/...
| nso wrote:
| One of the problems I have with the VP of the product discussed
| is that while I trust CloudFlare, Akami or any of the big
| players, to run a tight network -- with high and STABLE
| performance -- I can't say the same about a random vendor (uber
| driver?) in this network.
|
| Was the issues you were seeing in this general area?
| akamaka wrote:
| Will the server-side code for this be available as open source,
| so that it can be used by someone who wants to run their own CDN
| but doesn't want to be part of the network?
| mranton wrote:
| right now no, but we have an idea to opensource everything.
| eastdakota wrote:
| Travis built RedSwoosh
| (https://en.m.wikipedia.org/wiki/Red_Swoosh) and then went on to
| build Uber. Now someone saw Uber and claims it inspired them to
| rebuild RedSwoosh. -\\_(tsu)_/-
| mranton wrote:
| >The Red Swoosh peercasting tool is a _browser_extension_ that
| caches data, reflecting and sharing files delivered through the
| "Swoosh network"
|
| I don't know how it works in details, but it looks like you
| need to install a browser plugin to enable the p2p network
| here.
|
| Our solution works transparently for the end user.
| dedalus wrote:
| Quite an honor to see Matthew Prince (CEO of Cloudflare) chime
| here.
|
| Akamai acquired RedSwoosh precisely on the same economics
| promised and if this were economical/viable they would already
| have deployed it, no?
|
| I did hack with Travis on RedSwoosh code base optimizing his
| TCP Zipper algorithm which was pretty cool BTW
| amalcon wrote:
| Something doesn't add up here. They're actively telling third
| parties to lease cloud servers with large amounts of free
| bandwidth, and allocate them to this purpose. I see four
| possibilities:
|
| - They are paying the third party less than what it costs to rent
| the server, in which case the third party is losing money and
| shouldn't be doing this.
|
| - They are paying the third party at least what it costs to rent
| the server, in which case why aren't they just renting the server
| themselves? They could probably negotiate a bulk discount.
|
| - They are trying to hide this activity from the cloud server
| provider by involving third parties. While this may not be a ToS
| violation as such, I'd guess that if they're hiding it, it is
| something the cloud providers will want to crack down on if it
| becomes significant.
|
| - This entire scheme is actually a plan to get access to
| countries with weird regulatory requirements, e.g. foreign
| companies are not usually allowed to own servers in China. This
| one's pretty legit actually, but it's a really complex version of
| the scheme.
|
| I think at least one of those must be true, and none of them
| would be a good sign for the future.
|
| Edit: I guess "paying the third party less than their costs"
| and/or "not making a profit themselves" might be what they mean
| by "Uber-like", so it would just be an elaborate joke?
| boh wrote:
| I guess it really is Uber-like.
| tyingq wrote:
| I think it's indirectly exploiting providers that give you a
| fixed bandwidth rate for a low cost. Those providers do this
| knowing some large percentage of customers don't use it all.
|
| So, if this becomes popular, flat rate servers with unmetered
| bandwidth at a fixed speed would become more expensive, or come
| with a more specific ToS to prohibit that.
|
| Similar to how data hoarders ended the "really cheap personal
| cloud storage" bubble.
| jeroenhd wrote:
| I think it's quite simple. The math adds up, they just want
| third parties to network with them. The more servers join this
| system, the more their product is worth. At the same time, the
| cost of managing hundreds or thousands of servers is moved to
| others.
|
| The more this trick will be used, the higher the cost of the
| dedicated fast internet connections with cloud hosts will
| become. Most people buying dedicated, unlimited uplinks don't
| upload the amounts mentioned in this article, which is why
| bandwidth is relatively affordable. If everyone buys into this
| network to make a profit, the prices would probably rise to the
| point that profit drops. This isn't against any kind of ToS I
| know of.
|
| The "make 2.4x your investment" promise also assumes your
| server will actually serve 100TB of content. The more people
| join, the fewer files your server will serve and the less
| profit you'll make.
|
| In the end, the system is profitable until the network has
| reached the size where everyone breaks even. If people leave
| the network, profits go up again, if people join the network,
| profits drop and people back out. People renting servers game
| the system trying to one-up each other and in return Farba gets
| a worldwide CDN for cheap.
|
| The best trick may perhaps be that server providers may
| directly buy into the system, as they can get bandwidth and
| servers for lower margins than their customers. This gets Farba
| high-quality service for cheap while the service providers add
| a relatively easy income stream on the side.
|
| All of this comes at an additional cost to the customers,
| because now your battery-starved mobile phone is validating
| cryptographic signatures of images and other content.
|
| This also leaves legal questions: who's responsible for illegal
| content hosted on servers hooked into this network? If your
| hosting provider of choice gets a takedown notice for child
| porn or malware, will you still be able to use their services?
| mranton wrote:
| Thank you, @jeroenhd, you got it right.
|
| >now your battery-starved mobile phone is validating
| cryptographic signatures
|
| with a native app, it costs nothing. Actually, the main power
| consumer is the phone screen.
|
| >who's responsible for illegal content hosted on servers
|
| this is a very important question. We are going to run AI
| models against all content we host on Peers. Right now, we
| don't accept third-parties and "fake" all Peers ourself. Want
| to ask lawyers what is the best way to handle this case.
| thayne wrote:
| For the first bullet point, it could be worth it for the third
| party if they are using the server for something else, but have
| surplus resources that could be used for this CDN. And they
| might be paying higher rates initially to get enough peers to
| make the CDN worth using.
| asciii wrote:
| > Something doesn't add up here.
|
| Agreed, I'm mildly annoyed that farba.js repo has not been
| updated since 2021 (no further info) + grammar issues [1]
|
| [1] https://github.com/FarbaCDN/farba-
| public/blob/master/src/far...
| mranton wrote:
| >Something doesn't add up here. They're actively telling third
| parties to lease cloud servers with large amounts of free
| bandwidth:
|
| Bare-metal servers are much better is our case. Please note,
| bandwidth is not free. You can follow the links and see the
| actual price for it. If LeaseWeb tells you: 1 Gbps
| dedicated/unlimited per $143/month, it means you can use it
| 24x7.
|
| >- They are paying the third party less than what it costs to
| rent the server
|
| we don't know prices in all datacenters all over the world.
| That's why the _Peer_ sets the price for his service.
| SahAssar wrote:
| > If LeaseWeb tells you: 1 Gbps dedicated/unlimited per
| $143/month, it means you can use it 24x7.
|
| While that is true in theory, if you in practice saturate
| your bandwidth 24x7 by reselling it your contract will
| probably get cancelled (or at least that is the case for most
| "unlimited" contracts).
|
| I don't think you actually answered the questions of the
| parent.
| amadeuspagel wrote:
| What problem does this solve? Cloudflare is already free.
| mranton wrote:
| I'm collecting use cases when Cloudflare stopped service for a
| "non-web content" rule violation:
|
| https://news.ycombinator.com/item?id=34639212
| _joel wrote:
| Not for serious volumes, there's also a load more feaatures
| only paid and enterprise accounts get.
| yamtaddle wrote:
| Not if you hit large enough scale, or serve too much non-"Web"
| content.
| todd3834 wrote:
| How does this compare to IPFS? I was playing around with IPFS and
| it seems like a lower risk higher node version of this.
| Performance wasn't half bad either. I guess it depends on which
| node you end up downloading from and whether there was a nearby
| cache hit. P
| mranton wrote:
| IPFS is very slow. It's like a cold storage. Not good for web-
| cache at all. Though Cloudlare provides gateways for IPFS, and
| again, we have a CDN layer here. It would be nice to see IPFS
| performance numbers to compare.
| nektro wrote:
| If folks have the spare bandwidth they should run Tor nodes
| uncertainrhymes wrote:
| There are several red flags here, not the least of which is that
| a CDN isn't just about bandwidth, it is about _disk_. If a cache
| node doesn 't have the object, it has to go to origin to get it.
| That has a cost at origin, which most CDN customers want to
| minimize.
|
| Cache management (eviction, invalidation, etc) is done where? By
| the look of that diagram, you are actually doubling your origin
| traffic just to push to the cache after serving the original
| client request. Perhaps subsequent requests could get served from
| their 'peer' but the hit rate will be abysmal.
|
| Maybe there is more to it than they describe here, but I don't
| see how this can work. Are they going to rewrite the object in a
| way that doesn't require the host certificate?
| mranton wrote:
| >a CDN isn't just about bandwidth, it is about _disk_
|
| The disk performance - is the weakest part of the Peer. It is a
| part you can't guarantee any QoS on a cheap server. The idea is
| to keep popular cache on Peers in memory. Everything else we
| will host from a limited number of regular servers (belong to
| Farba), with high-performance disk subsystem.
|
| >Cache management (eviction, invalidation, etc) is done where?
|
| yes. You can invalidate either a single URL or a folder with a
| wildcard
|
| >you are actually doubling your origin traffic just to push to
| the cache after serving the original client request
|
| this is something we can proxy and do only one request per
| file. Current implementation was done for simplicity and
| reliability.
|
| >but I don't see how this can work. Are they going to rewrite
| the object in a way that doesn't require the host certificate?
|
| you can easily check this up. Please open the network console
| on our demo-page and see how it works:
|
| - we have two different certificates (one for the main site,
| for for the Peer)
|
| - everything works flawlessly
| uncertainrhymes wrote:
| Your example servers have 16-32 GB of memory, which is
| basically nothing. If you ever ramp to real world traffic the
| peer will be constantly evicting objects (if you use a normal
| LRU). Even if you backstop that with a mid-tier belonging to
| Farba to protect the origin, what kind of cache hit rate are
| you expecting at the peer?
| mranton wrote:
| >Your example servers have 16-32 GB of memory, which is
| basically nothing
|
| we just need to add more Peers and evenly distribute the
| workload.
|
| > what kind of cache hit rate are you expecting at the
| peer?
|
| I don't know. We have too many moving parts right now and
| can't predict anything.
| focusedone wrote:
| Ignorant question - How well does this scale? Are peers all
| necessarily running in major cloud environments?
|
| Do you imagine seeing peers in the form of smaller players with
| beefy home setups?
| mranton wrote:
| >How well does this scale? Are peers all necessarily running in
| major cloud environments?
|
| The network scales very well. The network is divided into
| multiple GeoZones. We can add remove these zones easily.
| GeoZones are not connected to each other, so there is no single
| point of failure. Actually the major cloud envs are the most
| expensive possible variants for Peers. Bare metal servers are
| much cheaper in our case.
|
| >Do you imagine seeing peers in the form of smaller players
| with beefy home setups?
|
| Home internet has a couple of disadvantages: - traffic is not
| "unlimited". Xfinity doesn't allow you more than 1 Tb in and
| out traffic per month - home internet is asymmetrical: you
| could have 1 Gbps in and only 5 Mbps out.
| focusedone wrote:
| So a small, regional ISP with a bunch of unused outbound
| bandwidth and spare rack space could be a good peer in this
| scheme, right?
| mranton wrote:
| correct
| yamrzou wrote:
| Related:
|
| P2P CDN: https://hn.algolia.com/?query=p2p%20cdn,
| https://hn.algolia.com/?query=peer%20to%20peer%20cdn
| mranton wrote:
| kind of. They are WebRTC-based CDN.
| datadeft wrote:
| >> At Farba, we are building a zero-cost CDN infrastructure where
| all edge servers belong to anonymous Peers.
|
| Zero-cost CDN for 2 USD/TB! Sounds like a great deal to me.
|
| On a more serious note, is price the problem with other CDNs?
| jsnell wrote:
| There's a bunch of really obvious objections to this idea around
| performance, privacy, economics, UX, and reliability.
| Impressively, the post did not manage to address any of them up
| front.
|
| 1. Performance. In terms of latency, this is adding an extra
| round-trip to each page request (to get the metadata). It's also
| likely going to load different resources from different servers
| (hostname, IP) preventing connection reuse. I guess they can't
| even start fetching data until the full page has loaded? And it
| feels like this scheme would make it much harder to reliably
| cache resources, since the browser will cache by URL while they'd
| struggle to make sure the URLs are stable across calls.
|
| 2. Privacy. Their threat model only addresses malicious peers
| changing content, not malicious peers trying to track users.
|
| 3. Economics. If running peer servers really was profitable, the
| company would do it themselves rather than outsource it. Given
| the "where can you get a server" section, it's not even that
| they're expecting this to just be running on spare capacity.
|
| The problem with the setup is that incentives of the peers are
| badly misaligned with the network, the clients, and the end-user.
| A peer wants to maximize traffic; the clients and end-users want
| to minimize it. So a peer would be incentivized to set cache-
| control headers to prevent caching, to increase traffic. Likewise
| a peer would be incentivized to only keep copies of the most
| accessed resources, to be able to serve as high a proportion of
| the traffic as possible with a given disk budget, while all the
| other stakeholders would like each file to be available just N+1
| times in each region.
|
| 4. UX. URLs will be really crappy compared to a proper CDN.
| Right-click copy a link to an image and send it to somebody?
| It'll initially be to some super-dodgy URL, and stop working in a
| day or two as the CDN node starts caching different content. And
| AMP showed that people do actually care about URLs.
|
| It's pretty hard to take this seriously.
| mranton wrote:
| >1. Performance. In terms of latency, this is adding an extra
| round-trip to each page request (to get the metadata). It's
| also likely going to load different resources from different
| servers (hostname, IP) preventing connection reuse. I guess
| they can't even start fetching data until the full page has
| loaded?
|
| fair point. We use same technique as lazy-load (you have to
| wait the full page has loaded).
|
| >3. Economics. If running peer servers really was profitable
|
| Uber doesn't drive taxis himself, airBnB doesn't own property.
| This is how Uber-economy works. We believe it works in the CDN
| industry as well. We just don't want to be another CDN
| provider.
| bhaney wrote:
| Completely skipping over the privacy concern doesn't seem
| like an amazing look
| tedivm wrote:
| Yeah this CDN is going to end up in ublock so quickly,
| especially since blocking it will just make the library
| fall back to the origin server.
| pwinnski wrote:
| Three things about "how Uber works":
|
| 1. It started out burning huge amounts of investor capital,
| basically transferring all of that money to drivers and in
| the form of subsidized rides, to riders.
|
| 2. It works today by a combination of the above and
| underpaying drivers, who generally make less than minimum
| wage when they manage to cover their completely costs at all.
| Despite this, prices have gone up dramatically, so that taxis
| are once again competitive on a price basis.
|
| 3. Uber was profitable in 2018, and not before nor since[0].
| Deeply, profoundly unprofitable. Why would anyone want to be
| the Uber of anything?
|
| 0. https://www.macrotrends.net/stocks/charts/UBER/uber-
| technolo...
| illiarian wrote:
| 1. Is basically price dumping using unlimited investor
| money and is blatantly illegal. But since it "this great
| big disruptive innovation unicorn", no one batted an eye.
| And still no one bats an eye in 3. :)
| [deleted]
| mranton wrote:
| >1. Performance.
|
| also please note:
|
| if you have 1 image on the page - we have to do 2 requests (1
| metadata, 1 download)
|
| if you have 50 images on the page - we have to do 51 requests
| only (1 metadata for all images, 50 downloads)
| jedberg wrote:
| > if you have 50 images on the page - we have to do 51
| requests only (1 metadata for all images, 50 downloads)
|
| But unless you return the same host for 50 objects, I'm going
| to have to do a bunch of DNS lookups too, right? And a bunch
| of SSL setups.
|
| Hopefully you at least make sure to try and return a single
| host for all the objects in a request.
| mranton wrote:
| >Hopefully you at least make sure to try and return a
| single host for all the objects in a request.
|
| our goal is to saturate the _User's_ network. Now it's hard
| to say how many Peers we need to meet this requirement. It
| looks like 2-3 Peers are enough.
|
| Also, when you have 50 Peers, you have to do 50 DNS/SSL
| requests in _parallel_. With current hardware, it's not a
| big deal.
| pwinnski wrote:
| > With current hardware, it's not a big deal.
|
| You should make this your marketing slogan.
|
| "Try our CDN, with current hardware the gross
| inefficiencies other CDNs spend a lot of time and effort
| eliminating are not a big deal."
| paxys wrote:
| 1 is something every CDN in the world already deals with. The
| metadata system ultimately needs to be its own little CDN, and
| can achieve the same level of performance (e.g. by deploying at
| the edge) just without the bandwidth costs.
|
| 2 is definitely an issue.
|
| 3 is a _maybe_. There 's definitely a model where people with
| access to cheap servers and bandwidth can get on board and not
| have to worry about the application-level details, finding
| customers, billing etc.
|
| 4 isn't an issue as long as you are selling the CDN
| specifically for website assets that don't need to have pretty
| URLs.
| jedberg wrote:
| > 1 is something every CDN in the world already deals with.
|
| Not exactly. For a normal CDN, your webpage HTML directs the
| client directly to the CDN server, which in theory already
| has the object. Only if it doesn't have the object does it
| need to go get it, but there isn't an extra lookup. The CDN
| does it's direction magic via DNS or IP routing, not with
| double lookups.
| jsnell wrote:
| 1. None of the issues I listed affect traditional CDNs
| though.
|
| 3. Why would the people running a peer have cheaper access to
| servers and bandwidth than a company specializing in that
| business? Everything about this business should have huge
| economies of scale (buying hardware, system administration,
| transit/peering).
|
| The only situation where I can see a distributed model
| winning out is if the entity running the peer does not pay
| for resources but gets them with fraud or theft, e.g. paying
| for cloud servers with stolen credit cards.
| danielrhodes wrote:
| The value of this largely depends on what you're optimizing for.
| If your constraint is you need a lot of cheap bandwidth but don't
| care that much about latency, there are many places to find it
| (find a cheap host, set up Varnish, you're good to go).
|
| However, a lot of people use CDNs because they are fast. They
| want the latency to be ultra low and they want it to be reliable.
| This is where the author's solution falls short: CDNs can get
| premium quality bandwidth because they have more scale than their
| customers. Sure you can find servers from various hosts in lots
| of different data centers, but you might not get the latency or
| quality of bandwidth you were looking for. This can be critical
| for some applications like streaming or serving images.
|
| If your constraint is speed, you almost have to go with a CDN
| (even if you are huge). If it's volume, you are in a better
| position. If your volume is small, you might as well go with
| something off the shelf like Cloudfront or Cloudflare. If you are
| ultra huge like YouTube or Netflix, then you have to peer
| directly with ISPs.
___________________________________________________________________
(page generated 2023-03-15 23:03 UTC)