[HN Gopher] Cloudflare Gen 12 Server: Bigger, Better, Cooler in ...
___________________________________________________________________
Cloudflare Gen 12 Server: Bigger, Better, Cooler in a 2U1N Form
Factor
Author : todsacerdoti
Score : 70 points
Date : 2023-12-01 18:46 UTC (4 hours ago)
(HTM) web link (blog.cloudflare.com)
(TXT) w3m dump (blog.cloudflare.com)
| fasteo wrote:
| >>> A single fan running at full speed consumes about 40W
|
| I thought fans consumed like 10 times less
| temp0826 wrote:
| A normal fan in your home PC yes. Not the 40mm jet engines you
| only find in data centers.
| buildbot wrote:
| Yeah some of these fans are capable of lifting off helicopter
| style...
| amluto wrote:
| I'm a bit surprised the big datacenter / OpenCompute buyers
| haven't moved toward a form factor that's more square. Big,
| high-power, quiet desktop systems have been using monster
| fans for many years, and ISTM one could do better with
| systems that are maybe 4U high but only half width, enabling
| a proper fan in back. Or maybe even a form factor that's
| packs a lot of individual machines in front of a full width
| truly immense fan (19" or more!).
| jeffrallen wrote:
| In 1999, Tellme was using square servers from Dialogic, a
| vendor of PCI Telco cards.
| Voultapher wrote:
| Oxide Computer has for their systems. Could be a trend.
| wmf wrote:
| Facebook/Meta has been using 2OU (which is slightly taller
| than 2U) for years. Oxide also uses 2U.
| buildbot wrote:
| I guess it must be, but does the performance per watt make sense
| going from 350 to 500W? That's a 42% increase in power! I suppose
| the # of cores per watt per u makes this solution make sense,
| especially with the 1u cooling overhead, but it seems like it
| might be closer to a wash than one might think initially.
| mastax wrote:
| The 500W is an estimate for the next generation CPUs so we have
| to speculate, but yes the performance per watt should make
| sense unless those products are awful. Thus far every EPYC
| generation has been substantially better performance/watt but
| the top-end power draw has gone up because the number of cores
| has gone up. EPYC 3 (Milan) maxed out at 64 cores but EPYC 4
| maxes out at 96 (Genoa Zen4) or 128 (Bergamo Zen4c). So you get
| twice as many cores, faster cores, and much less than twice the
| max power draw (~240W > ~360W). The highest core count CPUs
| tend to be the most efficient as well because you amortize the
| relatively static power draw of the IO-die (and the other parts
| on the motherboard) and because they tend to have lower more
| efficient clock frequencies.
|
| It's pretty safe to assume the situation will be the same with
| next generation EPYC 5 CPUs with maybe 192 or 256 cores, higher
| power draw, but more efficiency.
| buildbot wrote:
| Makes a lot of sense, thank you!
| soggybread wrote:
| I'm surprised. They mention that cooling is only going to get
| worse and that fans consume a lot of power, should liquid cooling
| not be something to look at? Maybe they did and found it either
| unnecessary or improbable to do at a colo depending on how many
| servers they have at the data center it might not be worthwhile
| but if they have 10 or 20 servers surely it would be worth while
| no?
| yaxe1 wrote:
| My guess is liquid cooling isn't reliable enough. Something is
| going to leak eventually.
| bluedino wrote:
| You need tanks, plumbing, pumps, coolant...if the datacenter
| doesn't have it available it's not a small undertaking.
|
| Example: https://www.supermicro.com/en/solutions/liquid-
| cooling#solut...
| soggybread wrote:
| You're right and while it might take up valuable rack space I
| would imagine there are some rack-based solutions that have
| most of the stuff included so that you don't have to rely on
| the dc providing it, pretty sure ServeTheHome has done a
| couple reports on them
| amluto wrote:
| There are at least a couple vendors with self-contained
| liquid cooling systems, kind of like the "all-in-one" systems
| available for desktop and gaming use. I don't know how
| effective they are.
| aftbit wrote:
| AIUI this is just starting to take off. Look at Supermicro's
| 2U4N and 1U2N offerings, which need to use liquid cooling to
| reach that density. I know Alibaba has been doing vertical
| mounted servers in a non-conductive oil bath for a while. I bet
| their generation 13 servers have liquid cooling.
| eastdakota wrote:
| If you're building your own facility, there's a lot of things
| you can do with design. In our case, these servers need to go
| into other people's facilities (e.g., ISPs around the world) so
| we need to live within their constraints. Things like liquid
| cooling either wouldn't be possible or would add significant
| complexity to install and maintain.
| alberth wrote:
| I might have missed it but ...
|
| What CPU did they end up selecting?
|
| Cores/RAM/Disk ... even GPU since they mention the importance.
|
| TL:DR - newer CPUs TDP is so high, they needed to change from 1U
| to 2U and that changes economics at a macro level per rack (since
| you're density now halves).
| wmf wrote:
| They didn't say what CPU they're using. Sapphire/Emerald Rapids
| sucks so it's either Genoa, Bergamo, or Ampere One.
| jeffrallen wrote:
| Anyone who has been following Oxide knows the physics of why big
| fans rock.
| alberth wrote:
| It's simple math of pr^2
|
| 40mm fan can push ~5,000 sqmm of air
|
| 80mm fan can push ~20,000 sqmm of air
|
| Doubling the fan size results in 4x more airflow.
| aftbit wrote:
| It's very interesting to see this. I wonder what Amazon,
| Facebook, Google, and Microsoft are going to do in the next few
| generations. Will they be moving to 2U1N as well? Or do they have
| liquid cooling in mind? Or maybe they'll be keeping lower power
| budgets? Or building data centers that run cooler and support
| higher power density (probably not great for the environment).
|
| I knew fan power usage could be a substantial amount of a
| server's power draw, but I didn't realize they would need 320W to
| move enough air to cool 500W of dissapation.
| nagisa wrote:
| The major cloud vendors have their own data centers & do not
| need to deal with space or power limits at a rack unit. They
| have the entire data center worth of space and power supply to
| work with.
| jonatron wrote:
| If the data centre is owned by digital reality, equinix,
| telehouse, etc, is it really theirs?
| https://aws.amazon.com/directconnect/locations/
| arrakeenrevived wrote:
| Direct Connect locations are not the locations where the
| main region data centers are. AWS has a network link from
| their DC to Equinix, etc to make it easier for customers
| who are in Equinix's colo to make a direct connect to the
| AWS network.
| toast0 wrote:
| All data centers have space and power/cooling limits. If
| you're in someone else's DC, they are usually going to quote
| you as per-rack power, but if you really want you can
| probably convince them to let you use 2 racks worth of power
| in one rack and hold a second rack empty. In owned and
| operated DCs, it's kind of similar, if the DC was built a
| while ago, as server power density has increased, a
| significant portion of the floor space may be unusable for
| racks.
|
| 10-20 years ago, 1U and blades made a lot of sense because
| space was a major constraint; now 2U or larger can make sense
| because 1U has a lot of tradeoffs for size, and power is the
| bigger constraint.
|
| If space density is still needed, 2 nodes in 2u or similar
| narrow but tall configurations provide a more square profile
| that allows generating airflow with larger fans which is more
| efficient than the 40mm fans needed in 1U systems.
| folmar wrote:
| OVH is doing liquid cooling to some extent since 2003.
| https://blog.ovhcloud.com/water-cooling-from-innovation-to-d...
| infogulch wrote:
| Yes it's insane to waste 70% of the CPU power again on cooling
| the CPU (read: cutting into power density).
|
| Oxide's rack uses 80mm fans that run at less than half the
| stock minimum speed to reduce this waste and run super quiet:
| https://oxide.computer/blog/the-cloud-computer
| mcronce wrote:
| It's nuts how much energy server fans can use.
|
| I recently picked up an R730xd. By default, a non-Dell-branded
| PCI-E device will force the fans to ~16000RPM, but there's an
| IPMI command to disable that and go back to regular,
| temperature-driven, fan control. Total server power came down
| by 70W after issuing that command.
| jjtheblunt wrote:
| They don't seem to mention why EPYC and not Ampere's new
| architecture.
|
| Is the power and performance / core count comparable, so it's
| more momentum preserving to stay with x86_64?
| robin_reala wrote:
| The post is specifically talking about their x86 servers:
|
| _It 's nearly time for another refresh to our x86
| infrastructure, with deployment planned for 2024._
| eastdakota wrote:
| We use both. This post was about the x86 version of our edge
| servers.
| nagisa wrote:
| Would love one of those decommissioned 7713s to upgrade my own
| server with!
| yardie wrote:
| I bought my last 1U server a few years ago. The space savings I
| thought I was gaining was quickly diminished by the fact that we
| weren't going to be able to fill a cabinet simply due to running
| over our power envelope. It was going to require a bigger
| circuit, an electrician to install it, a change in our cool
| billing, etc.
|
| So, these newer CPUs while being a lot more powerful also draw a
| lot more power as well. In the end the squeezing 1000s of CFM
| through a 1.75" aperture wasn't worth the headache.
|
| I'm glad to see cloud providers are still using datacenter
| standard 1U chassis. I assumed everyone had moved to Open Compute
| rack dimensions 537x44mm.
|
| The neighboring cabinet to ours has a 3x Walmart box fans. I
| thought that was pretty innovative. Just use big silent, slow,
| fans instead of these tiny, noisy airhorns.
| NicoJuicy wrote:
| Nice to see some of the hardware details behind cloudflare!
|
| One that I found nice too, Cloudflare is able to quickly install
| GPU's everywhere, because for over 6 years, they didn't utilize
| 1/6th of the slots for that purpose.
|
| https://www.fool.com/investing/2023/11/22/cloudflare-started...
|
| Would have loved to see that information from their blog, instead
| of their earning call :p
| antisthenes wrote:
| 1U is essentially obsolete due to the power/heating envelope of
| modern server CPUs that push 32-64 cores easily.
|
| I just don't see a rack full of 1U being anything but a giant
| liability in terms of trying to cool it and the following
| reliability issues.
| Am4TIfIsER0ppos wrote:
| What does "1N" mean?
| wiml wrote:
| It means there's one node in that chassis, eg here's a 2U4N
| product: https://www.servethehome.com/dell-emc-
| poweredge-c6525-review...
| Dylan16807 wrote:
| Seems to be right, but as far as I can tell almost nobody
| uses this terminology and it's pretty rude of the article not
| to explain.
|
| Like, this article is one of the top results for 2U1N, plus
| some dell pages and random jank.
| tekacs wrote:
| I mean this very positively, not condescendingly and I'm very
| much not trying to preach or push, but I've found that this
| sort of question is exactly the sort that I benefit a lot from
| handing to GPT to answer for me:
|
| https://chat.openai.com/share/37774e90-a271-4f6c-a5d3-3937e9...
|
| I was curious too because I hadn't heard '1N' in a long time so
| I did the above and you might like the answer it gave me too!
| Am4TIfIsER0ppos wrote:
| Based on your sibling's comment chatgpt has it wrong.
| nemothekid wrote:
| > _N: This refers to the redundancy of the power and cooling
| systems. "N" stands for "Need" or "Normal." In data center
| design, "N" is used to denote the capacity required to
| support the IT load. "1N" implies a single path for power and
| cooling, with no built-in redundancy. In other words, each
| component necessary to support the IT equipment is present,
| but there is no backup if one of these components fails._
|
| I'm pretty sure this is wrong. N means "node", so a 2U4N is a
| 2U chassis that houses 4 servers. I'm pretty ambivalent for
| this reason for using chatgpt for research in things I cannot
| easily verify. I will admit this is a scarily convincing
| hallucination.
| flumpcakes wrote:
| I think this is completely wrong from ChatGPT...
| solardev wrote:
| Cloudflare's blog posts are always so comforting. It's like
| listening to Grandpa tell war stories next to the fireplace.
___________________________________________________________________
(page generated 2023-12-01 23:01 UTC)