[HN Gopher] High-speed 10Gbps full-mesh network based on USB4 fo...
___________________________________________________________________
High-speed 10Gbps full-mesh network based on USB4 for just $47.98
Author : fangpenlin
Score : 218 points
Date : 2024-01-15 17:26 UTC (5 hours ago)
(HTM) web link (fangpenlin.com)
(TXT) w3m dump (fangpenlin.com)
| johng wrote:
| This is neat.
| riku_iki wrote:
| 15c/kwh estimate in California looks very far from the current
| situation.
| jms703 wrote:
| Yeah, I think the website the author linked to has old data.
| It's 3x that in California.
| kube-system wrote:
| It is 26.72 cents per kwh as of the latest official numbers
| which are from October 2023.
|
| https://www.eia.gov/electricity/monthly/epm_table_grapher.ph.
| ..
| riku_iki wrote:
| Not clear who are those "ultimate customers".
|
| Also, in case of server hardware, it is excessive over
| basic tier electricity consumption, so it will hit those
| 50c/kwh.
| kube-system wrote:
| > Not clear who are those "ultimate customers".
|
| The first column is residential customers.
| riku_iki wrote:
| It is residential "ultimate" customers.
|
| My current bill for initial tier is: 42c/kwh delivery +
| 15c/kwh generation, and even higher over initial
| consumption.
|
| I guess I am not "ultimate" customer.
| kube-system wrote:
| 'ultimate customer' means the customer using the power,
| which you are.
|
| The reason you don't pay the price listed on that page is
| because you are only one customer. You are not the
| average customer.
| riku_iki wrote:
| The fact that I live in the center of densely populated
| area and serviced by major provider who also holds
| monopoly, makes me think there is a chance that table's
| numbers are just not what you think. They may include
| just generation cost, and not delivery, or there is
| something about "ultimate customer" definition.
|
| Also, there were several hikes since Oct in my
| understanding.
| kube-system wrote:
| I read form EIA-861, which is where the data comes from.
|
| Looks like sales to "ultimate customers" means it
| excludes electricity that was not sold, electricity that
| was sold to resellers, energy lost, etc.
|
| The form also collects information about revenue from
| delivery "and other related charges"
|
| 42c/kwh delivery sounds insane. I couldn't find much data
| about average delivery rates, but I plugged in a few
| counties here and it looks like many areas have delivery
| rates significantly lower than that:
| https://www.cpuc.ca.gov/RateComparison
| riku_iki wrote:
| > I plugged in a few counties here and it looks like many
| areas have delivery rates significantly lower than that:
| https://www.cpuc.ca.gov/RateComparison
|
| That page says numbers are year old, I entered my zip
| there and it says delivery is 20c/kwh for me.
|
| If you don't know the story: PG&E is delivery monopolist
| in CA, with exception of some places with local
| powerplants. It was found guilty in causing wildfires,
| lost in court and need to pay $XXB damages, which it now
| shifts to customers through multiple rate hikes in 2023
| and more coming in 2024.
|
| 42c/kwh is initial tier cost, whatever is over limit will
| be charged at 50c/kwh for just delivery.
| nickstinemates wrote:
| Incredible. Traditional 10Gb network gear is very expensive to
| buy and to operate. $50 is a lot cheaper.
| Palomides wrote:
| not really true, a similar setup with 40Gbe (three nodes, no
| switch) would run you less than $100 for pcie cards and cables
| hamandcheese wrote:
| The only NICs I see in that price range are old Mellanox
| cards.
|
| Intel NICs are 5-10x that price. I'm not sure why, but my
| suspicion is that it has to do with driver support and/or
| interoperability.
| Palomides wrote:
| I run mellanox connectx3 cards, they work immediately with
| no extra drivers on windows 10/11 and every linux I've
| tried
|
| mellanox is/was quite good at getting code upstreamed
|
| maybe I need to do my own blog post about my pile of
| computers...
| TacticalCoder wrote:
| > maybe I need to do my own blog post about my pile of
| computers...
|
| Yes, several of us love to read about that! I haven't
| switched to 10 Gbit/s yet...
| thefz wrote:
| > I run mellanox connectx3 cards, they work immediately
| with no extra drivers on windows 10/11 and every linux
| I've tried
|
| Own three, can confirm for Windows and Linux.
| Hikikomori wrote:
| Mellanox works fine.
| hamandcheese wrote:
| Gotcha. Why the price discrepancy then?
| Hikikomori wrote:
| Are you comparing new Intel cards to old mellanox cards
| on ebay? If not idk why, I have not compared them myself,
| some feature maybe? Cost doesn't always make sense
| either.
| olavgg wrote:
| And with Mellanox you get working RMDA/ROCE.
| nickstinemates wrote:
| Can you provide links? I'll upgrade my home lab from 10g to
| 40g immediately at that price.
| Hikikomori wrote:
| Mellanox connect-x 3 probably.
| willglynn wrote:
| In 2024 I would suggest deploying 2x25G instead, via e.g.
| MCX4121. Pricing is similar (<$30 NICs), but:
|
| * 2x25G throughput is higher than 40G,
|
| * 25G latency is lower than 40G,
|
| * you can use 25G ports as 10G ports, and
|
| * you can use DACs to connect 4x25G <=> 100G
|
| That last point is particularly relevant given the
| existence of switches like the Mikrotik CRS504, providing
| 4x100G ports on 25W.
| Palomides wrote:
| those are all reasonable points, if I were doing mine
| again I would spend a little more and go up to 100gbe
|
| if you run all older mellanox gear the cx3 can do the
| kinda nonstandard 56gbe as well
| bombela wrote:
| What are you doing that you need 100gbe?
|
| I am still on 1gbe... I guess I don't transfer anything
| bigger than a few GiB time to time.
| Palomides wrote:
| cx353a and cx354a, prefer the fcbt models, but you can
| reflash them all to fcbt
| crotchfire wrote:
| What does fcbt stand for and why would I want it?
| Palomides wrote:
| the same hardware was sold at different prices with
| different capabilities enabled based on the exact model
| variant, stuff like higher speeds and infiniband support
|
| you can see them all in the datasheet, I believe fcbt is
| the one with all the stuff enabled
| zamadatix wrote:
| This is a great use of the Thunderbolt hardware that comes built
| into consumer devices.
|
| For throughput try sticking "--parallel 4" or shorthand "-P 4" on
| the iPerf3 command and see if total throughput changes.
| toyg wrote:
| _> I don't understand why it can only hit 11Gbps [...] other
| people building similar networks were able to hit 20Gbps. [...]
| So it could simply be that the machine only supports up to this
| speed [...] it might compete with [Intel 's] network controllers,
| so they capped the speed_
|
| I'm bad at low-level networking, but could it be a routing issue?
| Effectively every machine is also a router, so there might be
| some wasting going on.
| arghwhat wrote:
| The machines are not configured as "routers" in that example.
| It is just a point-to-point layer 2 network, where each network
| link serves exactly one possible destination. As simple as it
| can be.
| r1ch wrote:
| It's measuring single TCP connection performance which is
| already difficult to optimize. With jumbo frames and tuned
| buffer sizes I'd expect it to get higher, but it will likely be
| serialized to a single core's worth of CPU. Using multiple
| connections should give a better representation of available
| link bandwidth.
| adrian_b wrote:
| IIRC the specification of Thunderbolt from Intel (which was
| inherited by USB 4) limits the Ethernet emulation mode to 10
| Gb/s.
|
| Why did they set a limit so low is not known, but the
| supposition made by another poster that this is a market
| segmentation feature may be right, because such policies have
| always been typical for Intel.
| jms703 wrote:
| Neat.
|
| I stopped reading at "California's average residential
| electricity rate is 15.34 cents per kWh" and checked my
| electricity bill. Here in California, I'm paying 40.82 cents per
| kWh. The website https://www.electricitylocal.com/ seems wildly
| off.
|
| Edit: I did read the entire article. Just stopped to check what I
| was paying.
| alchemist1e9 wrote:
| At that rate how are there any data centers in California?
| diggan wrote:
| Not everyone has the same contracts and contacts.
| kube-system wrote:
| Industrial and commercial users of power almost always get
| different pricing than residential rates in the US.
| alchemist1e9 wrote:
| Different yes but not 4x different. I know in Illinois
| standard generic commercial is around 9 cents and
| residential is like 12 cents.
| zamadatix wrote:
| Probably just older data based on looking at the trends
| https://data.bls.gov/timeseries/APUS49E72610?amp%253bdata_to...
| and
| https://data.bls.gov/timeseries/APUS49A72610?amp%253bdata_to...
|
| That said, mentioning it is one thing (it makes the case for
| low power devices even stronger) but why stop reading anything
| after the first error you identify? That doesn't guarantee you
| only get accurate information it just guarantees you'll miss
| out on good information, like the cheap high speed connectivity
| the article is actually demonstrating.
| jms703 wrote:
| I meant that I stopped reading to check.
|
| I read the entire article.
| bhouston wrote:
| I saw this mini-computer recently, the UM790 Pro, and I almost
| bought one as well. The AMD Ryzen CPU, AMD Ryzen 9 7940HS CPU,
| has a rating of ~30,000 on the CPU Benchmarks page
| (https://www.cpubenchmark.net/high_end_cpus.html) which is quite
| high end for the size/cost.
|
| The only caveat is that I am running a 10Gbit ethernet network at
| home and it wasn't that costly to setup. A 10GigE switch costs
| around $500 CAD right now and that is all you need.
| Liftyee wrote:
| > A 10GigE switch costs around $500 CAD right now and that is
| all you need.
|
| Surely your machines all need 10GbE NICs as well? Admittedly my
| hardware isn't the newest (2.5GbE at most), but from a quick
| search 10Gig PCIe cards are around $150 each. Meanwhile 10-20
| Gbps USB3.0 is reasonably common already. Though, using
| Ethernet still has many other advantages over running USB C
| cables everywhere.
| adrian_b wrote:
| There are also adapters from USB4 to 10 Gb/s Ethernet, for
| the case when it is desired to use a switch or long cables.
| bhouston wrote:
| My MacMini M2 has one built in. My other machines, yeah, I
| did buy some, which go for around $100 CAD on Amazon (e.g.
| TX401). Ethernet is advantageous because my house is wired,
| so I can have my switching a central location.
| mrkstu wrote:
| On actual network gear you'd typically drop the interfaces into a
| subnet/broadcast domain, but the blogger here has put everything
| in /32s and it looks like it's creating a peer connection without
| a shared domain.
|
| Any particular advantage/disadvantage to doing it this way?
| ComputerGuru wrote:
| It doesn't scale. Number of connections/hardware is exponential
| to the number of nodes.
| cma wrote:
| Isn't it just N^2?
| ComputerGuru wrote:
| Absolutely. Sorry, I meant quadratic but had a brain fart.
| dfox wrote:
| In this case the primary reason for configuring it as point to
| point links is that it matches the hardware topology. In theory
| there is a standards compliant way to create an ethernet-style
| network from this, but there are two critical software
| components to that that are not implemented in mainline kernels
| (namely raw Ethernet over TB and SPB).
|
| The main disadvantage inherent in the physical topology is that
| you are always going to do switching/routing decisions on the
| CPU. But as long as you use that as a virtualization platform
| or run k8s on that you are going to do that anyway and the
| additional overhead is probably irrelevant. (This assumes the
| full mesh topology, which with this hardware is not scalable
| over 3 nodes, might be to like 5 with somewhat more expensive
| consumer grade HW and is not really scalable to more than 8
| nodes due to the sheer amount of cabling required)
| gaudat wrote:
| USB4/thunderbolt is a magical protocol. Turns out the fastest way
| to move data between 2 modern PCs is to connect their thunderbolt
| ports with a USB-C cable. The connection shows up as a ethernet
| port on Windows and I can easily saturate a SSD with its 1GB/s+
| transfer rate. And it just works (tm). Reminds me of firewire on
| these 20 yo Macs.
|
| And What happen if you wire both ports together on the same PC?..
| Do you get a broadcast (thunder) storm?
| jwells89 wrote:
| Target disk mode over FireWire was magical back in the day.
| Nothing like turning someone's laptop into an oversized
| external hard drive to rescue data or get a borked OS X
| installation booting again.
| xattt wrote:
| FireWire had IP networking stacks in Windows and OS X in
| 2007. You could daisy chain a bunch of devices together to
| share a network connection.
| FirmwareBurner wrote:
| Sure but it was very expensive compared to USB and Ethernet
| so Firewire never caught on with mainstream conumers other
| than some niche cases like camcorders.
|
| Thunderbolt was also expensive, which is why adoption was
| limited, but it's becoming more maisntream since Intel and
| Apple have been pushing it in the last years, adn
| piggibacking over USB-C makes it an easy sell comapred to
| requireing a separate connector like firewire.
|
| Still, thunderbolt peripherals are way more expensive than
| USB ones, so like Firewire before, use is still more in the
| enthusianst/professional space.
| GeekyBear wrote:
| > it was very expensive compared to USB and Ethernet so
| Firewire never caught on
|
| Compared to Gigabit Ethernet back in that time period?
| Firewire was a huge bargain.
| FirmwareBurner wrote:
| 100 Ethernet, not gigabit, was more than enough for what
| average consumers had to transfer back then.
| RachelF wrote:
| Yeah, we built a similar system a generation ago using
| FireWire on x86 and Linux.
|
| FireWire at 800Mbps beat Gigabit Ethernet in terms of latency
| for a rather hard real-time system.
| rabi_molar wrote:
| I remember the "good ol days" when I would always opt for a
| FireWire audio interface for music production and live
| performance over USB for exactly this reason. I'd get way
| better latency and stability.
| justnotworthit wrote:
| Any ideas what im I supposed to do with the firewire
| mixer i bought 14 years ago?
| wmf wrote:
| Get a Thunderbolt 3 to Thunderbolt 2 dongle and a
| Thunderbolt 2 to Firewire dongle?
| jwells89 wrote:
| Even now an ancient FW400 HDD enclosure of mine is less
| flaky than a lot of USB storage I've used.
| Fnoord wrote:
| Way before that there was SCSI.
|
| The low cost is price. Asus for example with their ASUS
| ThunderboltEX 4 allows you to have TB4 via PCIe card.
|
| The neat thing about USB4 was same as PATA and later SATA:
| widely and relatively cheap available in consumer hardware.
| SCSI and FireWire were technically superior but were neither
| cheap nor widely available.
|
| Oh and I don't know about SCSI but FireWire was actually a
| security risk.
| vardump wrote:
| You need more like 50 Gbps to saturate a modern Nvme SSD.
|
| 10 Gbps doesn't come close.
| genman wrote:
| There was a moment with the spinning rust drives where it
| would have made sense to have storage in a networked device
| and not locally but now it rarely makes sense unless an
| incredibly fast interconnect could be used.
|
| Of course this example is still interesting and cool.
| arglebargle123 wrote:
| For a couple of years I had a Linux NAS box under my desk
| with like 8 Samsung 850 pros in a big array connected to my
| desktop over 40GbE. Then NVMe became a common thing and the
| complexity wasn't worthwhile.
| NavinF wrote:
| Infiniband can match NVMe bandwidth and its latency is
| similar. Newer network cards can also present a NVMe-oF
| drive as a local NVMe drive
| mgerdts wrote:
| 25 Gb Ethernet roughly matches a PCIe Gen 3 NMVe drive's
| max throughput 50 Gb will match Gen 4. These are RDMA
| capable.
|
| It seems 25Gb dual port Mellanox CX-4 cards can be found
| on eBay for about $50. The cables will be a bit pricey.
| If not doing back to back, the switch will probably be
| very pricey.
| genman wrote:
| It can match a single drive.
| genman wrote:
| Yes, that was my point - 10Gbps is just way too slow,
| even full Thunderbolt bandwidth can be easily saturated
| in raid configuration - NVMe are just incredibly fast.
| nine_k wrote:
| 10 Gbps does not, but 10 GBps, as written above, is 80 Gbps,
| matches your estimate.
| vardump wrote:
| I've tried the same method and it's about 10 Gbps, not 10
| GBps.
| ThePowerOfFuet wrote:
| Sounds like you were actually using USB 3.1 Gen 2 or USB
| 3.2 Gen 2[x1], not Thunderbolt 4.
| bhaney wrote:
| Thunderbolt 4 can't do 80Gbps either
| znpy wrote:
| > And What happen if you wire both ports together on the same
| PC?
|
| A literal loopback interface? Two of them, most likely.
| ComputerGuru wrote:
| > The connection shows up as a ethernet port on Windows
|
| Do you know if this is the case for all thunderbolt generations
| (speed differences aside)? Does it apply to thunderbolt using
| mini DisplayPort too or only over USB PHY?
| vineyardmike wrote:
| I think it works for older thunderbolt too. Been years since
| I tested it though.
| chx wrote:
| It was possible on Thunderbolt 2 , on the Mac side OS X
| Mavericks enabled it, this Intel whitepaper from 2014 talks
| about the Windows side. https://www.thunderbolttechnology.net
| /sites/default/files/Th...
|
| https://www.gigabyte.com/Press/News/1140 this doesn't mention
| it and the Intel whitepaper specifically requires TB2 so I
| would guess TB2 was it.
| raffraffraff wrote:
| Pity Thunderbolt 2 is basically non-existent nowadays. I
| have a few Macbook Pro 13 (2015) and I'd love to be able to
| use the thunderbolt 2 ports, but peripherals were too
| expensive and the standard short-lived. Try finding a
| thunderbolt 2 dock anywhere. Filter out all the false-
| positives (USB-C docks) and the are maybe 10 total on ebay,
| and they're stupidly expensive for 6+ year old used
| devices, most without cables or power supplies. Such a pity
| because they can really extend the useful life of those
| laptops.
| davkan wrote:
| My TB3 mesh network shows interfaces as thunderbolt0 etc.
| this is on Linux using thunderbolt _net from the kernel.
| Latency is worse than regular twisted pair Ethernet.
| bobim wrote:
| Dang, that wouldn't play nice for MPI then.
| davkan wrote:
| I was seeing 1-1.5ms latency using linux bridges for the
| mesh. Not a huge issue for ceph replication but
| significantly more than switched lan. It may be possible
| to get it lower with routed instead of bridged but my
| understanding is thunderbolt_net on Linux is not perfect
| in that regard.
| samstave wrote:
| WTF!
|
| I have two identical machines that I need to do this with...
| lemme test it and see.
|
| ...
|
| ARGs usb-3... so nopes. (How FN lame is that)
| SloopJon wrote:
| I've been playing with Thunderbolt networking over the past
| week with mixed results. I can get 16Gbps between a couple of
| Macs. Between a Mac and a PC running Windows 10 I get similar
| speeds in one direction, but less than 1Gbps in the other
| direction.
|
| In terms of scaling this to multiple hosts as the author does,
| I've read that it is possible to daisy chain, or even use a
| hub, but it doesn't strike me as the most reliable way to build
| a network. For an ad hoc connection, though (like null modem
| cables of yore), it's a great option.
| phyzome wrote:
| > The connection shows up as a ethernet port
|
| OK but what then? I've had ethernet ports on my computers since
| I can remember, and that hasn't magically allowed me to
| transfer data back and forth just by plugging a patch cable
| into both machines. What software is at work here?
| bpye wrote:
| I'm guessing if they're using Windows it's boring old SMB?
| ThinkBeat wrote:
| My problem with setups like these is storage. Not much room for
| expansion in the small boxes and I am not thrilled with external
| drives. Still had a QNAP or other NAS and hope to get one with
| compatible USB4 port
| notorandit wrote:
| Maybe he can get 20 Gbps between 2 servers. You get half because
| of USB limitations (2 ports on the same hub).
| rahimnathwani wrote:
| average residential electricity rate is 15.34 cents per kWh
|
| This didn't seem right, as I pay more than double that here in
| San Francisco. (I calculated $0.35/kWh by dividing the total I
| paid for electricity generation and delivery, and dividing it by
| the number of kWh consumed.)
|
| The linked page cites data from over a decade ago (2012).
| mlyle wrote:
| https://www.globalenergyinstitute.org/2022-average-us-electr...
|
| Within the state, there's huge variation. The average is around
| 25 cents. E.g. if you live in Santa Clara, you pay 16.6 cents
| per kilowatt hour to Silicon Valley Power, while all
| surrounding cities pay 45 cents-ish.
|
| https://www.siliconvalleypower.com/residents/rates-and-fees
| krallja wrote:
| Mine is currently 9.4C//kWh, no TOU, net metered. PG&E is the
| problem, not the data.
| delfinom wrote:
| 32 cents / kWh here in NY.
| throwaway-blaze wrote:
| Where in CA roughly is this?
| rahimnathwani wrote:
| I was responding to what OP said about _California_ rates.
| belval wrote:
| > This didn't seem right, as I pay more than double that here
| in San Francisco.
|
| Their data might be a bit outdated but the December 2023
| average is $0.168/kwh according to [1].
|
| [1]
| https://www.bls.gov/regions/midwest/data/averageenergyprices...
| rahimnathwani wrote:
| OP was talking about California, not the entire US.
|
| The page you linked shows four California cities, each with
| Dec 2023 rates over $0.27/kWh.
| JumpCrisscross wrote:
| Wow. I guess I won't get mad at my guests for running the
| heat 24/7 at 5C/ per kWh.
| jtriangle wrote:
| That whole section is BS in general. The server linked isn't
| going to use 1KW of power ever, not on the worst of days. The
| only real point that they're making is that "it's loud" which
| is true, very true, but, it's designed to run in a rack away
| from humans..
|
| While their solution is more livable for them, the hardware is
| vastly inferior for actually hosting serious services on, and
| they don't seem to understand that because they're software
| guys who're getting away with it.
| g8oz wrote:
| Given the electricity rates he mentioned, what would you
| estimate the monthly running cost to be?
| alphabettsy wrote:
| His numbers are probably 8x higher than actual. So maybe
| $8-15/mo for electricity for one of the old servers, not
| $100.
| tyrfing wrote:
| Depends on load. Somewhere around $20-40? Assuming idle
| around 80-120w, and another 100w for the e-waste drives in
| their pictured listing.
| szundi wrote:
| It will eat much more than 80W though
| pstrateman wrote:
| The average rate is only ~15 cents per kWh because of places
| like SF that are effectively just scamming the consumers.
|
| Electricity in free markets is more like 5-8 cents per kWh.
| NavinF wrote:
| Yep electricity prices in CA are set by the gov't. Market
| rates are typically 10 cents/kWH
| Fb24k wrote:
| I still remember connecting two MS-DOS computers together with a
| parallel cable, that was the easiest way to do "large" file
| transfers there for a while (used a program called LapLink,
| iirc)...
| farkanoid wrote:
| Laplink! That part ot my brain hasn't activated in decades...
|
| My favourite was bootstrapping using 'COPY /B COM1: C:LL3.EXE'
| to get the Laplink executable to the target machine over
| Serial, when you didn't have a spare floppy.
| myself248 wrote:
| You could also run PLIP over the same cable. It's like SLIP
| (which is like PPP) but much faster.
|
| Where Laplink isn't really a network, just a file transfer
| thing that requires Laplink to be running on both PCs, PLIP is
| a network driver that lets you do all the usual network things
| over the connection.
|
| And since PCs can have up to 3 parallel ports before things
| start getting stupid, it's pretty straightforward to have a row
| of machines with PLIP links going both ways, bridging or
| routing the interfaces. Or, do PLIP-SLIP-PLIP-SLIP without
| adding any ports, and you could have a functional-but-brittle-
| and-slow network for pennies.
| TacticalCoder wrote:
| > You could also run PLIP over the same cable.
|
| I was running that in the nineties. My main desktop, running
| Linux, and an ultra old, ultra crappy laptop running Linux
| too. They'd be connected using PLIP and the desktop, more
| powerful, was running its own X server but also applications
| for that were running on the laptop's X server.
|
| So my brother and I could both be using Netscape to surf the
| net (we'd call it that back then) at the same time, over the
| 33.6 modem connection.
|
| It was really easy to run PLIP and was saving me the trouble
| to try to get network card running under Linux on my desktop
| _and_ most importantly saving me the trouble to try to get
| the PCMCIA crap to work on my laptop.
|
| Fun times...
|
| P.S: and, yup, back then laptops had a full parallel port!
| rubatuga wrote:
| Using slip over minimodem over FM receiver/transceiver was
| quite an interesting experiment. With the low power
| exemptions, you can design your own radio networking
| protocol without FCC approval. They may require the FM
| radio signals to be music/audio, in that case, have you
| heard of noisecore?
| z3t4 wrote:
| I suspect there will be some extra network latency using usb-c
| compared to a RJ hardware switch.
| dfox wrote:
| It is almost certainly the other way around. Even 1000-base-T
| PHY is complex block of analog magic that draws significant
| power and adds measurable latency over direct SGMII/optical
| link. In comparison USB4 is even more directly connected to CPU
| than the whateverMII interface of PCIe NIC.
| johnwalkr wrote:
| For a while at work I was trying to use ethernet for embedded
| stuff because it seemed more modern than CAN, i2c etc. I
| couldn't figure out why so few microcontrollers support it.
| It turns out just normal copper 100Mbps ethernet uses about
| 0.25W per port (0.5W for 1 point-point ethernet link) so it
| doesn't make sense for embedded applications. Gigbit uses
| more but I forget how much more.
| Sweepi wrote:
| 8 Zen4 cores dont beat 16 Zen2 cores in multicore. Either
| Geekbench 6 does not utilize all cores, or the 3950x setup is
| borked.
| diamondlovesyou wrote:
| GB6 will use the Zen4's AVX512, which Zen2 doesn't support.
| jltsiren wrote:
| Geekbench 6 multi-core is a single-task benchmark. It runs one
| task at a time, using multiple threads when possible. This
| makes it scale worse and favor single-core performance more
| than benchmarks that run multiple independent tasks in
| parallel.
| wtallis wrote:
| Geekbench 5 pretended that all multi-core workloads were
| embarrassingly parallel by running N independent copies of the
| single-core test across N cores, with zero communication
| between threads. Geekbench 6 takes a more realistic approach so
| Amdahl's Law applies and it doesn't scale linearly with more
| CPU cores.
| daxfohl wrote:
| This would only work for close range, right? To do a whole-house
| thing you'd need to get a bunch of USB-cat6 adapters which would
| end up being more expensive than a switch?
| manmal wrote:
| Yes a couple meters max, at full speed. The full speed cables
| are also quite expensive.
| robocat wrote:
| The full-mesh is three Mini-PCs where each PC connects to the
| other two PCs by USB4: unfortunately the image crops out the USB4
| cable that links the top PC to the bottom PC which makes it a bit
| unclear.
| langarus wrote:
| it's linked in the article. It's this cable
| https://www.amazon.com/dp/B0BLXLSDN5?th=1
| thekombustor wrote:
| 10Gbps mesh for only $48, assuming you already have very modern
| (expensive) USB4 capable machines.
|
| The article also seems to make the assumption that a server would
| be pulling 1000W all the time, 24/7, which is rarely the case (of
| course, I can't comment on what their workload might be, but it
| would be quite unlikely)
|
| Still, I like the direct networking being done here. But saying
| "you can build a 10Gbps mesh for $50" when you have 3x $750+
| machines seems a bit disingenuous. It is not unreasonable to get
| 10Gb SFP+ NICs on ebay for ~$50 a pop ($150 for 3)
| E39M5S62 wrote:
| $50 would be roughly all-in for the SFP+ NIC, two optics and a
| generous length of multi-mode fiber. I just did this, and
| here's the breakdown of my costs from eBay:
|
| 1x Juniper EX3300-24p - $75
|
| 7x SFP+ optics - $7/each
|
| 3x Intel X520-DA2 NIC - $20/each
|
| 4x 3 meter OM3 LC-LC fiber - $6/each
|
| 1x 30 meter OM3 LC-LC fiber - $24
|
| -----
|
| Total: $232
|
| The EX3300-24p has 24x 1gb copper ports with PoE+ on them, and
| 4x SFP+ ports. If you need more SFP+ ports you'll want to find
| a different switch - but for a small multi-use home network the
| EX3300-24p nicely matched my requirements.
| mcoliver wrote:
| Re speed, looks like the cables are the right ones. Would be nice
| to find a wiring diagram of the motherboard to see how the pcie
| lanes are allocated but hard to find on these consumer devices.
| Perhaps each usb4 port is a single lane gen2 which would top out
| around 10Gbps. You could try paralleling iperf3 to give you a bit
| more info. Also check the tx_speed and rx_speed in
| /sys/bus/thunderbolt/devices to see what it is negotiating.
|
| Company specific TB driver/firmware updates can be found here
| https://www.thunderbolttechnology.net/updates
| stoltzmann wrote:
| >While the machine itself isn't expensive, it is not cheap if you
| consider the cost to operate. A machine like this is very power-
| hungry. Suppose the power consumption is 1000W per hour.
|
| Alright, a server will be more power hungry than e.g. your
| desktop, but... This specific Dell R630 has 2x 750W PSUs in it.
| That 750W is the maximum rating of one power supply, and there's
| 2 of them for redundancy - not for increased power intake. That
| server will run at 750W maximum - but that is the absolute,
| absolute maximum power it should draw. It's when you have all the
| rails loaded to the limit and running the server to the ground.
|
| A more realistic scenario would be e.g. 100W or so on average.
|
| The worse problem if running this server at home would be the
| terrible small high-RPM fans they have in 1RU servers. The loud
| high-pitched whine of them will drive you nuts. A better idea
| would be to get either a lower-power 1RU pizza box, or something
| larger that can take larger fans - replacing the fans with
| something quieter and adjusting the fan controller to spin at
| lower RPMs.
|
| >1000W per hour
|
| This is just wrong.
| livueta wrote:
| Yeah, I have a bunch of fully loaded R720XDs and with 12 3.5"
| SATA drives they sit at about 120-140W, both according to the
| lights-out console and wall wart meters.
| november84 wrote:
| An addition, you have the ability to chain the PSUs for
| increased total power at the loss of redundancy.
| strobe wrote:
| noise actually not that bad with some bios configuration tricks
| - I used to run dell r720 and r420 together with ~20 HDDs near
| my desk table and most of the day noise was close to some non-
| silent gaming PC/workstation. Only problem was if is some need
| to reboot it, in that case it just blow fans to full power and
| it's loud like vacum cleaner that you for sure don't wanna to
| use at middle of the night.
| mmastrac wrote:
| I swapped out the screamer fans for Noctuas on a 1U server and
| it's significantly better. Needed a custom mini PCB and a bios
| setting to ignore fan speed feedback but it's extremely quiet
| and runs cool.
|
| I don't think it uses much power when idle either. I think that
| rack servers being expensive is a myth.
| mysteria wrote:
| I run rack servers at home and I 100% call BS on that 1000W
| number. My older Ivy Bridge Xeon servers only consume 50W on
| idle, 100W with some load, and <200W with all cores fired
| running P95 or something. The noise is a concern but it's not
| an issue if you put it in your basement.
| Yukonv wrote:
| Related, Intel was showing off Thunderbolt Share at CES[1].
| Allows Thunderbolt 4/5 device-to-device transfer of files.
| Theoretical speeds in the 20Gbps and 40Gbps for Thunderbolt four
| and five respective.
|
| One idea for why they were only able to reach 11Gbps is having
| only one Thunderbolt/USB4 controller[2], meaning the two USB4
| ports split the 40Gbps PCIe lane. Throw in a full-duplex
| connection and you get 10Gbps in one direction.
|
| [1] https://youtu.be/GqCwLjhb4YY?t=81 [2] Just a theory but seems
| like a sane assumption.
| myself248 wrote:
| > I recall reading some networking books that mentioned
| interesting ancient network structures a long time ago, such as
| ring topology networks or daisy chain networks.
|
| IP-over-SCSI was great, you could throw 8 PCs on one SCSI chain.
| Put two SCSI controllers in each machine, do rows-and-columns,
| and you could have 64 hosts a maximum of 2 hops from each other,
| at U320 speeds, in the 1990s.
|
| https://www.linuxjournal.com/article/2344
|
| Imagine a beowulf cluster of hot grits! Er, sorry...
| Havoc wrote:
| Would it not make sense to put the k8s network on 2.5gbe and give
| the three nodes a usb4 connection to a storage NAS instead?
| Assuming you can find a nas with 3x usb4 I suppose
| alphabettsy wrote:
| A NAS typically doesn't use or support connecting to a PC over
| USB, that would be DAS(direct attached storage), but even then
| they don't support connecting to multiple PCs simultaneously.
| BloodyIron wrote:
| Didn't read the article, but the math they did about the R730
| power draw is NOWHERE near realistic. A 1000Watt power supply
| unit (PSU) will only draw what is actually being used, not the
| full capacity 100% of the time. I have a fleet of R720's
| (generation before the example R730's) and their typical at-wall
| draw when running many VMs on them is about 150 Watts. So their
| math for that aspect is 100% WRONG.
|
| That being said, neat concept IMO.
| armhole2625 wrote:
| I also have a fleet of R720s and can confirm BloodyIron's
| numbers. Average draw is around 150 watts.
| dgacmu wrote:
| Somewhat related, I recently wanted to fix a slow nfs problem (1
| server, 1 client -- basically, a file server and a machine with
| some GPUs). It turns out that used previous generation mellanox
| 100gig NICs are quite cheap now, and you can direct-connect two
| machines for about $600.
|
| (What I really wanted was the low latency, but the bandwidth is
| handy to have sometimes)
| theogravity wrote:
| > Given California's average residential electricity rate is
| 15.34 cents per kWh
|
| In the bay area it's 2.5x - 3x the rate.
|
| I stopped using my rack server for that reason. I was dumb and
| didn't consider the electricity costs when buying mine at the
| time years ago. It sits around collecting dust now.
|
| The author's alternative idea is really good considering 8w idle
| and 80w peak and I doubt it hits peak for what the author wants
| to do.
| matmatmatmat wrote:
| It's actually even worse in SDG&E territory. A kWh costs around
| 32 cents, but "transmission" and "distribution" are again twice
| that. The end result is about $1.00 / kWh.
| johnwalkr wrote:
| I use a shorter, similar cable when I travel with 2 macbooks
| (work and personal). I like to use barrier (software kind-of KVM)
| to share my mouse and keyboard between them and work on one, but
| use the other one for music and reference material. I get
| work/personal separation with the convenience of using one mouse
| and keyboard.
|
| I simply set up static IPs on each one and also setup internet
| sharing from one to the other. Surprisingly barrier, internet
| sharing and even charging always works with this arrangement. I
| only have to connect 1 macbook to wifi and power.
| arghwhat wrote:
| Nit: This is not a mesh. With only 3 nodes, it's not even a ring
| but just a direct connection between servers.
|
| A "real" ring is formed when you have more than 3 nodes and so
| some destinations require more than one hop. A mesh implies a
| network formed by somewhat arbitrary point-to-point connections
| with multiple possible routes between two points.
| toast0 wrote:
| Full-mesh typically means each node connects directly to each
| other node (for some value of directly), so IMHO, a three node
| system with each node connected to the other two fits the name.
| I do agree that it's not very interesting with only three
| nodes.
| tssva wrote:
| This may not be a ring, but a "real" ring doesn't require more
| than 3 nodes. A ring can be formed with as little as 2 nodes.
|
| If routing was enabled this would definitely be a mesh network.
| It would be a full mesh network where every node is directly
| connected to each other and a secondary path through another
| node is available should the direct path fail.
| nabilt wrote:
| This is a pretty cool solution. I didn't know the capabilities of
| USB4 before this.
|
| The comparison with the Dell r630 power numbers got me interested
| since I just purchased a Dell r430 to host my site so I decided
| to benchmark mine.
|
| Specs: * 2x Xeon E5-2680 v3 (same CPU) *
| 64GB RAM * 2x power supply (can't remember if it's 500W or
| 750W each and too lazy to look) * 1 SSD & 1 7300 RPM HDD
| * Ubuntu server 22.04.3 LTS
|
| Using a Kill-A-Watt meter I measure ~100 watts after boot.
| Running sysbench (sysbench --test=cpu --threads=12 --cpu-max-
| prime=100000 --time=300 run) I get up to ~220 watts.
|
| If my calculations are correct that's 72 kW per day or $11.05 per
| month at idle: 0.1 kW * 24 hours * 30 days = 72
| kWh 72 kWh * 15.34 cents/kWh = $11.05
|
| and 158.4 kW or $24.3 per month during load: 0.22
| kW * 24 hours * 30 days = 158.4 kWh 158.4 kWh * 15.34
| cents/kWh = $24.3
|
| I'm not sure of OP's use case, but these numbers are probably
| more realistic than using the max wattage of the power supply for
| most people. I will still be hosting in a co-location for the
| reliable internet and so I can sleep without the sound of a jet
| engine taking off. Those fans are loud!
___________________________________________________________________
(page generated 2024-01-15 23:01 UTC)