[HN Gopher] Oxide Computer: Docs
___________________________________________________________________
Oxide Computer: Docs
Author : avrong
Score : 284 points
Date : 2023-07-20 20:48 UTC (1 days ago)
(HTM) web link (docs.oxide.computer)
(TXT) w3m dump (docs.oxide.computer)
| EvanAnderson wrote:
| Everything about Oxide's gear sounds like fun. I imagine it must
| be a bit like what working with minicomputers in 70s thru the 90s
| was like.
|
| I did a little work in the late 90s with Alpha-based machines. I
| was impressed at those machines didn't seem like the hack-job
| crap that PC-based stuff was (with simulated chips from the early
| 1980's hiding out in dark corners because "compatibility") and
| still is today. I'm betting working with Sun gear felt similar,
| though I never got to work with it. Just having an honest-to-God
| serial console, as opposed to crappy bag-on-the-side things that
| scrape video memory and pretend to be "legacy" PC input devices,
| would be an amazing thing.
|
| I'll never be able to work with their stuff because I don't work
| with Customers at that scale. I'm also vastly unqualified to work
| for them at their current stage. I suppose maybe someday they'll
| need field service technicians... I can hope, I guess.
| soneil wrote:
| I really wish they'd do a tour of it, hardware & software.
| They've gotta be proud as hell of what they've pulled off, it
| boggles my mind that they're not more eager to show it off.
| mst wrote:
| Given it was only relatively recently they were talking about
| getting their first deployment out the door, I'd suspect that
| staying mostly quiet is strategically wise in terms of
| avoiding having more prospective customers come in the door
| at once than they can (as yet) service to the standard they
| want to.
|
| So "they probably really really -want- to show it off, but
| are showing remarkable restraint for good reason" seems at
| the very least plausible to me.
| EvanAnderson wrote:
| I'd fly out to their factory on my dime and pay to watch a
| dog-and-pony show and hear a Q&A. I'm that excited about this
| stuff. Even if I just end up being a Customer of somebody who
| hosts my VM on this gear it's plenty exciting. It actually
| feels like a new computer, as opposed to the same old, same
| old offerings from the incumbents.
| newsclues wrote:
| They need to get Linus Tech Tips to do a video on the
| machines and people of Oxide.
| wmf wrote:
| And then void the warranty by water cooling it.
| newsclues wrote:
| Drop a sled first, then try to watercool one! It would be
| fun content.
| Aurornis wrote:
| > I imagine it must be a bit like what working with
| minicomputers in 70s thru the 90s was like.
|
| I had a chance to talk to someone who worked at Oxide once. I
| got excited because my experience matches up nicely with their
| products on several levels.
|
| The person got excited as well and started talking about how to
| interview there, but the conversation got kind of weird. They
| kept emphasizing how important it was to not talk about
| compensation in the interview. Apparently they pay engineers
| all the same comp (not bad, but would have been a significant
| step down from every offer I received during that time of my
| life) and they select for people who aren't interested in
| getting paid a lot for their unique skills.
|
| Probably not a bad deal for people who like working in that
| domain with like minded people. At the time I got some very
| uneasy feelings from being aggressively coached to not bring up
| compensation or ask any questions about equity during the
| interview like it was some unspoken rule that would get me
| disqualified. Maybe the person was exaggerating, but I found at
| least one other person with a similar story.
|
| Honestly, their comp would have been awesome if I was a single
| guy living in a low cost of living location and working remote,
| but at the time it would have meant giving up quite a bit to
| work for an early stage startup with high expectations and an
| unspoken rule that I should never ask about compensation.
| bcantrill wrote:
| This conversation strikes me as unlikely on several levels.
| First, no one would have coached you on "how to interview at
| Oxide" because that's not where the process starts -- it
| starts with you preparing your materials.[0] (Our review of
| the materials constitutes ~95% of our process.) Second, we
| have always been very explicit about compensation (that is,
| we ourselves brought it up early in conversations); no one at
| Oxide would tell you to "not bring it up" because everyone at
| Oxide knows that it is a subject dealt with early in the
| process. And finally, this is all assuming that you were
| talking to someone before March 2021, when we published our
| blog post on it.[1] After the blog post, compensation simply
| doesn't come up: everyone has seen it -- and indeed, our
| approach to compensation is part of what attracted them to
| the company!
|
| [0] https://docs.google.com/document/d/1Xtofg-
| fMQfZoq8Y3oSAKjEgD...
|
| [1] https://oxide.computer/blog/compensation-as-a-reflection-
| of-...
| Aurornis wrote:
| > First, no one would have coached you on "how to interview
| at Oxide" because that's not where the process starts
|
| I was using "interview" as a generic term for applying to a
| company, not literally referring to your internal process.
|
| I didn't interview with or even pursue Oxide after the
| conversation (and never said I did)
|
| This all came up because I asked the person what the equity
| compensation was like. Not an unreasonable question when
| talking about a startup. That's when they started advising
| me that I shouldn't bring it up and it's not something they
| talk about. After that I got uncomfortable about pursuing a
| company that discourages any conversation about
| compensation to the extent that someone felt necessary to
| warn me about it when I hadn't even applied.
|
| > no one at Oxide would tell you to "not bring it up"
| because everyone at Oxide knows that it is a subject dealt
| with early in the process.
|
| They were trying to tell me that it was important that I
| avoid giving the impression that I cared about
| compensation, as that would be a negative if I talked to
| anyone else at the company. Just repeating what I was told.
|
| > And finally, this is all assuming that you were talking
| to someone before March 2021, when we published our blog
| post on it.[1]
|
| No, they told me to look up the blog post, but I had not
| read the company blog before taking to this person.
|
| > After the blog post, compensation simply doesn't come up:
| everyone has seen it -- and indeed, our approach to
| compensation is part of what attracted them to the company!
|
| Or maybe your approach to compensation is what filters
| people out of the application pipeline? I don't think it's
| realistic to think that this compensation strategy is what
| attracts people to the company rather than pre-selecting
| people out.
|
| I read the blog post, but I feel like I'm missing the
| equity portion of the conversation still.
|
| Regardless, is it so hard to believe that compensation
| "simply doesn't come up" because potential candidates (like
| me) are sometimes coached to not bring it up? Or that the
| company's stance appears to discourage bringing it up? This
| feels like some circular logic: Nobody brings it up because
| we discourage people from bringing it up.
| bcantrill wrote:
| It definitely wouldn't be viewed as a negative to talk
| about it, and in fact our transparency on this topic
| makes it very easy to talk about directly. So we
| absolutely don't discourage talking about it -- but it's
| also true that people for whom the compensation is going
| to make Oxide impossible do self-select out. And that's
| okay! People have different needs at different stages of
| their career, and there are different things that they
| want; there is nothing wrong with optimizing for
| compensation -- but it's also true that Oxide is very
| unlikely to be a fit for someone optimizing for
| compensation, for many reasons.
| petesergeant wrote:
| Hey Bryan, I enjoyed reading the cash compensation article,
| but I'm curious about how it meshes with your equity
| compensation? Is it awarded purely on when people joined?
| Does everyone who wasn't a founder get the same amount? It
| feels like you could run into much of the same problems the
| article points out about transparency and so on if the
| equity component isn't as straight-forward?
| alexgartrell wrote:
| The person to whom you are replying clearly meant (IMO)
| that you shouldn't ask for more compensation or you will
| make people act defensive. Frankly, your reply reads a
| little defensive so maybe that's not awful advice?
|
| It also seems like this was a spontaneous initial
| conversation and not part of the process, so I'm not sure
| why you are suggesting that they made it up.
| gary_0 wrote:
| I didn't realize there are two ways of looking at it
| until your comment. It didn't occur to me that if you
| randomly have Oxide on a list of 40 other companies to
| apply to, and you expect to talk to them about
| compensation as you would with most others, you're going
| to have a weird time because they have an uncommon
| policy.
|
| But on the other hand, Oxide is very up-front about it,
| and their CTO is happy to go on HN and chat about it. So
| not knowing about it makes you look like you didn't do
| your research before the interview, or knowing about it
| and trying to force the issue anyways makes you look kind
| of arrogant (if you don't agree with it, you can just not
| apply).
| Aurornis wrote:
| > It also seems like this was a spontaneous initial
| conversation and not part of the process, so I'm not sure
| why you are suggesting that they made it up.
|
| Yes, it was an initial conversation as I said. When I
| asked about equity compensation I was told that
| compensation discussions are to be avoided because
| bringing it up could be considered a negative by the
| company.
| [deleted]
| paywallasinbeer wrote:
| After giving the "Known Behavior and Limitations" a scroll here
| https://docs.oxide.computer/release-notes/system/1-0-0 It seems a
| little... half baked? Especially for a company whose specialty is
| the integrated management platform. Interested to see customer
| reviews.
| convolvatron wrote:
| this looks like a pretty normal bug list
| goalonetwo wrote:
| Oxide is an HN darling. I have never seen anything even close to
| negative about them here. I hope someone writes a case-study
| about this. It seems to be a mix of the charisma of the
| team/founders and their product that makes everyone love them.
|
| I would put Wireguard/Tailscale in the same category as well.
| panick21_ wrote:
| It partly because they have a good podcast that is perfectly
| aligned with what many people on HN like.
|
| Other then that, the code is open source. They also help
| sponsor both Rust and OpenFireware conferences.
| azinman2 wrote:
| But are people buying their machines?
| intvocoder wrote:
| They only started shipping this month... so, jury's out?
| monocasa wrote:
| They recently shipped their first rack IIRC.
| LoganDark wrote:
| no, companies are :)
| newsclues wrote:
| Given they shipped their first machine, it seems so.
| HL33tibCe7 wrote:
| fly.io too, although the scales have fallen from the eyes of HN
| to some extent after their repeated outages
| [deleted]
| owenmarshall wrote:
| > I would put Wireguard/Tailscale in the same category as well.
|
| Sometimes the hype exists for a good reason - as a paying
| enterprise Tailscale customer, I'll fight you if you _ever_
| suggest I have to staff people to admin an IPsec or SSL-based
| VPN again.
| Tuna-Fish wrote:
| Here's the big negative things I can come up with quickly:
|
| 1. It's not clear the market case closes, because while there
| are customers who would greatly benefit from their systems,
| those same customers are also very averse to change, meaning
| that most of them will probably sit out for the first few
| generations to see if they have staying power. If everyone does
| that, it becomes a self-fulfilling prophecy.
|
| 2. ... but this would have been fine, a few years ago. It's not
| fine in the current market situation, where there is much less
| easy money looking for a place to go. I sure hope they are
| sitting on a long runway.
|
| 3. Speaking of, they are very late in their execution. A SM4
| platform that only shipped in 2023 is not great. I really hope
| they are far along on their SP5 development. (... But this also
| dampens current sales. I bet a lot of potential customers are
| thinking that a SP5 Oxide Rack seems a much more appealing than
| a SP4 one, so why not wait?)
|
| But the reason they are not discussed that much is that people
| who understand the market are all really, really hoping that
| they pull it off. Because the current situation in server
| hardware is _dire_. In every thread people who don 't know much
| about servers ask why not get a similar system from Dell or
| Supermicro or whoever. The answer is that the commodity servers
| are pieces of shit that requires significant local engineering
| resources to manage. Software quality of firmware is generally
| horrendous, and when this is pointed out to the vendors, their
| answer is that they know, but everyone else is just as bad,
| wontfix. A significant draw of the cloud is that all that pain
| goes away, because the hyperscalers realized how shit
| everything was and fixed it in their systems. It just never
| trickled down to the market below them.
| panick21_ wrote:
| There are just very few people regularly buy 500k-1M in
| Server hardware and have a good understanding of the market
| and the finances involved in the alternative.
| wmf wrote:
| It's cool to see Oxide shipping RIFT routing well before
| networking vendors.
| ConanRus wrote:
| [dead]
| mwcampbell wrote:
| I'm a little surprised that, according to the docs, "the Oxide
| rack does not come with any preloaded machine images" [1]. But I
| guess it makes sense that the initial, early-adopter customers
| wouldn't have a problem with rolling their own VM images. I'm at
| least glad that Oxide didn't decide to start by only allowing VMs
| that use a pre-defined set of pre-made images in some custom
| format.
|
| [1]: https://docs.oxide.computer/guides/creating-and-sharing-
| imag...
| wmf wrote:
| The preloaded images would all be Solaris anyway. ;-)
| jclulow wrote:
| We don't call it Solaris anymore! :P
| 59nadir wrote:
| Surely Illumos? Solaris is probably dead to most people
| involved, a husk of what it was and could have been.
| [deleted]
| engagthe wrote:
| I'd love to see someone provide a turnkey managed bare metal
| container platform, complete with L4 / L7 routing. I haven't
| heard if oxide has a container play, and I suspect it may require
| virtualization based on their choice of host OS.
| doctorpangloss wrote:
| By "L4 / L7 routing" do you mean specifically something that
| works like Amazon's Application Load Balancer and Network Load
| Balancer for Kubernetes? And by "turnkey managed" you mean, you
| """just""" """configure""" """IP addresses""" and it """all
| just works"""?
|
| You can certainly install Ubuntu on a very powerful machine
| with a WAN interface (e.g., a NIC connected to a residential
| cable internet connection). Then, use something like k0s to
| provision other bare metal workers. Those three steps, and
| you've got a "bare metal container platform." You don't need a
| "LoadBalancer", you can specify that nginx-controller runs on
| the host network of specifically the machine with the WAN
| interface and configure its service's external IP to the WAN
| IP, and now you support Ingress.
|
| But how do you imagine having multiple LoadBalancer resources
| without multiple IPs? And how do you imagine having multiple
| IPs without ARIN? The turnkey challenging part is the public
| IPv4 addresses, not the platform.
| dilyevsky wrote:
| > You don't need a "LoadBalancer", you can specify that
| nginx-controller runs on the host network of specifically the
| machine with the WAN interface and configure its service's
| external IP to the WAN IP, and now you support Ingress.
|
| And if you need to service that machine or it goes down?
|
| > But how do you imagine having multiple LoadBalancer
| resources without multiple IPs? And how do you imagine having
| multiple IPs without ARIN?
|
| Most colos/transit providers will happily lend you their IPs
| javajosh wrote:
| I'm particularly impressed with their anti-tamper measures. "For
| each server sled, shine a light into the cubby to look for any
| physical tampering or damage."
|
| https://docs.oxide.computer/guides/system/rack-installation-...
| soneil wrote:
| That's pretty standard? Anything that's going straight into
| 400V (I assume) deserves a good eyeball before it's energised.
| I doubt their "root of trust" knows how to protect the system
| from a dead mouse in the bus bars.
| Tuna-Fish wrote:
| > Anything that's going straight into 400V (I assume)
|
| I think the sleds blind mate to a 55V DC common bus.
| javajosh wrote:
| I'm not worried about dead mice; I'm worried about tampering.
| I would have hoped there'd be more guidance than "just look
| and see if anything looks weird". A determined threat actor
| can easily side-step that check with, for example, careful
| desoldering/resoldering technique.
| gorkish wrote:
| If this is seriously your level of supply chain scrutiny,
| being an Oxide early adopter probably isn't the best idea.
| scq wrote:
| I suspect this step is intended to protect you from FedEx,
| not the NSA.
| rhinoceraptor wrote:
| The rack also comes with built-in security features to ensure
| all hardware and software are genuine Oxide products:
| purpose-built hardware root of trust (RoT) - present on every
| Oxide server and switch - cryptographically validates that its
| own firmware is genuine and unmodified encryption of
| data at rest via internal key management system built on the
| RoT trust quorum establishment at boot time to ensure
| the cryptographically-derived rack secret is verified before
| unlocking storage
|
| https://docs.oxide.computer/release-notes/system/1-0-0
| corysama wrote:
| Oxide's "On the Metal" podcast is an incredibly fun deep dive
| into technical issues no one should have to deal with as told by
| people who lived them. "Deep" as in: software DRAM drivers, Ring
| -1 security, bespoke motherboard designs... I only wish there
| were more episodes.
|
| https://oxide.computer/podcasts/on-the-metal
|
| I just noticed they have a second podcast. I'll assume it's just
| as good.
|
| https://oxide.computer/podcasts/oxide-and-friends
| bcantrill wrote:
| For whatever it's worth, I think the reason the doc site was
| submitted to HN recently is in fact because of our recent
| episode on the frontend.[0] We have really enjoyed doing Oxide
| and Friends, and if you're an On the Metal listener, we think
| you'll find a lot to like!
|
| [0] https://oxide-and-friends.transistor.fm/episodes/the-
| fronten...
| pjmlp wrote:
| By the way, Robot was initially tested at Nokia Networks Oy,
| and we on the NetAct business unit weren't that found of it.
|
| Otherwise, another interesting episode.
| rtpg wrote:
| Oxide + Friends is very good. It's very motivating to listen to
| and gets me excited to try and build new things. Way more
| positive vibes being put out there than a lot of stuff.
| [deleted]
| helf wrote:
| This sounds like a great listen. Thanks for mentioning it
| howinteresting wrote:
| [flagged]
| mburns wrote:
| looking at his (recent) comment history, I can't imagine
| this is the one in particular that got him shadow banned.
| usefulcat wrote:
| For that one comment, probably not, but maybe for dozens of
| previous flagged and/or dead comments
| howinteresting wrote:
| Once you're shadowbanned your comments start off as dead
| by default and have to be vouched to be visible.
| usefulcat wrote:
| Ah, I wasn't aware of that--thank you for the
| clarification.
| dang wrote:
| We didn't shadowban them, we told them we were banning
| them: https://news.ycombinator.com/item?id=34916347, we
| told them why, and it wasn't the reason you've presumed.
|
| When people presume things about moderation, they almost
| always get it wrong. That's a pity because all anyone has
| to do to get the correct answer is ask us.
|
| If you see an account that's banned and you don't think it
| should be, please let us know at hn@ycombinator.com so we
| can take a look. We've unbanned quite a few accounts that
| way. In the meantime, you can vouch for their good comments
| (see https://news.ycombinator.com/newsfaq.html#cvouch). But
| please don't divert the thread with offtopic noise that is
| likely to be incorrect.
| stefantalpalaru wrote:
| [dead]
| howinteresting wrote:
| Yes, you told them that you prioritize vibes over
| justice:
|
| > repeatedly breaking the site guidelines and using HN
| for flamewar and ideological battle
|
| helf's comment is absolutely correct and them being
| heated was very reasonable given the context. I'd be
| heated too if I had a daughter nearing puberty.
|
| (And, in an attempt to be intellectually honest, I'll say
| that their recent comment on the post about Kevin
| Mitnick's death was undoubtedly crass.)
| LoganDark wrote:
| Are these available in text or article format?
| gorkish wrote:
| O+F is posted to youtube, I get the auto-generated transcript
| from there. It's unscripted/unstructured, so if you are
| primarily wanting to hear discussion of the oxide
| hardware/software/progress it's pretty difficult to consume
| as an audio program.
|
| This is not a dig on the program at all; I'm glad they are
| making the time to produce it, and I'd rather they spend
| their effort getting racks out the door instead of generating
| marketing hype.
| realitythreek wrote:
| I always miss the live Discord call and never realized they
| were published to YT! Thanks for the notice.
| yardie wrote:
| O+F is really good especially if you can make it to the Discord
| Server
|
| https://discord.gg/gcQxNHAKCB
| lwhsiao wrote:
| Anyone know if a documentation framework is used for this site,
| or whether it is bespoke?
| dcre wrote:
| It's bespoke. Remix + AsciiDoc + OpenAPI.
| codetrotter wrote:
| I wonder how much the smallest, cheapest configuration will cost.
| I would really love to buy one of these. But I suspect it will
| cost 100x more than I can afford heh.
| wmf wrote:
| Estimates in the previous thread were at least half a million.
| codetrotter wrote:
| Yeah that's about 100x more than I can afford exactly.
|
| I think I will go back to my little collection of RPi Compute
| Module machines. And maybe some day in the very distant
| future I can buy big boy servers lol
| masklinn wrote:
| Depending on your exact wants, I think there was an LTT
| video recently on how you can purchase older server blades
| for very reasonable prices, and upgrade them using also
| reasonably priced used server-grade CPUs and RAM. Obviously
| you won't have the bang of new hardware, but if what you
| want to do is play around with enterprise-class systems...
| Roark66 wrote:
| It sound nice, but I've been building systems like that since
| 2012 (usually vmware vCloud director plus custom code). Using
| hardware like Fujitsu's cx1000, Nexus 1000 distributed vswitch,
| brocade network switches, and San storage. The systems I built
| back then could dynamically provision separate
| network/conpute/storage segments for PCI compliance and more. If
| I were to build something like this today I'd probably consider
| using Kubernetes too and integrating it with public cloud for
| scale out. This way a business can have their "own" cloud that is
| much more cost effective at certain scales and access to
| "unlimited" public cloud resources at the same time (at the cost
| of increase in system complexity).
|
| Is this just more of the same, or is there some innovation there?
| It is interesting that computing trends go in
| centralise/decentralise cycles. We can observe this as far back
| as 1980s. I can't wait for the next "decentralise" cycle as I'm
| under an impression a lot more innovation happens during that
| phase.
| panick21_ wrote:
| I think the vision is that a company doesn't need to heir and
| pay you and not pay vmware and they can have that kind of thing
| turn key.
|
| The innovation is mostly in the integration between the
| hardware and software of both the server and the switch.
|
| This podcast goes into what that enables:
| https://www.youtube.com/watch?v=AkWh2Sms3aw
|
| > I can't wait for the next "decentralise" cycle as I'm under
| an impression a lot more innovation happens during that phase.
|
| I think we are beyond that cycle. Centralization and
| decentralization happen at the same time. We have much more
| much cheaper chips now everywhere that do a lot more, and then
| we have local and distributed compute deployed everywhere and
| we also have large datacenters at the center of it.
|
| Neiter datacenters nor distributed compute is gone go away
| anytime soon.
| benrockwood wrote:
| It's great to see Oxide shipping. They have an incredibly
| talented team thats worked very hard for a long time in the best
| tradition of Sun Microsystems.
| dsies wrote:
| Not being pedactic, I swear :)
|
| Can you elaborate on the Sun comparison? I am a huge fan of Sun
| and what they did for computing at large - designing hardware,
| creating specs, their contributions to evolving unix and so on.
| I'm not sure how Oxide compares. Unless you're talking about
| "in the spirit of Sun".
| jeffrallen wrote:
| Sun was a combination hardware and software shop, which Bryan
| appreciated and has tried to replicate at Oxide. The only
| reason they have a chance today is because the
| hardware/firmware interface in most servers is terrible
| quality.
| cbarrick wrote:
| I'm not the one you replied to, so I don't know what they
| mean specifically about the "tradition of Sun", but
|
| Bryan worked at Sun where he helped create DTrace.
|
| After the Oracle acquisition, he left to join Joyent as VP of
| Engineering and then CTO. Steve was COO of Joyent. And I have
| heard similar comparisons between Joyent and Sun.
| ocdtrekkie wrote:
| As an on-prem sysadmin, I'm not sure I will ever in my life work
| on an environment that has the minimum specifications of an Oxide
| system. Every entire datacenter I've worked in has less total
| capacity than a single "sled" here.
|
| Are there intentions to go smaller than this?
| earthboundkid wrote:
| Does anyone know an order of magnitude for the price of a rack?
| sitkack wrote:
| 500k-1M-1.5M
| ThinkBeat wrote:
| Can you configure your Oxide mainframe for different uses? Like
| max it out with 90% being GPUs?
|
| I like the idea, though it may be dumb, if you could fill it with
| different types of processors for what you need.
|
| Sort of like chiplets but at a different scale
| sam_bristow wrote:
| I don't believe they have any GPU option in this initial rack.
| In one of the podcasts they said something about none of the
| GPU vendors being open enough to allow the kind of deep
| integration they're going for. It might change in future of
| course.
| ThinkBeat wrote:
| I recently saw a horrible presentation by Microsoft on how they
| built the OpenAI supercomputer on Azure. (with custom bits).
|
| How would an Oxide computer systems look like for for an "OpenAI
| super computer" compared to Azure?
| pitaj wrote:
| That kind of magnitude of compute is more datacenter level. But
| either way, you're probably better off buying one of the all-
| in-one tightly integrated solutions from Nvidia or Intel.
| 0x6c6f6c wrote:
| > Oxide Computer Model 0, also known as the "0x1"
|
| So is it 1 or is it 0..
| haimez wrote:
| Hopefully they end up with a German market only SKU before the
| next generation of hardware ships, otherwise it's a real missed
| opportunity.
| OJFord wrote:
| Yes.
| rafram wrote:
| It was a 0 in both names, but cosmic rays flipped one of the
| bits.
| dist1ll wrote:
| As someone who's only dealt with commodity server hardware, these
| specs make me salivate. And all these boot/management TUIs are
| just so satisfying to look at.
|
| (Sorry to be that guy, but just a friendly suggestion: high
| contrast dark themes are difficult to read for people with
| astigmatism. Especially since this is technical documentation,
| intended to be thoroughly read, you might want to consider a
| light theme toggle.)
| tiffanyh wrote:
| > _As someone who 's only dealt with commodity server hardware,
| these specs make me salivate_
|
| Commodity servers have the same specs.
|
| Here's a reseller of SuperMicro servers, where you can buy
| similar compute on the cheap.
|
| https://www.siliconmechanics.com/systems/servers/rackform
| Aurornis wrote:
| > Commodity servers have the same specs.
|
| Probably not a coincidence. It would be interesting to know
| which ODM they partnered with for the hardware.
|
| I've done some work with SuperMicro in the past. Some of
| their boards come with extensive headers and customization
| options right out of the box. They're also happy to work on
| board level customizations with the right contracts in place.
| bcantrill wrote:
| We didn't work with an ODM: the ODMs were unwilling to
| contemplate some of the most basic things we needed (e.g.,
| replacing the BMC with a much lighter weight service
| processor, having a true root-of-trust, etc.) -- let alone
| the more things we wanted to do (e.g., our own switch). The
| compute sled and the switch are both of our own design and
| look nothing like what you'll find from an ODM; if you're
| curious in the details, we have discussed them quite a bit
| in our Oxide and Friends podcast.[0][1][2]
|
| [0] https://oxide-and-friends.transistor.fm/episodes/tales-
| from-...
|
| [1] https://oxide-and-friends.transistor.fm/episodes/the-
| sidecar...
|
| [2] https://oxide-and-
| friends.transistor.fm/episodes/bringup-lab...
| mst wrote:
| Please consider generating some sort of automated
| transcript from these.
|
| I'd hope your target audience would understand the
| limitations of such a thing, and I'm probably not the
| only person who'd rather read than listen even with the
| obvious caveats.
|
| (these days automated transcripts seem to be no harder to
| mentally fix up the errors in as I read them than
| "somebody typing fast on a software keyboard and
| suffering the inevitable tyop and autocorrupt related
| issues" is, though of course others' mileage may vary)
| actionfromafar wrote:
| They could probably have someone proofread the
| transcripts, there aren't that many episodes.
| mst wrote:
| I was going for "set up some code once and don't think
| about it again" to maximise the odds of the idea sounding
| tempting.
|
| Proofreading would set up an expectation on the part of
| readers that it -had- been proofread and corrected and
| therefore a commitment to perform a repeated "boring but
| important" task going forwards for whoever's doing said
| proofreading.
|
| That way would likely lie either delayed transcripts or
| never getting to initial activation energy to provide
| anything at all.
|
| So I think "add a quick bit of code to your podcast
| publishing workflow and a CAVEAT IN BIG LETTERS" is
| better to do first.
|
| If it turns out enough people care about the transcript,
| doing it a more labour intensive nicer way later is
| something they can decide, well, later.
|
| Shipping is feature zero, as ever.
| EvanAnderson wrote:
| I haven't worked with SuperMicro (aside from having it inside
| "appliance" devices that I've worked adjacent to), but I
| assume my experience with Dell and HP commodity servers are
| similar.
|
| The thick layer of hardware contrivances necessary to
| maintain IBM PC compatibility is unnecessary for the task of
| bulk hosting of x86/x64 VMs. There's a lot of hardware and
| software that just doesn't need to be there.
|
| Bare metal out-of-band management ends up being bolted-on to
| these "legacy" contrivances (scraping video memory for remote
| consoles, faking being USB peripherals). A serial console or
| SSH connection to a service processor would be vastly
| superior. I can't begin to count how many times an iDRAC
| "lied" to me about issues with a machine, or how many times
| the solution was "upgrade the iDRAC firmware and reboot it".
|
| I have been mostly unimpressed with the quality of firmware
| for motherboards, baseboard management controllers, RAID
| adapters, NICs, HBAs, power supplies, backplanes, front
| panels, etc. Every new model of system or component ends up
| being an exercise in fear / anticipation of problems. The
| integrator has very little power over the firmware quality
| and I can be assured that if I do have a firmware-induced
| issue I'm many, many steps away from actually communicating
| with somebody who can help.
|
| Granted, maybe if I was buying at the scale of Oxide's
| prospective Customers I'd have some pull with the
| integrators, but I'm skeptical of that, even.
|
| Oxide is actually building computers. Putting commodity
| motherboards into boxes with other commodity components won't
| ever have the level of attention to detail and integration
| that Oxide can provide.
| sjdmdlakziggy wrote:
| They are not even close to the same specs.
| upon_drumhead wrote:
| https://www.siliconmechanics.com/system/rackform-a235.v8.1/
| 2...
|
| This is pretty darn close to the "Gimlet" Compute Sleds
|
| One 225W TDP 64-core AMD Milan CPU 1 TiB of DRAM across 16
| DIMMs 12 front-facing hot-swappable PCIe Gen 4 U.2 storage
| devices 2 internal M.2 devices 2 ports of 100 GbE
| networking
|
| Point for point, the same hardware as specified.
| intelVISA wrote:
| As much as I wish Oxide the best, most of us here would
| probably prefer our own libre, hand-crafted OS atop Coreboot.
|
| (Or maybe I speak for myself only...)
| jxf wrote:
| > boot/management TUIs
|
| Where are you seeing these in the docs? Or are those somewhere
| else?
| dist1ll wrote:
| Here https://docs.oxide.computer/guides/system/initial-rack-
| setup
| all2 wrote:
| > sorry to be that guy ... astigmatism
|
| Darkreader is a lovely plugin that might make your life much
| easier: https://addons.mozilla.org/en-
| US/firefox/addon/darkreader/
|
| I believe there is a Chrome extension, too.
|
| It lets you set BG, FG, sepia, total contrast, etc. It is quite
| a neat piece of kit.
| bpye wrote:
| I would never have thought to use an extension intended for
| adding dark mode, to instead view dark mode websites in a
| light mode, but that makes total sense!
| pxc wrote:
| I recommend Midnight Lizard as a backup for difficult sites.
| It's a little heavier/slower, but sometimes works on pages
| where Dark Reader doesn't. You can configure Midnight Lizard
| not to apply to all sites by default, then selectively turn
| it on where Dark Reader fails.
| benjaminleonard wrote:
| All of our sites and indeed the web console itself are driven
| by the same design system and we are planning on a light theme
| as soon as we can.
|
| Thanks!
| pxc wrote:
| > high contrast dark themes are difficult to read for people
| with astigmatism.
|
| Are you yourself affected by this? I have astigmatism and I
| keep hearing this, but I've never experienced it. If you are
| affected, do you keep your screen at high brightness? I'm
| wondering if it doesn't happen to me only because my
| astigmatism is mild, or if the fact that I tend/have to keep my
| screens at relatively low brightness plays a role.
|
| Incidentally, for different reasons, high contrast dark themes
| can be problematic for me as well (especially at high
| brightness). Dark Reader and Midnight Lizard are essential for
| me in keeping contrast in a comfortable range.
| benjaminwootton wrote:
| I was about to ask the same question. I think I have moderate
| astigmatism and find dark mode easier on the eyes.
| jna_sh wrote:
| I have astigmatism and loathe dark mode. I find low
| contrast and soft colours easiest.
| whalesalad wrote:
| Would be cool to play with this hypervisor on some lab hardware.
| mkeeter wrote:
| It's been done before!
|
| See the excellent "Oxide at Home" blog post [1], and HN
| discussion [2]
|
| [1] https://artemis.sh/2022/03/14/propolis-oxide-at-home-
| pt1.htm... [2] https://news.ycombinator.com/item?id=30671447
| ChrisArchitect wrote:
| What's new here
| wmf wrote:
| I think these docs were just published.
| benjaminleonard wrote:
| These have been online for a few months actually, though the
| release notes have only just been added.
| dang wrote:
| OK, let's document that in the title above. Thanks!
| slater wrote:
| Yeah, few days ago:
|
| https://docs.oxide.computer/release-notes/system
| gorkish wrote:
| As someone who has been attempting to smash computers together
| into 'hyperconverged infrastructure' since before Y2K, I could
| not be happier for the buzz around Oxide; hopefully we'll see
| smaller scale versions of the concept out soon -- something in
| the 3-10 node size for SMB and/or a small scale-out system for
| SOHO. It seems insane to me that the tech to do this with off the
| shelf free software exists, but there's no way to buy it ready to
| go at small scale. Go get a quote for an HCI vmware buildout and
| DISMAY.
| drbawb wrote:
| I am super interested in learning more about the storage
| subsystem! I figured they'd be using ZFS, given the people
| involved, but it appears they've also gone ahead and built a
| clustered FS (crucible) on top of it? I figured something like
| that would be necessary to handle fault tolerance at the gimlet
| level. (Losing an entire shelf / drive controller, etc.) Getting
| ZFS to go multi-node is surely a neat trick.
|
| Second to that I just want to say the presentation of these docs
| is top notch. (I so desperately wish I was the target customer
| for these systems; reading these docs makes me want to do
| terrible things to my electrical service and play with one of
| these racks.)
| kaliszad wrote:
| They use Crucible on top of ZFS.
| https://github.com/oxidecomputer/crucible I don't think they
| have anything for S3-like service but there are other options
| for that, e.g. https://garagehq.deuxfleurs.fr or MinIO. I am
| not sure whether they have their own SSDs or use of the shelf
| SSDs just with their firmware or something.
| ori_b wrote:
| They're not the first company to do that. https://panzura.com/
| did something similar.
| oxide wrote:
| This is great, thanks for bringing this to my attention. Cool
| name, too.
| solarkraft wrote:
| Finally I actually understand what they're building. Now I must
| ask: Why?
|
| On-prem servers aren't a new invention. The market seems pretty
| saturated (and shrinking). Virtualization isn't a new invention
| either. The market seems pretty well served, at least
| commercially. Can the integration of both be a convincing enough
| advantage?
|
| The management UI certainly looks nice; it's something I'd like
| to have on my KVM box at home (any good Proxmox alternatives?). I
| don't see why it'd have to be bound to an enormous server.
| rtpg wrote:
| The short answer is that on-prem is used by a lot of companies
| for many reasons that go beyond "legacy/people don't know about
| cloud" (and even go beyond "regulatory environments"!), and
| these boxes are meant for people who have serious requirements.
|
| I'd recommend the O+F ep posted in sibling, but I think here
| the pitch is "well you need this hardware anyways right? How
| about buying one that's easy to use and doesn't take a month to
| get working?" All built by people who are so obsessed with root
| cause analysis that they've ended up writing their own
| firmware, running on an OS where these people are common
| contributors.
| solarkraft wrote:
| Thanks, I think _now_ I get the excitement (from the
| engineering perspective more than the business one)! "Fixing
| everything" is probably everyone's dream.
| faitswulff wrote:
| One benefit is that they're competing in an industry where the
| time to get a sled up and running for compute can be measured
| in weeks (they say they've heard up to 90 days) whereas their
| solution is basically plug and play - Bryan was trying to get
| Steve to admit set up took "hours" whereas Steve was hedging
| and saying customers could get started "within a week."
|
| They go into more details on their podcast, and this section in
| particular covers the bootstrap time:
| https://youtu.be/5P5Mk_IggE0?t=3381 Pretty fascinating stuff.
| throwawaylinux wrote:
| Do I have it right that they ship their own hypervisor
| (that's based on maybe Solaris, not KVM) as firmware? Let's
| assume a very small team can compete on a technical level, it
| still seems like it could cut out a lot of the potential
| market.
|
| I can't imagine that large cloud / "web scale" companies
| would want that. Most want a fair bit of control of their own
| hypervisor and management stacks based around KVM. And
| "enterprise" type companies are going to have issues with
| certification I would have thought -- will RedHat, Microsoft,
| SAP, Oracle, etc certify their supported products on top of
| this hypervisor? Seems like a difficult and expensive
| process.
|
| So what's left? Companies that support their own virtual
| machine software but don't support their own hypervisor and
| don't like what's available from vmware or Microsoft or
| RedHat. A small niche. Or are my assumptions wrong?
| panick21_ wrote:
| I think quite often when we assume 'most want to fair bit
| of control' is just not true. Enterprises want something
| that just works, they want control if they can't have
| something that just works.
|
| If you have a team that is struggling building an internal
| cloud with all this control (and problems) and all this
| commodity hardware (and its problems) then maybe they would
| be happy to switch to something that just works.
|
| > And "enterprise" type companies are going to have issues
| with certification I would have thought -- will RedHat,
| Microsoft, SAP, Oracle, etc certify their supported
| products on top of this hypervisor? Seems like a difficult
| and expensive process.
|
| If that was the case and nobody running any of these would
| run their rack, then I wouldn't think they would not have
| received any funding. But I don't know enough about these
| certification process to really comment.
|
| > Companies that support their own virtual machine software
| but don't support their own hypervisor and don't like
| what's available from vmware or Microsoft or RedHat
|
| Non of these come with a fully integrated rack.
|
| The competition would be somebody willing to buy a rack of
| Dell servers with VMWare software. Or somebody willing to
| buy a rack of Dell server and then use RedHat and set up
| all their own cloud style infrastructure.
| throwawaylinux wrote:
| > I think quite often when we assume 'most want to fair
| bit of control' is just not true. Enterprises want
| something that just works, they want control if they
| can't have something that just works.
|
| That's not what I'm assuming here. Read carefully, I
| divide the market into 3 categories. Those who support
| their own VM image software and hypervisors, those who
| support neither, and those who support VM image but not
| hypervisor.
|
| First is Amazon, Google, Facebook and the like (and it's
| not an assumption we can see their public contributions
| to KVM, QEMU, etc., and hear their talks about some of
| what they use internally). Second is "enterprise" who
| wants something that just works. Third is ? and would
| they want to support their software on a niche
| hypervisor?
|
| > If that was the case and nobody running any of these
| would run their rack, then I wouldn't think they would
| not have received any funding. But I don't know enough
| about these certification process to really comment.
|
| Well it is the case that enterprise (supported) software
| is not just supported on any hypervisor.
| https://access.redhat.com/articles/973163 RHEL runs on
| their own KVM as well as MS, VMware, some cloud vendors.
| Some application software also gets certified to hardware
| and hypervisors, not just operating system (e.g., SAP
| does this).
|
| > Non of these come with a fully integrated rack.
|
| It's not fully integrated if it doesn't come with the
| guest software though, is it?
|
| > The competition would be somebody willing to buy a rack
| of Dell servers with VMWare software. Or somebody willing
| to buy a rack of Dell server and then use RedHat and set
| up all their own cloud style infrastructure.
|
| Right. And the problem for Oxide is that the competition
| will have fully certified and supported operating system
| and application software for their virtual machines.
| kaliszad wrote:
| The hypervisor is based on Bhyve from FreeBSD + Propolis in
| user space. Illumos actually has/ had KVM and there is a
| talk by Bryan Cantrill where he speaks about the porting
| effort. All of that information is readily searchable.
| [deleted]
| throwaway2037 wrote:
| Stupid question (no trolling, I promise): Could this product be
| valuable to public cloud vendors, like AWS, Google, Oracle,
| IBM, etc? Or even medium sized ones, like Linode? My thinking:
| Could this be the cloud inversion moment akin to TSMC and
| outsourcing semi-conductor manuf?
| dbish wrote:
| Probably not. These vendors have massive supply lines that
| are designed and tuned to their needs.
| littlestymaar wrote:
| For big vendors, no. Because what oxide does is basically
| sell the kind of server that the hyperscalers have been
| building for themselves internally. But for smaller cloud
| provider, who aren't running custom hardware made for
| hyperscale and are instead using of the shelf servers with
| all their flaws, then it could make a lot of sense.
| chamakits wrote:
| I'm not affiliated with them, but I recall this being marketed
| at some point as giving you the flexibility and customization
| powers that the Googles and Facebooks of the world have with
| their on-prem infrastructure without needing to have as deep of
| a dedicated staff as they do to just this which allowed them to
| develop all their custom tooling in the first place.
|
| Basically if you are on-prem, and you are dissatisfied with
| what you are getting out of today's onprem sellers. Things like
| bad firmware with slow update cycles, issues with rack/power
| supplies/cabling/interconnecting systems. Closed down systems
| that don't allow much customization, etc. They are open
| sourcing a lot of their work along the way
|
| Again, I'm not affiliated with them, and my info may be
| outdated so take it with a grain of salt. But that's how I've
| seen them for some time.
| wmf wrote:
| I would say Oxide is inflexible and non-customizable since
| they have exactly one hardware configuration and few software
| features at this point. Their claim is more that their rack
| works and everything else on the market is full of bugs.
| throwaway2037 wrote:
| I'm not an infra engineer, but this claim "everything else
| on the market is full of bugs" might be the killer app. Of
| course, it needs to be true. What if they iterate to an
| insanely stable embedded code base (BIOS, etc.)? Then,
| continuously upgrade the hardware to use the latest
| CPU/RAM/NVME. I could see that being very valuable.
| strangemonad wrote:
| A dumbed down interpretation. What most people can buy off the
| shelf for servers more or less looks like a pc shoved into an
| odd looking case (1u, 3U rack). To get anywhere near the cost /
| performance of the big players you need something that's
| designed for the data center server workloads.
| bpye wrote:
| It feels like they're trying to give people the benefits that
| hyperscale cloud operators have, without being one?
| superb_dev wrote:
| That is exactly their goal. Hyperscale for the rest of us
| jitl wrote:
| Public cloud like AWS is a premium product. When your web scale
| business wants to increase profits by cutting costs and you've
| already done a few passes making your software run faster,
| owning your own metal & renting colo space starts looking like
| a big avenue for savings. Especially if you're doing something
| bandwidth intensive, where AWS makes you pay through the nose.
| I think we'll see a fair number of companies move back towards
| owning metal, especially if the metal is super easy to manage.
| I think the Oxide pitch is making owning racks sensible for
| ~2000 eng companies instead of ~20,000 eng companies.
| mhh__ wrote:
| YOLO - sometimes it's worth paying more money for something
| someone has actually sat down and designed properly.
| throwawaylinux wrote:
| So what has been designed properly and what hasn't?
| mhh__ wrote:
| The random piles of servers in most offices above a certain
| headcount
| panick21_ wrote:
| Cars aren't a new invention either, and yet people build new
| ones.
|
| There is an existing market, it is large. Making a product for
| that market that is just simply 'really good' has potential to
| make money.
|
| Almost all great computer companies started into markets where
| things already existed that were comparable.
___________________________________________________________________
(page generated 2023-07-21 23:02 UTC)