[HN Gopher] IPMI
___________________________________________________________________
IPMI
Author : pabs3
Score : 249 points
Date : 2024-09-03 04:23 UTC (18 hours ago)
(HTM) web link (computer.rip)
(TXT) w3m dump (computer.rip)
| smitty1e wrote:
| Intelligent Platform Management Interface
|
| https://en.m.wikipedia.org/wiki/Intelligent_Platform_Managem...
| TMWNN wrote:
| I have found HP iLO very useful on my MicroServer Gen8 with
| UnRAID.
| renewiltord wrote:
| Personally, for smaller deployments (I.e. about 10k cores) I
| would just self deploy out from some integrator. A
| Gigabyte/AsRock Rack Motherboard + Epyc 9003 series + some 384 G
| of RAM with the usual dual PSUs and all that stuff will be
| $7k/node and can be very power efficient.
|
| The integrated IPMI is pretty good on these and plays well with
| ipmitool. They'll usually have some Redfish stuff going on.
| buserror wrote:
| IPMI is such an improvement on the old ISA-style iochips, and not
| even sure they are more expensive these days! It's not just the
| remote/web access which is great, plain 'introspection' with
| tools like ipmiutil and ipmitool are super useful, even if you
| don't have a huge rack 'enterprise' installation.
| brcmthrowaway wrote:
| Was IPMI useful in solving the CrowdStrike boondoggle?
| xaduha wrote:
| Since one of the solutions was to reboot "up to 15 times" I
| would imagine so.
| sofixa wrote:
| Yep, because on enterprise servers in a rack somewhere in a
| remote datacenters, IPMI is what allows admins to connect
| remotely, restart, and get a remote console to remove the
| offending files (or just keep restarting).
| throw0101d wrote:
| > [...] _and get a remote console to remove the offending
| files (or just keep restarting)._
|
| Sometimes (often?) you need a more 'advanced' OOB/LOM license
| to get remote console from Dell/HP/Lenovo/ _etc_.
| omgtehlion wrote:
| I suppose it was useful only for a part of the problem, as most
| CrowdStroke(tm) machines were client and kiosk...
| sandreas wrote:
| I really like IPMI, but what I don't like about it for homelab
| use cases is the additional ~5W of idle power consumption. Using
| the Gigabyte MC12-LE0 Board with Ryzen Pro 5650 for a home server
| should be a no brainer costing ~50 bucks, but the higher power
| consumption is something I'm not totally happy with.
|
| On older devices (like Dell T20/T30) there is Intel AMT which is
| way less powerful and has security flaws, but at least SOME way
| to do remote administration when used with MeshCommander
| (unfortunately discontinued and releases deleted from everywhere
| - but I luckily saved the MSI and Node-Package on my server).
|
| I'm planning to try out PiKVM (V2) with a Raspberry 4 and a
| simple 8 bucks USB-HDMI-Capture-Card
| (https://docs.pikvm.org/v2/). Besides the lack of some features,
| it looks promising and more universal for devices that do not
| support remote management in any form.
| dugite-code wrote:
| Similarly I'm looking into the nano kvm now the firmware is
| open https://github.com/sipeed/NanoKVM
| dugite-code wrote:
| Update, perhaps I was miss-informed. Looks like the firmware
| isn't in the repo, just the web interface.
|
| Disappointing, I had read on a blog somewhere recently that
| they had released the firmware and didn't look into it deeply
| enough
|
| > the backend will be open-sourced soon (after the GitHub
| repository reaches 2K stars).
|
| https://en.wiki.sipeed.com/hardware/en/kvm/NanoKVM/system/in.
| ..
| sandreas wrote:
| I would love to see an OpenWRT implementation, so that your
| router can be your KVM device :-)
| stragies wrote:
| I believe, that few, if any of the currently supported
| boards feature all of: USB Device Port, HDMI-Port (or
| sufficiently fast USB-Host-Port for Capture Adapter), HW-
| Video-Encode (or beafy enough CPU), enough available
| (programmable) GPIOs
| sandreas wrote:
| I think with Wifi 6e and 7 this is no longer true. Most
| of these devices are really powerful supporting at least
| one usb3 port. This is probably no longer a problem in
| the near future, especially when 1080p is enough.
| guenthert wrote:
| > I really like IPMI, but what I don't like about it for
| homelab use cases is the additional ~5W of idle power
| consumption.
|
| How much more power is dissipated clearly depends on the
| quality of the PSU (constructing a power supply which is
| efficient at high loads _and_ idle is quite a challenge). I
| doubt the BMC (sometimes built-in to the NIC) takes nearly that
| much.
|
| > I'm planning to try out PiKVM (V2) with a Raspberry 4
|
| That'll exceed 5W though.
| sandreas wrote:
| > How much more power is dissipated clearly depends on the
| quality of the PSU
|
| Yeah I know - while "quality" (I assume efficiency) is not
| the only factor, because low tier PSU (<200W, like Pico PSU)
| are far less consuming below 20W.
|
| > That'll exceed 5W though.
|
| Only running all the time. Paired with a Tasmota Shelly Plug
| it consumes near 0W. Additionally it is Board independent and
| together with a HDMI- / USB-KVM-Switch it can be used for
| multiple hosts.
|
| I also thought about using PiKVM similar devices on my Banana
| PI 3 OpenWRT router, so this thing is running all the time
| and powerful enough to handle this.
| progbits wrote:
| RPi 4 idles at around 3W and peaks at 5-6W so on average
| should be better.
| geerlingguy wrote:
| Most of the boards I've tested with ASPEED chips consume
| between 5-7W when IPMI is running, and the system is powered
| off. They aren't all that efficient.
| petee wrote:
| MeshCommander 0.96 is available on its website again; from what
| ive read the dev has been settling into a new job
| mikeocool wrote:
| MeshCommander releases are still available --
| https://www.meshcommander.com/ Can also still be installed from
| NPM.
|
| Also haven't tried it out, but I believe this is aiming to be
| the successor: https://meshcentral.com/
| sofixa wrote:
| A few notes because there are a few pieces of information which
| aren't a 100% up to date:
|
| As most people know, Intel have lost the plot and are behind AMD
| on everything CPU and GPU related (with the tiny exception of the
| N100 series CPUs from Intel which are perfect for low power and
| even fanless applications). As such, their CPUs are mostly bought
| by organisations updating previous environments with like for
| like replacements for whatever reason (like vSphere's EVC which
| allows you to run newer processors from the same manufacturer as
| if they were an older model, and enable hot migration between CPU
| architectures and thus minimum disruption on hardware refreshes).
| Pretty much everyone else is getting the better and cheaper (per
| processing power) AMD CPUs.
|
| Intel NICs are pretty good overall (the X710 which spent a year+
| with broken drivers causing silent network failures/downright
| crashes while still being in the hardware compatibility list of
| VMware and other "enterprise" vendors notwithstanding), and
| they're getting more popular for consumer devices too. An Intel
| CPU on a non-low cost PC/laptop almost always means an Intel NIC
| (WiFi and/or Ethernet).
|
| For many organisations buying servers, Supermicro* is probably
| their best option. They're cheaper, have much higher flexibility
| in terms of form factors, chassis, components, number of whatever
| slots and mostly reliable. However their support is less reliable
| than the theoretical Dell/HPE support, so it works best for
| redundant setups.
|
| Also, the IPMI specification is being replaced by Redfish which
| is much more complete, secure, normalised and presents normal
| usable APIs. Any mainline server from a few years ago should have
| Redfish alongside IPMI.
|
| * There's recent research from Hindenburg, a short selling
| research firm, that exposes some shady stuff from Supermicro -
| but their hardware is still top notch and used by the
| hyperscalers. https://hindenburgresearch.com/smci/
| tucnak wrote:
| I really wish Supermicro PSU's were accessible over PMBus
| without their proprietary IPMI utilities[1] but they're not.
| Moreover, it's x86-only so there's no way to interface it from
| ppc64el. Had it been open source, it could trivially be built.
|
| [1] https://www.supermicro.com/en/solutions/management-
| software/...
| buserror wrote:
| Having had a play with both Supermicro and ASRock Rack boards
| for workstations (admittedly, only H12's so not the "most
| recent" but from my cursory glance and newer boards, nothing
| has changed that much), SuperMicro board feels like they were
| made in 2005, not 2024. It is ridiculous in fact.
|
| * No support for ACPI sleep. In 2024. Seriously. * No support
| for 4 _and_ 3 pins fans. 3 pins are 100% speed all the time. *
| IPMI web interface straight out of 2010. * NVME placement
| prevents you from using heatsinks. * Tons of opaque jumpers on
| the board, with no board labelling.
|
| The ASRock Rack equivalent board is _amazing_ in comparison.
| thelastparadise wrote:
| I don't want ACPI sleep on a server.
| kbolino wrote:
| Why not? Assuming your servers are not all at or near full
| load at all times, you can put some to sleep as warm
| spares.
| fer wrote:
| Asking myself the same question, but IME Linux support
| for sleep states is (was?) generally horrible due to
| absolute lack of standard enforcement, and that's with
| laptops where it's a priority. I've only had one laptop
| (my most recent one) work 100% with sleep states, and
| only after replacing the NVMe that froze roughly 33% of
| the wake ups due to some obscure bug (but worked fine
| otherwise)
| mixmastamyk wrote:
| Maybe lucky, but haven't had trouble with Linux and sleep
| for about 15 years. Dell and Framework.
| gosub100 wrote:
| They support wake on lan (most likely) and you could
| power up from IPMI REST API
| DannyBee wrote:
| Agreed. If nothing else, the IPMI web on supermicro randomly
| deciding you need to re-login constantly definitely feels
| very 2004.
|
| It's been this way for at least a decade, too. It's like it
| just forgets all its session variables sometimes.
| slavik81 wrote:
| Every vendor has their quirks. I have a pair of ASRock Rack
| ROMED8-2T/BCM boards, but my M.2 NVME boot drive is no longer
| detected if I update the BIOS past v3.50. Unfortunately, that
| means no support for resizable BAR in my configuration.
|
| I have a pair of Supermicro H12SSL-NT boards to use for my
| next couple builds. I might be trading one set of issues for
| another, but I'm optimistic they'll work well for my
| purposes.
| doublepg23 wrote:
| I agree! I love my ASRock Rack board (X570si). I had written
| off ASRock as "budget" but it's been rock solid and had loads
| of features like you said.
| crest wrote:
| If you think the latest SuperMicro IPMI webinterface is from
| 2010 you haven't seen their 2010 interfaces. Would you like
| to download the Java Web Start file to start your Remote
| Console? Use a Java Applet to see just the screenshot of the
| VGA output? How about getting RAID controllers with a (PS/2)
| mouse based GUI inside the OptionROM (looking at you LSI)
| only to find out the damn RAID controller manual lied about
| having a HBA passthrough mode. It just creates a single RAID
| volume per disk, but still stores controller specific
| metadata on the disks.
|
| If ASRock Rack got the SSH serial console redirection latency
| down from ~1 second to the low milliseconds like the other
| vendors (SuperMicro, Dell, HP, etc.) it would actually be
| useable without taking Valium before logging in.
|
| </rant>
| starttoaster wrote:
| > No support for ACPI sleep.
|
| To be fair, these are meant for datacenter applications where
| it would be absolutely normal for these to be either fully on
| or off all of the time. You could make an argument for warm
| spare servers, but there's other ways to accomplish it than
| ACPI sleep, and I'd probably consider warm spares a
| relatively niche need, considering I'd rather leave them hot-
| but-outside-production for system monitoring purposes. Would
| be more annoying to wake a sleeping system and find it has a
| failing drive/RAM/NIC, bad switchport config, etc.
|
| > No support for 4 and 3 pins fans. 3 pins are 100% speed all
| the time.
|
| To be fair, these are meant for datacenter applications where
| it would be absolutely normal to run your fans at full speed
| all of the time.
|
| > IPMI web interface straight out of 2010
|
| Does it have HTML5 remote console, and basic component
| diagnostics? If so what else would you like? I've been a
| datacenter tech, a linux sysadmin, and a devops engineer,
| depending on the company for a decade now, and really only
| ever used IPMI for those two things so I'm curious where the
| other use cases are. I've also used Dell PowerEdge and HPE
| servers, which have slightly nicer looking UIs but perform
| the exact same functions more or less.
| theevilsharpie wrote:
| > To be fair, these are meant for datacenter applications
| where it would be absolutely normal to run your fans at
| full speed all of the time.
|
| Case fans for rack mount chassis are very powerful, and
| also can take up a significant amount of energy when
| running full-bore (not to mention the mechanical wear).
|
| I haven't used a server chassis in over a decade where the
| fans were running at full beyond a few brief seconds at
| startup. I'm not sure if they used a four-pin header or
| some other mechanism, but fan speed control is a normal and
| expected feature in server hardware.
|
| > Does it have HTML5 remote console, and basic component
| diagnostics? If so what else would you like?
|
| IPMI specifically is meant to be a standardized remote
| management interface. It (mostly) works for basic things,
| but more advanced functionality is hit-or-miss, or absent
| entirely, leaving you at the mercy of proprietary tools.
| Redfish is supposed to be better, although I personally
| haven't used it.
|
| Web interfaces can be hit-or-miss in terms of functionality
| and UX. Additionally, I've often found these web interfaces
| to be unstable -- either being very slow or not loading at
| all, to certain features of the UI not loading data or
| hanging the interface.
| _a_a_a_ wrote:
| Word. I borrowed a circa 2005 - 2010 server and the fan
| _roared_ on start-up. I would not want that on full-time
| for reasons of noise, power and wear.
| fgonzag wrote:
| I've had 40%+ (8 out of 20) RMA rate with ASrock boards (from
| pcie drives dropping to weird gremlins), while I have
| replaced two out of over 300 super micro boards, all of them
| running 10y+
|
| The IPMI interface sure is nicer though
| DannyBee wrote:
| Ah yes, RedFish.
|
| I automate keeping IPMI ssl certain up to date using redfish
| apis. Importing the new certs requires slightly different magic
| (cert naming, encoding, etc) on every single different vendor
| implementation of redfish. I have a collection of Python
| modules to deal with each vendors eccentricity just around
| uploading and swapping ssl certs when it should be a set of
| standard put requests that work everywhere (that is what the
| redfish API docs would convince you of!)
|
| So not sure I would agree that it is either normalized or
| usable, as the whole point of something like redfish is that I
| shouldn't have to do this. It is equally as annoying as it was
| when I had to drive their web interfaces directly.
| veber-alex wrote:
| Yep.
|
| I had to write code to mount an iso image to a server, set
| the server to boot from it once and reboot the server to
| start the install.
|
| I had to do it for 4 vendors. Dell, Lenovo, Fujitsu and
| Supermicro.
|
| 4 different vendors, 4 different Redfish APIs.
|
| And what's annoying is that Redfish always looks similar so
| it fools you into thinking all the vendors do things the same
| way but no, suddenly some endpoint just doesn't exist in this
| vendors implementation and you have to scour the docs for
| this vendors unique magic uri.
| jpgvm wrote:
| The problem with Supermicro isn't the short report, the problem
| is the secure boot keys were leaked so a huge amount of their
| hardware is impossible to secure because the root of trust is
| broken.
|
| https://arstechnica.com/security/2024/07/secure-boot-is-comp...
| immibis wrote:
| Do you secure boot your systems?
| jpgvm wrote:
| I do but with my own PK. The problem is if the box has run
| untrusted code prior to that you can't trust it anymore.
|
| This is a bigger problem when you have a huge fleet of
| these already rather than my few servers in my home lab
| that I can manually enroll my own keys in.
| wang_li wrote:
| >The problem is if the box has run untrusted code prior
| to that you can't trust it anymore.
|
| If that's your threat model you shouldn't trust your
| computers anyway. You don't know what things have been
| inserted into the chips on the motherboard. You don't
| know what code is in your operating system and
| applications. You don't know what code the controllers on
| your storage devices are running. You don't know if there
| is a cellular chip on your motherboard or a chip that is
| waiting for a certain frequency radio wave to leak all
| your shit. You don't know that all the code on your
| system is RCE free or LPE free. You don't know that there
| aren't any insiders at your software vendors signing bad
| code to send to you. You haven't personally audited the
| binaries on your system or even their source code.
| EvanAnderson wrote:
| > ...the X710 which spent a year+ with broken drivers causing
| silent network failures/downright crashes...
|
| Did the X710 actually become stable? I got burned with them
| pretty early in their life have ended up staying away from them
| ever since.
| sofixa wrote:
| Yep, as soon as the patched drivers were released (1.8 IIRC)
| it became rock solid and we never had any issues with it in
| the 3-4 years afterwards that I spent at the same place.
| PaulHoule wrote:
| Recent Intel CPUs don't look too bad on paper but in practice
| they burn up.
|
| It's not too big of a surprise for me because lifespan will
| eventually become an increasing problem as feature size goes
| down (e.g. a mobile defect can make more trouble more easily.)
| They say it is a problem of the microcode asking for too much
| voltage from the motherboard and that's true, but it is also
| true that chips are going to be more sensitive to environmental
| upsets. Ever since the industry went past 10nm I figured there
| was a chance I'd buy something that was affected by I guess I
| dodged the bullet going with AMD.
|
| I remember loving Intel NICs for performance and Linux
| compatibility back in the day before you just used the NIC that
| came on the motherboard. For that matter I loved Intel SSDs but
| you had to read between the lines to see: (1) you got a little
| better performance in the P95-P99 range than cheaper
| competitors and (2) when you are getting frustrated that your
| computer is slow you are experiencing P95-P99. It's some of the
| reason why I both loved and hated Anandtech because they so
| often missed the plot.
| hexmiles wrote:
| Does anyone know about AMD ST? It is the first time I've seen it
| mentioned. I tried to google it, but without result.
| voxadam wrote:
| AMD ST (Secure Technology) is the official name for what most
| people call AMD PSP (Platform Security Processor).
| xaduha wrote:
| IPMI and other solutions are nice, but what I would like to have
| is a standard serial interface to an UEFI shell running at all
| times. How I access that serial port should be my problem.
| bpye wrote:
| Server hardware often will let you access UEFI via serial.
| You'd still need remote power control no?
| blipvert wrote:
| Silicon Graphics servers used to have a separate serial port
| that proved access to a (very) simple state machine that
| controlled power to the system. Send a 'u' it powered on.
| Send a 'd' it powered down. 's' reported the state (IIRC).
|
| There was literally nothing that could go wrong. Then install
| the OS from the regular console port over the network with
| bootp/tftp/http.
|
| The complexity of DRAC/iLO setups to control an emulation of
| a VGA PC setup blows my mind.
| gnufx wrote:
| > an emulation of a VGA PC setup
|
| IPMI does rather more than than giving you console access,
| though it's serial, not VGA. Typical server BMCs which
| embed IPMI do more again. Not to defend the quality of
| various BMC firmware and support I've encountered...
| blipvert wrote:
| Oh, yes, absolutely - fair point.
|
| It was more the remote power/boot/install process which
| seems unnecessarily complicated these days compared to
| serial.
|
| An elegant weapon for a more civilised age, or something
| ;-)
| qhwudbebd wrote:
| If you're up for DIY, it's easy to turn a sufficiently long
| serial break into a toggle on a reset line and/or power
| button with a couple of discrete components.
|
| A serial break is the only situation in which an RS232 line
| is driven +ve with greater than 90% duty cycle: charge a cap
| slowly on +ve, discharge quickly on -ve (diode), drive a
| mosfet gate to pull reset line low only when it's been +ve
| for quarter of a second or so.
|
| Easily the hardest part of doing this is finding a 'clean'
| way to get a wire attached to the serial RX and GND pins from
| the inside the case rather than bodging something really
| ugly. Some boards have a front-facing serial port, though,
| which has a pin header and cable => easy to tap into.
|
| I'd soldered onto the little legs on a stand off board DB9
| port on one batch of machines I installed, and then ended up
| being thwarted on the next batch of boards which had a
| slightly different (more enclosed) style of DB9 port that
| made it much harder to get at the pins.
|
| Whatever you do along these lines will be infinitely better
| than the insecure, overengineered catastrophe of vendor
| IPMI/BMC firmware. I wish someone less lazy than me would
| make a product along these lines... ;-)
| mjg59 wrote:
| The UEFI boot services that the shell relies upon aren't
| available after the bootloader or OS calls ExitBootServices()
| (the code is literally dropped out of ram and those regions
| handed back to the os) so this is not an easy thing to
| implement
| throw0101d wrote:
| > _IPMI and other solutions are nice, but what I would like to
| have is a standard serial interface to an UEFI shell running at
| all times._
|
| One thing I miss about Sun SPARC (and other Unix) systems: you
| had 'proper' remote access at a very low level.
|
| I always found BIOS/UEFI remote console very fiddly and hit and
| miss (you often have to play with GRUB/kernel settings to get
| input/output).
| grw_ wrote:
| If anyone wants to add remote access capability to machines
| without IPMI you can try with something like the $30 RISC-V
| NanoKVM[1][2]. It provides HDMI capture (and encoding),
| Ethernet/Wifi, ATX power control, and runs a normal linux
| distribution.
|
| [1] https://www.aliexpress.com/item/1005007369816019.html [2]
| https://github.com/sipeed/NanoKVM
| mananaysiempre wrote:
| Notably, you need the $60 "full" version to get the ATX power
| control breakout that attaches to the perverse physically-USB-C
| connector that has the ATX signals (why? why did they make this
| abomination?..). Or make your own, I guess, that's always an
| option (if only USB-C connectors weren't absolute ass to
| solder).
| jdboyd wrote:
| Probably because USBC connectors are really cheap. If this
| was 5 years ago, they probably would be doing something
| terrible with a usb 3 A connector instead.
| riobard wrote:
| I didn't use the ATX aux board because my mini PC does not
| have such headers. I just use Wake-on-LAN from NanoKVM web UI
| to power up my PC :P
| eqvinox wrote:
| There's no Reset-on-LAN though, sadly...
| gsa wrote:
| I use a Zigbee plug to toggle the power off and on, and
| the bios is configured to boot when power is restored.
| riobard wrote:
| Yeah I have this setup too just in case :p
| RamRodification wrote:
| OT: What emotion/message are you trying to convey with
| ":P" in your comments? I never really get these.
| gjvc wrote:
| "my computer is better than your computer" i'm
| guessing...
| riobard wrote:
| No no definitely not that... it's more like "my stuff is
| a bit weird"
| riobard wrote:
| I don't know if it's common but for me ":p" is basically
| the text version of this emoji
|
| [edit]: oops HN does not allow emoji ... go here
| https://emojipedia.org/winking-face-with-tongue
| snvzz wrote:
| The software side is not OSS, it's no better than the
| alternatives.
|
| Yet another KVM that cannot be trusted.
| TMWNN wrote:
| I guess the best choice would be something built on Raspberry
| Pi? Like PiKVM, or TinyPilot (both frequently discussed on
| HN)?
| jdboyd wrote:
| I've been interested in (but haven't yet tried) the BliKVM
| devices that are a for of PiKVM but are on a PCIeb card to
| put the device in the server. First generation from them
| used Pis the second switched to some other Linux SoC.
| justinclift wrote:
| Is this not the software side?
|
| https://github.com/sipeed/NanoKVM
|
| Link copied from another comment:
| https://news.ycombinator.com/item?id=41432060
| dugite-code wrote:
| I was mistaken it seems. Only the front end is open ATM.
|
| > the backend will be open-sourced soon (after the GitHub
| repository reaches 2K stars).
|
| https://en.wiki.sipeed.com/hardware/en/kvm/NanoKVM/system/i
| n...
| justinclift wrote:
| Heh Heh Heh
|
| Wonder if they're serious about that?
|
| Do they know places are around (or at least used to be)
| which will inflate the # of stars for a repo for some
| minimal amount of cash? Like $20 per thousand(?) or so
| when I was reading an expose about that years ago... ;)
| toast0 wrote:
| A closed source $30 external network KVM is less expensive
| (and less featureful) than closed source onboard IPMI which
| typically increases board price by $100-$200.
|
| I think there are some open source IPMI products, but that
| involves talking to an ODM about a run of thousands of boards
| which is a bit much for me; I just want two servers at home.
| I'm tempted by the Gigabyte MC12-LE0 boards that are low cost
| in Germany right now, but getting dedicated server boards
| disrupts my pattern of upgrade a desktop, and give the old
| board to my oldest server.
| riobard wrote:
| I'm playing with one now (the full kit) attached to my toy mini
| PC. It's a little RISC-V device consuming a couple watts. No
| WiFi yet, but it attaches to PC and emulates 4 things in
| addition to capturing HDMI output to a web UI:
|
| 1. A USB keyboard.
|
| 2. A USB mouse.
|
| 3. A USB flash drive (using a partition on its internal SD
| card) for storing bootable ISO to install/rescure systems.
|
| 4. A USB NIC which is very neat. I can use it on the PC to
| expose management SSH port just on that instead of the primary
| NIC, so that the PC has a sorta dedicated IPMI interface.
|
| Newer versions of the device software come with WireGuard and
| Tailscale support, so I can just connect to it over VPN.
|
| Still have some minor issues, but the developers are working
| fast to fix them.
| Ir0nMan wrote:
| Do we know why AliExpress doesn't have the device available for
| sale to US customers?
| qwertox wrote:
| Can an AST2600 BMC be used as a graphics card by Ubuntu Desktop?
| I've read that it has a "2D Video Graphic Adapter with PCIe bus
| interface", which makes it sound like it is exposed to the OS
| like any other graphics card.
|
| The desktop would be there only for troubleshooting, not for
| normal stuff which would usually require an iGPU or better.
|
| If yes, would it work out of the box with Ubuntu?
| grw_ wrote:
| yes, it does-
|
| lspci | grep ASP c2:00.0 VGA compatible controller: ASPEED
| Technology, Inc. ASPEED Graphics Family (rev 52)
| vardump wrote:
| Yes, but it's not going to be speedy, as it is just a dumb
| framebuffer and the maximum resolution is 1920x1080.
|
| The most modern feature is probably double buffering. :-)
| eqvinox wrote:
| Can it? Yes. Do you want to? No. It's essentially just a
| framebuffer, not a GPU; it barely has 2D graphics acceleration.
| Using any kind of modern DE on it will drive you insane within
| minutes. Xfce and classic X11 might work.
| geerlingguy wrote:
| I use it with ASPEED's latest Windows driver at 1080p, and it
| does a passable job on my Ampere Altra system for simple tasks.
|
| Most servers don't run an accelerated server desktop and do
| things like play AAA games and YouTube.
| concerndc1tizen wrote:
| Conspiracy theory: I wonder if the ____KVM people are betting
| that most people won't flash their own firmware, and just keep
| using what was provided, which may contain malware, both for
| accessing the network, and controlling the connected hardware.
| Animats wrote:
| A big question with IPMI is what the default keys are. If someone
| manages to install some extra IPMI keys somewhere in the supply
| chain, or there's a default key, they can remote admin the
| computer.[1]
|
| [1] https://www.rapid7.com/blog/post/2013/07/02/a-penetration-
| te...
| gnufx wrote:
| Well, it should always be on a physically isolated network, and
| avoid BMCs which share their NIC with the system.
| jabroni_salad wrote:
| An additional quote from that link:
|
| > In short, the authentication process for IPMI 2.0 mandates
| that the server send a salted SHA1 or MD5 hash of the requested
| user's password to the client, prior to the client
| authenticating.
|
| ipmi also has a maximum password length of 20 characters. In
| realworld you have to note that we only expect hashes to remain
| secret from known pentesters who have a limited contractual
| runtime, not a real attacker with unlimited time on their
| hands.
|
| I'm very critical of this. It's been in the spec for 20 years.
| Is not the whole point of software that it is easier to change
| than hardware? It's easy to say "you should put this on a vlan"
| but I pretty much ALWAYS see ipmi on the business network in my
| assessments. If you put dumb stuff in your defaults then you're
| gonna end up with dumb stuff all over the globe and that
| includes every business without a knowledgeable security
| administrator that decides to "buy a server".
| _joel wrote:
| ipmitool has dug me out of many sticky situations in the past,
| although it needs to be handled as proper oob, in terms of
| network isolation.
| OSI-Auflauf wrote:
| Seen devices with IPMI that had by design unauthenticated admin
| login to the IPMI from the host side that was not removable. They
| also could flash IPMI firmware from the host. So if your server
| with such an IPMI is infected you can't trust reimaging it via
| IPMI because that can be hijacked as well.
| dfox wrote:
| I would consider that mostly a feature. The situation where
| that is useful (you somehow lost the credentials for BMC, but
| have root access to the host) is in my experience significantly
| more common (I see that multiple times in a year) than attacker
| implanting stuff into the BMC firmware (never seen that).
|
| Obviously if you rent out whole physical machines and automate
| the provisioning by IPMI, then the last thing you want is the
| customer having admin access to the BMC.
|
| Dell iDRAC has an interesting feature that allows you to make
| all of the BMC configuration read-only which can only be
| disabled by factory resetting the iDRAC by means of physical
| (and IIRC not exactly documented) switch on the BMC board.
| (Well, it is still _i_DRAC as in "integrated", but on current
| higher-end PowerEdges the iDRAC is separate OCP-like card, but
| well, the system does not work without it)
| ggm wrote:
| Hands up if you used a real weasel! Cute advertising imagery.
|
| I depend on idrac. The newer one is bearable. The old Java was
| torrid.
| dfox wrote:
| As for the Intel ME and AMD PSP: these things have a huge role in
| getting the CPU to state that looks like x86 PC so that the host
| firmware can actually run. The complexity of the initialization
| got so high, that doing that in SW on a separate embedded core
| that is "well-behaved" and can be programmed in C makes more
| sense than trying to write all that logic in weirdly limited
| assembly that is inherent in traditional early stage BIOS startup
| code.
|
| I believe that at least on some HPE ProLiants some part of this
| low-level initialization is in fact done by iLo, which at that
| boot stage also directly controls the framebuffer (on G10 you can
| even briefly see a flash of message like "handing over the
| console to host", after which the display reinitializes and
| starts showing the function key legends). I assume that Dell does
| something similar, but in the early stages you simply get "Please
| wait" and giant throbber (and not any kind of progress
| indication).
| swiftcoder wrote:
| Does anyone else see a pixel or so of mismatch between the
| background and the text when scrolling?
| EvanAnderson wrote:
| In the late 90s I installed several Intel-produced servers (i.e.
| Intel reference platforms shipped as "bare bones" computers w/
| RAM and storage to be provided by the integrator) that used the
| LANDesk Server Manager Pro[0] software and "Emergency Management
| Card" (EMC) for "lights out" management. (I'm talking about boxes
| like the AP450GX, BB440FX, RC440FX-- Pentium Pro to early Pentium
| II timeframe.)
|
| Given how reference code for the x86 platform never dies I've
| often wondered how much of the current IPMI edifice dates back to
| this hardware and software. As a point of note, the default
| password for the Intel LANDesk Emergency Management Card was
| "calvin". If you worked with earlier versions of the Dell iDRAC
| this will be a familiar password. I assume it's not coincidence.
| (As an aside: I was told by an Intel employee that the code name
| for the EMC was "Hobbes" but I can't find that documented
| anywhere.)
|
| You see versions of the EMC periodically on eBay.[1] There were
| both ISA and PCI versions. They're an x86 PC on a card. Some
| (all?) of them have an integral UPS. They had a PCMCIA slot to
| attach out-of-band management, an external power supply, and
| connected to the server motherboard through both the card's host
| bus interface and via proprietary connectors.
|
| I've downloaded firmware for some of the EMC versions and looked
| it over. It looks like some of the EMC versions an embedded DOS
| machine. (This has been one of my idle "when I get to it"
| projects and I haven't actually tried to reverse-engineer any of
| the code or get it running under qemu, though I'd like to.)
|
| Third-party companies who sold the Intel reference platforms
| (Unisys, Fujitsu, ALR/Gateway, NCR) offered these cards, too.
| It's a good giveaway that they're using an Intel reference
| platform when you see the card on-offer. References to "LDSM" are
| a giveaway.
|
| It'd be really wild if anybody on here knows anything about the
| heritage.
|
| [0]
| https://www.intel.com/pressroom/archive/releases/1998/ld1030...
|
| [1]
| https://web.archive.org/web/20240903131630/https://www.ebay....
| Neil44 wrote:
| Not a fan of the web based OOB management, it's convenient for
| now but this morning I had a right faff trying to connect to an
| old iLO3 service, which is new enough to force https but old
| enough that all it's protocols are deprecated on modern machines.
| I've used vPro and found it almost deliberately difficult to use,
| and poorly documented. It could be so easy but just isn't. Like
| so many things... shakes fist at sky.
| toast0 wrote:
| > which is new enough to force https but old enough that all
| it's protocols are deprecated on modern machines
|
| This is the danger of https only. http services may be
| insecure, but don't ever expire; https implementations will
| expire without updates, and there's not going to be updates on
| these, and it's unknowable when the expiration will be.
| quesera wrote:
| Firefox can be configured to allow obsolete SSL/TLS ciphers:
| about:config security.tls.version.min = 0
|
| Obligatory: Do not leave this setting in place for routine
| browsing. Reset to default (currently 3), or only run in a
| dedicated profile, etc.
| neilv wrote:
| > _IPMI is very common in enterprise servers but very rare
| elsewhere, much to the consternation of people like me that don
| 't have the space or noise tolerance for a 1U pizzabox in their
| homes. If you are trying to stick to compact or low-power
| computers, you'll pretty much have to go without._
|
| One option: you can get IPMI on an Atom-based Supermicro MicroATX
| board that can be cooled quietly with a small Noctua fan, in a
| short-depth 1U chassis.
|
| I have an older one at home. The quietness and small size made it
| much more attractive than the Dell R2x0 models I saw. And it has
| server features like IPMI and ECC RAM that made it more
| attractive than a mini-PC, and it was also more reliable than a
| flimsy RasPi. (I'd be happy with a Dell quiet short-depth 1U with
| comparable characteristics, but I'm not aware of one.)
|
| Personally, I wouldn't plug the IPMI port into my main LAN, but
| it's come in handy isolated from that, and was also a little fun
| to play with.
| organsnyder wrote:
| I keep my IPMI and other management interfaces segregated to a
| separate management VLAN that's only accessible via a dedicated
| VPN.
| temp0826 wrote:
| This is probably the minimum you should do. I remember a time
| when IPMI on (some?) Supermicro boards was really really
| insecure. From the IPMI client, you could set the encryption
| mode to "0" ("null encryption" or something, I dunno it's
| been years)- setting it allowed you to _bypass_ the password
| completely. Assume if you can touch the IPMI, the system is
| yours.
| crest wrote:
| IPMI is still a festering cesspool no matter which vendor.
| Assume that layer 3 access to the IPMI grants you
| unrestricted persistent code execution on the managed
| system and design for with it in mind.
|
| Restrict access as tightly as possible. If you only need to
| power up/down/reset the system and access the serial
| console most IPMI implementations expose that via SSH. A
| small SSH proxy that exposes only those features would be a
| good investment. e.g. `ssh bastion
| [status|up|down|cycle|reset|console] <server>`. You could
| probably write it in <100 lines as SSH forced command.
| Deploy the SSH client to a different VRF (aka network
| namespace/vnet jail/rdomain) than the SSH server to make it
| harder to leak traffic from the bastion.
| nyrikki wrote:
| That was Dell iDRAC boards had a bug, where while they
| asked for a password, they would accept ANY password.
|
| IPMI is like typical IoT, a rarely updated full system with
| often questionable code quality.
|
| Private VLANs and separate interfaces on dedicated ports is
| not out of line.
|
| Often shared interfaces with vlan isolation depends on the
| IPMI honoring that isolation.
|
| By Private VLANs I am talking about RFC 5517 style, a
| switch level feature where IPMI cannot reach other IPMI
| interfaces.
| doublepg23 wrote:
| There's ASRock Rack boards with X470, X570, X670, etc. chipsets
| that take standard AM4/AM5 chips and have loads of server
| features like IPMI and ECC (I think that's already common on
| the AMD side). I managed to snag a 5950x to throw into mine but
| I was rocking a 5600G for a while. They're mATX/ATX too so
| they'll fit in a regular case and take a regular power supply -
| no racks needed!
| dboreham wrote:
| We have a bunch of those atom boards. It was the first thought
| I had reading the article, which takes a while to say that IPMI
| means large machines.
| ultra_nick wrote:
| Dell is working pretty hard to replace IPMI with Redfish these
| days. Redfish is an out of band management protocol like IPMI,
| but using an HTTP/JSON REST API that doesn't assume a secure
| management network.
|
| https://www.dmtf.org/standards/redfish
| jayofdoom wrote:
| Don't just give Dell the credit here :)
|
| OpenStack Ironic (ironicbaremetal.org) now ships a redfish
| driver that we have tested working on: - Dell machines - HPE
| machines (ilo 5 and 6) - SuperMicro - Most generic redfish
| endpoints as deployed by bulk "off label" server places
|
| A lot of folks in the provisioning automation space worked hard
| to try and enhance the state of the art (including some folks
| at Dell who were great :D) and get redfish accepted.
|
| It's a bit weird seeing this post; from our perspective, IPMI
| is a dangerous, attractive nuisance: it's nearly impossible to
| properly secure and has a lot more bad failure scenarios than a
| protocol built on http, that most of the time can support TLS
| with custom CAs and similar.
| johnklos wrote:
| IPMI has its place, but it clearly shows how none of us can trust
| commercial entities to properly support hardware long term. The
| OS that runs on IPMI is usually relatively secure when the system
| is new, but as soon as there's a new CPU socket, the manufacturer
| starts caring less and less about keeping the older systems up to
| date, even if the same IPMI hardware is on both old and new
| boards.
|
| If we could load our own OS on to the IPMI hardware, it's be
| worlds more useful, because we could safely connect it straight
| to the Internet. As it is, we need sideband communications (VPN,
| port forward over ssh, separate network segment, whatever), which
| makes it so that we end up with lots of extra hardware and
| configurations just to support IPMI. For large installations, the
| extra cost is better aggregated, but for small installations,
| it's a rather large burden. Heck - for colocating a single
| machine, it's not even worth it.
|
| Since it's not possible to run IPMI directly on the Internet
| securely, I've ended up pairing machines with Pis of some sort.
| Doing this makes it just as easy - heck, easier - to use a serial
| port. That means we're back to the standard serial port control
| we've had since the days of VAXen and Sun and Alpha systems,
| which, the more I think about it, made much more sense than a
| network interface that's insecure.
| peter_d_sherman wrote:
| My takeaways:
|
| >"Out-of-band management, sometimes also called lights-out
| management, identifies a capability that is almost unheard of in
| clients. A separate, smaller management computer allows for
| remote access to a server even when it is, say, powered off. The
| terms out-of-band and in-band in this context emerge from their
| customery uses in networking and telecom, meaning that out of
| band management is performed without the use of the standard (we
| might say "data plane") network connection to a machine. But in
| practice they have drifted in meaning, and it is probably better
| to think of _out-of-band management_ as meaning that the
| _operating system and general-purpose components are not
| required_. This might be made clearer by comparison: a very
| standard example of in-band management would be SSH, a service
| provided by the software on a computer that allows you to
| interact with it. Out-of-band management, by contrast, is
| provided by a dedicated hardware and software stack and does not
| require the operating system or, traditionally, even the CPU to
| cooperate. "
|
| [...]
|
| >"The IPMI software runs on a processor conventionally called the
| baseboard management controller or BMC, and the terms IPMI and
| BMC are sometimes used interchangeably. Lights-out management or
| LOM is mostly an obsolete term but sticks around because HP(E) is
| a fan of it and continues to call their IPMI implementation
| Integrated Lights-Out. The BMC should not be confused with the
| System Management Controller or SMC"
|
| [...]
|
| "IPMI offers "sideband" networking. With sideband management, the
| BMC communicates directly with the same NIC that the operating
| system uses. The implementation is a little bit weird: the NIC
| will pretend to be two different interfaces, mixing IPMI traffic
| into the same packet stream as host traffic but using a different
| MAC address."
|
| [...]
|
| "Details of network features vary between IPMI implementations,
| but there is a standard interface on UDP 623 that can be used for
| discovery and basic commands. There's often SSH and a web
| interface, and VNC is pretty common for remote console."
|
| This is arguably the best educational article on IPMI and related
| subsystems (I did not know about 90% of this!) that I've read for
| a long time.
|
| Well worth a re-read in the future...
| saltminer wrote:
| > This is not to make excuses for Intel ME, which is entirely
| unauditable by third parties and has harbored significant
| security vulnerabilities in the past. But, remember, we all use
| one processor architecture from one of two vendors, so Intel
| doesn't have a whole lot of motivation to do better. Lest you
| respond that ARM is the way, remember that modern ARM SOCs used
| in consumer devices have pretty much identical capabilities.
|
| I actually took advantage of one such security vulnerability to
| unlock my old Moto X's bootloader! The details of the exploit are
| quite interesting:
|
| https://bits-please.blogspot.com/2015/03/getting-arbitrary-c...
|
| https://bits-please.blogspot.com/2015/08/exploring-qualcomms...
|
| https://bits-please.blogspot.com/2015/08/full-trustzone-expl...
|
| https://bits-please.blogspot.com/2016/02/unlocking-motorola-...
___________________________________________________________________
(page generated 2024-09-03 23:01 UTC)