[HN Gopher] EC2 Serial Console
___________________________________________________________________
EC2 Serial Console
Author : TangerineDream
Score : 52 points
Date : 2021-03-30 21:02 UTC (1 hours ago)
(HTM) web link (aws.amazon.com)
(TXT) w3m dump (aws.amazon.com)
| banana_giraffe wrote:
| I suppose this isn't much of a surprise, but it's kinda sorta
| pointless for Windows instances. I was hoping it would be
| something, but it's just a dump of the standard Windows system
| log output.
|
| And adding to the fun, watching it on an initial instance bootup
| seems to block the process AWS uses to grab the encrypted
| password out of the log. So, it's not useful, and makes the
| instance a bit hard to remote into.
| my123 wrote:
| You can enable the SAC for Windows. Then, you can have a
| command prompt too over serial.
| nn3 wrote:
| "Accessing your serial console is only 0.04$/per character,
| except if you bid on peak console then ..."
|
| I made that up, but it would totally not surprise anyone, would
| it?
| WrtCdEvrydy wrote:
| I did certifications and couldn't believe that it wasn't
| $5/missed question the first time around :D
| commandlinefan wrote:
| Not gonna lie... if I ever got into a situation where I needed
| serial access to an EC2 instance, I'd just retire the EC2
| instance and spin up another one.
| rsstack wrote:
| If something like this happens to an instance, I don't trust
| the VM's state even after recovery and I'd retire & replace
| regardless. But I'd love to have serial access to do a root-
| cause analysis to prevent the issue from occurring in the
| future.
| pickle-wizard wrote:
| Most of the time, that is what I'd suggest to do. However this
| is like the 4 wheel drive on my truck. I usually don't need it,
| but when I need it, I really in a bind and glad to have it.
| sargun wrote:
| This is amazing.
| bombcar wrote:
| I wonder if this is implemented at the hypervisor level (gives
| you access to the "console") or if this is actually implemented
| as a serial port.
| rwmj wrote:
| What would be the difference exactly? The console of a Linux VM
| _is_ either the emulated serial port or the emulated VGA device
| + emulated keyboard.
|
| I once implemented a tool called virt-dmesg which read out the
| log_buf from a running Linux kernel (surprisingly useful for
| those tricky crashes, but difficult from a maintenance point
| view so the tool is now abandoned). I suppose that's the
| closest you could get to a "real" console at the hypervisor
| level.
| bombcar wrote:
| A "text console" is almost always there but a serial console
| has to be enabled on the instance itself.
|
| Based on a quick SSH it looks like it's a serial thing:
| root 1221 /sbin/agetty -o -p -- \u --keep-baud
| 115200,38400,9600 ttyS0 vt220 root 1229
| /sbin/agetty -o -p -- \u --noclear tty1 linux
| wahern wrote:
| These days hosts and guests often implement virtio-serial, so
| there's no need to emulate serial hardware, per se. For
| example, AFAIU, OpenBSD's VMM doesn't implement any hardware
| emulation, but relies on the guest supporting virtio-serial,
| virtio-net, virtio-block, and virtio-rng (so the guest isn't
| at the mercy of RDRAND). Though, for some reason EC2 seems
| allergic to adopting the virtio device model.
| Daviey wrote:
| This is super useful if you are in the game of building images,
| or highly tweeking init systems and/or the kernel.
|
| For general consumers, not much value IMHO.
| whalesalad wrote:
| I see a ton of value when you have instances on a private
| network with no shell access and want to debug them without
| setting up a bastion instance.
| Spivak wrote:
| Yep, this might very well enable whole deployments where
| there is no SSH access anywhere.
| petethepig wrote:
| This seems to be such an obvious feature that I'm surprised they
| are only adding this now.
|
| I'm not an expert in hypervisors or anything like that and so I'm
| wondering what was stopping them from adding it in the past?
| cperciva wrote:
| The biggest reason is probably security. This is not something
| you want to take chances with.
| pwarner wrote:
| I get this sense they are very conservative in their feature
| set there? Nested VMs for example are supposed in GCP, Azure
| and Oracle clouds I think, but not AWS. VM migration too I
| think exists in GCP at least, but not AWS. It's interesting.
| oblio wrote:
| AWS is super slow to deprecate or remove features and
| services.
|
| A few people were noting that SimpleDB has been deprecated,
| it's not listed in the AWS web console, you can't find docs
| for it anymore, but if you have a running instance, your
| service API calls still work. And I think there have been
| many deprecation warnings since, plus migration messages. But
| they don't want to break existing clients.
|
| I'm guessing this is a similar case where they want to be
| really, really sure that it's worth offering the service.
| chousuke wrote:
| Likely they just had no need for it themselves. Might be that
| they didn't prioritize having a feature over the risk of
| someone taking over their hypervisors thanks to a buggy serial
| port emulator.
|
| Pretty much all hypervisors support serial consoles, but
| usually those interfaces are limited to trusted admins. For
| something like AWS, they'll also have to connect it from the
| hypervisor hosts into their public UI, and they can't trust the
| users.
| Waterluvian wrote:
| Is this the cloud equivalent to hooking up a monitor and keyboard
| to a server?
|
| I remember having an EC2 terminal in the browser years ago and
| recently I went back and it seemed far more locked down.
| monocasa wrote:
| Oh thank god. I had a customer once that erased their SSH keys,
| and had a running database cluster on EC2 that they couldn't get
| access too anymore. That was... fun.
|
| This is a long time coming.
| banana_giraffe wrote:
| I've used SSM Agent to get out similar hot spots in the past,
| but this will be a nice option for those instances that are
| somehow both broken to the point where you can't connect to
| them and still important enough that they need to be up _now_.
| koolba wrote:
| Sure there's down time involved, but you can always stop and
| mount the volumes elsewhere and then update the SSH keys.
| monocasa wrote:
| They could not accept downtime. They were also using a
| database that had it's consistency guarantees scaled back for
| "web scale" reasons and were terrified of what a hard reset
| would do. Yes, the VM could just disappear randomly anyway,
| which made doing that a very poor choice.
|
| There were a whole lot of questionable prior decisions that
| had been made that did not help.
|
| You learn a lot in situations like this about being helpful
| with the customer without being judgey. Get them back on
| their feet with a smile when they thought they were screwed
| keeps a lot of contracts around.
| growse wrote:
| Probably shouldn't have erased their SSH keys then.
| monocasa wrote:
| I mean, giving customers the finger is one strategy.
| Bending over backwards for the ones that pay out the ass
| for a good support contract is another.
|
| "Your business might not exist if our engineers didn't
| dedicate themselves to your problem even though we didn't
| need to according to our SLA" goes a real long way at
| reup negotiation time with the right sized business (not
| too big, not too small).
| koolba wrote:
| To use something like this you'd still need an SSH root
| password setup in advance. So not much different than not
| losing your SSH keys or creating fallback ones.
|
| One other option is to exploit a bug in managed software to
| escape to a shell. One man's CVE or backdoor, is another
| support engineers magic sword to save the day.
| monocasa wrote:
| Is that strictly true?
|
| Other implementations I've seen drop you right into a
| root shell, relying on equivalents to IAM to govern
| access to the other side of the virtual serial port
| rather than machine local permissions.
| wahern wrote:
| Depends on how getty (or other local TTY manager) is
| setup. Most (if not all) Linux and BSD distributions
| attach login(1) to console and serial TTYs by default,
| and login(1) will normally require password
| authentication. Though, whether login prompts for a
| password is (I think) also a function of flags in
| /etc/passwd (or /etc/shadow or equivalent), so you may be
| able to login as `root` or `$USER` without entering a
| password if the account was setup that way.
| tingletech wrote:
| if you have serial console access, can't you key-combo it
| to boot into single user mode?
| wahern wrote:
| Not by default as far as I'm aware. Though, it's been a
| long time since I've used a desktop Linux distribution.
| My experience is mostly with serial access for servers--
| Linux, BSD, and Solaris. I much prefer serial ports for
| backup administrative access as there's little chance of
| a misconfiguration. Whereas w/ port failover and other
| IPMI, BMC, etc non-sense if you disconnect the ethernet
| cable from the dedicated network management port you may
| still have admin/admin access live on the network.
| commandlinefan wrote:
| That's what I was thinking... I mean, this is cool and all,
| but the only reason I'd ever muck around at this level is if
| I was actually responsible for the physical hardware and
| there was a cost associated with replacing it. The whole
| point of moving to the cloud is to pay somebody else to have
| to worry about anything I can comprehend needing this for.
| monocasa wrote:
| A lot of medium sized companies moved to the cloud without
| understanding the tradeoffs. And they have big pockets when
| it matters.
___________________________________________________________________
(page generated 2021-03-30 23:00 UTC)