[HN Gopher] Why does SSH send 100 packets per keystroke?
___________________________________________________________________
Why does SSH send 100 packets per keystroke?
Author : eieio
Score : 195 points
Date : 2026-01-22 19:27 UTC (3 hours ago)
(HTM) web link (eieio.games)
(TXT) w3m dump (eieio.games)
| snowmobile wrote:
| > That 20ms is a smoking gun - it lines up perfectly with the
| mysterious pattern we saw earlier!
|
| Speaking of smoking guns, anybody else reckon Claude overuses
| that term a lot? Seems anytime I give it some debugging question,
| it'll claim some random thing like a version number or whatever,
| is a "smoking gun"
| Hikikomori wrote:
| It's a smoking gun of Claude usage.
| lloydatkinson wrote:
| smoking gun, you're absolutely right, good question, em dash,
| "it isn't just foo, it's also bar", real honest truth, brutal
| truth, underscores the issue, delves into, more em dashes, <20
| different hr/corporate/cringe phrases>.
|
| It's nauseating.
| cubano wrote:
| Come on...haven't we all had to deal with the crazy smart
| lead who was loaded with those same types of annoying tics?
|
| Considering what these LLMs bring to the table, I think a
| little tolerance for their cringe phrases is in order.
| jcynix wrote:
| It's what they read on The Internets when training, so don't
| expect them to generate new phrases, other than what they
| learned from it?
| Terretta wrote:
| ### The answer that fits everything (and what to do about
| it)
| calvinmorrison wrote:
| cant wait for chatgpt to make me read about grandmas
| secret recipe and scroll through 6 ads to see the
| ingredients for my chicken teriyaki dinner
| jcynix wrote:
| Maybe we need a real AI which creates new phrases and
| teaches the poor LLMs?
|
| Looking back we already had similar problems, when we had
| to ask our colleagues, students, whomever "Did you get
| your proposed solution from the answers part or the
| questions part of a stackoverflow article?" :-0
| MaxBarraclough wrote:
| That's the point though, it doesn't reflect human usage of
| the word. If _delve_ were so commonly used by humans too,
| we wouldn 't be discussing how it's overused by LLMs.
| hamdingers wrote:
| You might find this a fun read:
| https://en.wikipedia.org/wiki/Wikipedia:Signs_of_AI_writing
| Telemakhos wrote:
| I've love to delve into that.
|
| https://pshapira.net/2024/03/31/delving-into-delve/
| bdamm wrote:
| Without knowing how LLM's personality tuning works, I'd just
| hazard a guess that the excitability (tendency to use excided
| phrases) is turned up. "smoking gun" must be highly rated as a
| term of excitability. This should apply to other phrases like
| "outstanding!" or "good find!" "You're right!" etc.
| yread wrote:
| ChatGPT too. And "lines up perfectly" when it doesnt actually
| line up with anything
| dave78 wrote:
| Same with Gemini.
| MonkeyClub wrote:
| You can absolutely see this pattern in Gemini in 2026.
|
| Btw, is the injection of "absolutely" and "in $YEAR"
| prevalent in other LLMs as well, or is it just in Gemini's
| dialect?
| cristoperb wrote:
| It's just Gemini. I'm guessing they changes the system
| prompt for the new year or something, but it's pretty
| annoying.
| locallost wrote:
| I chuckled out loud. It's funny cause it's true.
| redwall_hp wrote:
| "You're so right, that nice catch lines up perfectly!"
| smallmancontrov wrote:
| It's not just a coincidence, it's the emergence of spurious
| statistical correlations when observations happen across
| sessions rather than within sessions.
| f1shy wrote:
| You can add an M-dash, and we completed the bs-bingo. :)
| cipehr wrote:
| I don't think claude has even once used this in my
| conversations (Claude Desktop, Claude Code, Voice
| conversations...) Sycophancy, yes absolutely!
|
| Maybe it has something to do with your profile/memories?
| eieio wrote:
| Yes! While this post was written entirely by me, I wouldn't be
| surprised if I had "smoking gun" ready to go because I spent so
| much time debugging with Claude last night.
| gf000 wrote:
| Reminds me of ethimology nerd's videos. He has some content
| about how LLMs will influence human language.
| grim_io wrote:
| The "maybe" of yesterday is the "you're absolutely right!"
| of tomorrow.
| hinkley wrote:
| Some day in the future we will complain about AIs with a
| 2015 accent because that's the last training data that
| wasn't recursive.
| ranger_danger wrote:
| shouldn't it be "human language influences human language"?
| rubslopes wrote:
| It's interesting how LLMs influence us, right? The opposite
| happened to me: I loved using em dashes, but AI ruined it for
| me.
| andai wrote:
| I still love using emdashes, and people already thought I
| was a robot!
|
| https://xkcd.com/3126/
|
| _Soon the Andy 3000 will finally be a reality..._
| thadt wrote:
| I used to love using em dashes.
|
| I still do - but I used to, too.
| jcynix wrote:
| You might see certain phrases and mdashes ;-) rather often,
| because ... these programs are trained on data written by
| people (or Microsoft's spelling correction) which overused them
| in the last n years? So what should these poor LLMs generate
| instead?
| simonjgreen wrote:
| I see it from GPT5 too a lot
| Fnoord wrote:
| > Speaking of smoking guns
|
| Oh shoot! A shooting.
|
| So the TL;DR of this post is: don't change this setting unless
| you know what you're doing.
| kevin_thibedeau wrote:
| Chastise it with a reminder that you're using smokeless powder.
| observationist wrote:
| Or the "Eureka! That's not just a smoking gun, it's a classic
| case of LLMspeak."
|
| Grok, ChatGPT, and Claude all have these tics, and even the pro
| versions will use their signature phrases multiple times in an
| answer. I have to wonder if it's deliberate, to make detecting
| AI easier?
| WesolyKubeczek wrote:
| A computational necromancer has likely figured out a way to
| power a data center by making Archimedes spin in his grave
| very fast.
| nurettin wrote:
| At this I'm just so glad that "you're absolutely right!" phase
| is over.
| layer8 wrote:
| Yes, it's kind of a corpus delicti. ;)
| jcims wrote:
| I'm working on a little SRE agent to pre-load tickets with
| information to help our on-call and I'm already tired of Claude
| finding 'smoking guns'.
| HPsquared wrote:
| They love cliches, and hate repeating the same words for
| something (repetition penalty) so they'll say something like
| "cause" then it's a "smoking gun" then it's something else
| ycombinatrix wrote:
| You can also use TCP_CORK to reduce the number of packets without
| any increased latency.
|
| Disabling TCP_NODELAY would also reduce number of packets + be
| portable & simpler to implement - but would incur a latency
| penalty.
| eieio wrote:
| Oh wow - I've never heard of TCP_CORK before. Without disabling
| pings I'd still pay the cost of _receiving_ way more packets,
| but maybe that 'd be tolerable if I didn't have to send so many
| pongs. This is super handy; excited to play around with it.
|
| I _am_ aware of TCP_NODELAY (funny enough I recently posted
| about TCP_NODELAY to HN[1] when I was thinking about it for the
| same game that I wrote about here). But I think the latency hit
| from disabling it just doesn 't work for me.
|
| [1] https://news.ycombinator.com/item?id=46359120
| danudey wrote:
| Haven't heard of TCP_CORK, very interesting.
|
| For people who don't feel like googling it:
|
| 1. You TCP_CORK a socket
|
| 2. You put data into it and the kernel buffers it
|
| 3. If you uncork the socket, or if the buffer hits MSS, the
| kernel sends the packet
|
| Basically, the kernel waits until it has a full packet worth of
| data, or until you say you don't have any more data to send,
| and then it sends. Sort of an extreme TCP_YESDELAY.
|
| See https://catonmat.net/tcp-cork for where I learned it all
| from.
| cheschire wrote:
| I enjoyed this write up as it touched on several topics I enjoy
| reading about.
|
| Also I was unfamiliar with SSH being vulnerable in the past to
| keystroke timing!
| pixl97 wrote:
| https://news.ycombinator.com/item?id=37307708
|
| 2023 discussion about it here.
| dathinab wrote:
| > Keystroke obfuscation can be disabled client-side.
|
| please never do that (in production)
|
| if anyone half way serious tries they _will_ be able to break you
| encryption end find what you typed
|
| this isn't a hypothetical niche case obfuscation mechanism, it's
| a people broke SSH then a fix was found case. I don't even know
| why you can disable it tbh.
| lazypenguin wrote:
| They literally explain the mechanism in the post and then
| explain why the security tradeoff made sense for their ssh
| game.........
| shadowgovt wrote:
| But they'd have to be on the same network as me to do that
| attack, right?
| benlivengood wrote:
| Yep, like ECHELON and friends are. The metadata recorded
| about your (all of our) traffic is probably enough to perform
| the timing attack.
| shadowgovt wrote:
| Hey, if ECHELON snuck a listener into my house, where six
| devices hang out on a local router... Good for them,
| they're welcome to my TODO lists and vast collection of
| public-domain 1950s informational videos.
|
| (I wouldn't recommend switching the option off for anything
| that could transit the Internet or be on a LAN with
| untrusted devices. I _am_ one of those old sods who doesn
| 't believe in the max-paranoia setting for things like "my
| own house," especially since if I dial that knob all the
| way up the point is moot; they've already compromised every
| individual device at the max-knob setting, so a timing
| attack on my SSH packet speed is a waste of effort).
| advisedwang wrote:
| That doesn't sound right to me. This obfuscation isn't about a
| side-channel on a crypto implementation, this is about
| literally when your keystrokes happen. In the right
| circumstances, keystroke timing can reduce the search space for
| bruteforcing a password [1] but it's overstating to describe
| that as broken encryption.
|
| [1] https://people.eecs.berkeley.edu/~daw/papers/ssh-use01.pdf
| Mystery-Machine wrote:
| THANK YOU!
|
| I'm baffled about this "security feature". Besides from this
| only being relevant to timing keystrokes during the SSH
| session, not while typing the SSH password, I really don't
| understand how can someone eavesdrop on this? They'd have to
| have access to the client or server shell (root?) in order to
| be able to get the keystrokes typing speed. I've also never
| heard of keystroke typing speed hacking/guessing keystrokes.
| The odds are very low IMO to get that right.
|
| I'd be much more scared of someone literally watching me type
| on my computer, where you can see/record the keys being
| pressed.
| eikenberry wrote:
| It is to prevent timing attacks but there are many ssh use
| cases where it is 100% computer to computer communications
| where there is no key based timing attack possible.
| OneDeuxTriSeiGo wrote:
| There is an argument that if:
|
| - you are listening to an SSH session between devices
|
| - and you know what protocol is being talked over the
| connection (i.e. what they are talking about)
|
| - and the protocol is reasonably predictable
|
| then you gain enough information about the plaintext to start
| extracting information about the cipher and keys.
|
| It's a non-trivial attack by all means but it's totally
| feasible. Especially if there's some amount of observable
| state about the participants being leaked by a third party
| source (i.e. other services hosted by the participants
| involved in the same protocol).
| eikenberry wrote:
| I agree it is more nuanced than a simple 'good for
| computer-to-computer' and 'bad for person-to-computer'. I'm
| sure there are cases where both are wrong but I don't think
| that necessarily changes that it makes a reasonable
| baseline heuristic.
| Romario77 wrote:
| this only works for manually typed text, not computer to
| computer communication where you can't deduce much from
| what is being "typed" as it's not typed but produced by a
| program to which every letter is the same and there is no
| different delay in sending some letters (as people have
| when typing by hand)
| Mystery-Machine wrote:
| I'd love to hear more about this kind of attack being
| exploited in the wild. I understand it's theoretically
| possible, but...good luck! :)
|
| You're guessing a cipher key by guessing typed characters
| with the only information being number of packets sent and
| the time they were sent at. Good luck. :)
| PhilipRoman wrote:
| I haven't given this more than 5 seconds of thought, but
| wouldn't it make sense to only enable the timing attack
| prevention for pseudo-terminal sessions (-t)?
| simplicio wrote:
| The fix seems kind of crazy though, adding so much traffic
| overhead to every ssh session. I assume there's a reason they
| didn't go that route, but on a first pass seems weird they
| didn't just buffer password strokes to be sent in one packet,
| or just add some artificial timing jitter to each keystroke.
| bot403 wrote:
| I'm just guessing but this chaff sounds like it wouldn't
| actually change the latency or delivery of your actual
| keystrokes while buffering or jitter would.
|
| So the "real" keystrokes are 100% the same but the fake ones
| which are never seen except as network packets are what is
| randomized.
|
| It's actually really clever.
| kevin_thibedeau wrote:
| SSH has no way of knowing when a password is being typed. It
| can happen any time within the session after SSH auth.
| swiftcoder wrote:
| > Obviously forking go's crypto library is a little scary, and
| I'm gonna have to do some thinking about how to maintain my
| little patch in a safe way
|
| This should really be upstreamed as an option on the ssh library.
| Its good to default to sending chaff in untrusted environments,
| but there are plenty of places where we might as well save the
| bandwidth
| eikenberry wrote:
| +1... Given how much SSH is used for computer-to-computer
| communication it seems like there really should be a way to
| disable this when it isn't necessary.
| jacquesm wrote:
| In practice I've never felt this was an issue. But I can see
| how with extremely low bandwidth devices it might be, for
| instance LoRa over a 40 km link into some embedded device.
| geocar wrote:
| Hah no.
|
| Nobody is running TCP on that link, let alone SSH.
| jacquesm wrote:
| https://github.com/markqvist/Reticulum
|
| and RNode would be a better match.
| mkj wrote:
| It looks like it is only applied for PTY sessions, which most
| computer-computer connections wouldn't be using.
|
| https://github.com/openssh/openssh-
| portable/blob/d7950aca8ea...
| BoppreH wrote:
| Yes, but I wouldn't be surprised if the change is rejected. The
| crypto library is _very_ opinionated, you 're also not allowed
| to configure the order of TLS cipher suites, for example.
| mystraline wrote:
| In my experience, I call them "Suck-u-rity engineers".
|
| They're the "wellakshually" types, with "what-ifs" of
| increasingly inane and stupid conditions. And add in the CVE
| seekers looking for anything to grab on to... Yeah, thats why
| they're not security, but suck-u-rity.
|
| Ive got users to defend. And sure, security is absolutely a
| part. But for those diminishing returns without a real
| exploit, I'm not doing those. Or if I'm required by some
| regimen, I'm doing those in the least impact to my userbase.
| They have enough to fight with tech-wise without me
| interfering in their honest work.
| throawayonthe wrote:
| that's the point of opinionated crypto libraries, yes
| otabdeveloper4 wrote:
| Those same security guys also think that "just hope that no
| bad guy ever gets root access, lol" is a valid threat model
| analysis, so whatever.
| Calvin02 wrote:
| Threats exist in both trusted and untrusted environments
| though.
|
| This feels like a really niche use case for SSH. Exposing this
| more broadly could lead to set-it-and-forget-it scenarios and
| ultimately make someone less secure.
| smallmancontrov wrote:
| Resource-constrained environments might be niche to you, but
| they are not niche to the world.
| pixl97 wrote:
| >very confidently told me that my tcpdump output was normal ssh
| behavior:
|
| I mean, for modern version of Openssh it's not exactly wrong. The
| failure was to tell you _why_ that is the normal behavior.
| raggi wrote:
| > I am working on a high-performance game that runs over ssh.
|
| WAT. Please no.
| shitter wrote:
| Why not? If it's high-performance, it's fine.
| pseidemann wrote:
| Performing with highly elevated privileges? (Joke)
| jabedude wrote:
| ssh the protocol doesn't imply any privileges of any kind
| zamadatix wrote:
| Very interesting, I hadn't heard of this obfuscation before so it
| was well worth clicking.
|
| Another good trick for debugging ssh's exact behavior is patching
| in "None" cipher support for your test environment. It's about
| the same work as trying to set up a proxy but lets you see the
| raw content of the packets like it was telnet.
|
| For terminal games where security does not matter but performance
| and scale does, just offering telnet in the first place can also
| be worth consideration.
| charcircuit wrote:
| It made the front page when it was added.
|
| https://news.ycombinator.com/item?id=37307708
| Veserv wrote:
| The really mysterious part is how ~10,000 packets per second
| costs ~20% of a core. That would mean SSH is bottlenecking in its
| code at ~50,000 packets per second per core which would be ~500
| Mbps per core (assuming full packets) which is ludicrously slow.
| It is trivial to do 10x that packet per second rate. Is SSH
| really that poorly designed?
| diath wrote:
| > It is trivial to do 10x that packet per second rate.
|
| When making this statement, are you taking into account that
| SSH encrypts the traffic by default?
| Veserv wrote:
| I do not know where people get the idea that encryption is
| that slow. Standard AES hardware acceleration instructions do
| ~25 Gbps per core (on a 2023 CPU) which is ~50x that rate
| [1]. I have heard modern cores can do ~40-50 Gbps, but I have
| not been able to find any independent benchmarks of that.
| Even the Intel i5-2500, a CPU from 2011, averages ~10 Gbps
| which is ~20x that rate. Even unaccelerated encryption can do
| ~2-5 Gbps in pure software which is 4-10x the SSH rate.
|
| And in this situation, the amount of encrypted payload in
| each packet is 36 bytes which is ~40x less than a full packet
| of ~1500 bytes. You would almost surely hit packet per second
| limits before you hit payload throughput limits at these
| small sizes.
|
| Encryption _is_ slow when compared to data throughput you can
| get with a properly designed transport stack, but that is
| because it is in comparison to 100 Gbps per core even with no
| hardware offload. Anything less than ~10 Gbps /1 million
| packets per second (ignoring other bottlenecks, so only the
| software transport is the limit) is not merely unoptimized,
| it is pessimized.
|
| [1] https://calomel.org/aesni_ssl_performance.html
| svnt wrote:
| > I am working on a high-performance game that runs over ssh.
|
| Found your problem.
|
| But it is an interesting world where you can casually burrow into
| a crypto library and disable important security features more
| easily than selecting the right network layer solution.
| ycombinatrix wrote:
| Yea UDP is technically more performant, but then you need a
| crypto layer + reliable message delivery layer + bespoke
| client. Using a plain old SSH client is cool.
|
| However, there are existing libraries for exactly this use case
| - see https://github.com/ValveSoftware/GameNetworkingSockets
|
| I guess QUIC libraries would also work.
| convolvatron wrote:
| its not really a question of 'udp performs better'. in tcp we
| have to live to head-of-line blocking on losses and
| congestion control. if you don't care about receiving every
| packet, but only the most recent, then udp is a good choice.
|
| running without congestion control means that you avoid
| slowstart. but at a certain rate you run into poorly defined
| 'fairness' issues where you can easily negatively impact
| other flows. past that point, you can actually self-interfere
| and cause excessive losses for yourself.
|
| quic uses congestion control, but uses latency estimates and
| variance as a signal to back off. it still imposes an
| ordering on a per-stream basis. so it might not be ideal
| either.
|
| sctp has a mode which supports reliable and unordered, which
| might be something to consider
|
| so really - if you care about latency and have a different
| reliability model, its worth unpacking all these
| considerations and using them to select your transport layer
| or even consider writing a minimal one yourself
| ycombinatrix wrote:
| >in tcp we have to live to head-of-line blocking on losses
| and congestion control.
|
| Is this not a performance consideration?
|
| Either way, using plain old SSH means a metric bajillion
| computers have a client for your game built in.
| eieio wrote:
| the obtuseness is the point! This is true of a lot of my
| work[1][2][3].
|
| The problems you run into when doing things you shouldn't do
| are often really fun.
|
| [1] https://news.ycombinator.com/item?id=42342382
|
| [2] https://news.ycombinator.com/item?id=37810144
|
| [3] https://news.ycombinator.com/item?id=42674116
| arwineap wrote:
| This is hackernews not consumer news
|
| You should feel free to explore / abuse all options :)
| properbrew wrote:
| These were great reads, thanks for linking. The writeup
| around the UUID page was super interesting!
| PaulHoule wrote:
| I find it disturbing.
|
| One thing you notice if you have ADSL is that some services are
| built as if slower connections matter and others are not. Like
| Google's voice and audio chat services work poorly but most of
| the others work well. Uploading images to Mastodon, Bluesky,
| Facebook, LinkedIn, Instagram and Nextdoor is reliable, but for
| Tumblr you have to try it twice. I don't what they are doing
| wrong but they are doing something wrong and not finding out what
| they're doing wrong because they're not testing and they're not
| listening to users.
|
| Nobody consulted me about their decision not to run fiber by my
| house. If some committee decides to make ssh bloated they are,
| together with the others, conspiring to steal my livelihood and I
| think it would be fair for me to sue them for the $50k it would
| take to run that fiber myself.
|
| It's OK if you work for Google where there is limitless dark
| fiber but what about people in African countries?
|
| It's the typical corporate attitude where latency never matters:
| Adobe thinks it is totally normal that it takes 1-5s for a
| keystroke to appear when you are typing into Dreamweaver.
| starttoaster wrote:
| There's a good chance you have other options. Regardless of how
| you feel about the company's head, Starlink would probably be
| one of them, with likely better performance than you're dealing
| with on ADSL.
|
| But you cannot just sue a company because their network
| connected software doesn't work well on slow networks. Let
| alone a project like OpenSSH. It would be like me suing a game
| studio because my PC doesn't meet their listed minimum
| requirements to play the game.
| PaulHoule wrote:
| Hey, it is one thing to buy a new computer, it is another
| thing to ask people to move.
|
| A better analogy is a bank redlining neighborhoods. The cost
| to run fiber to difficult rural locations pays itself easily
| if you look at a 25-year time span and is an order of
| magnitude less than building a new housing unit on the West
| Coast.
| Refreeze5224 wrote:
| You're not ok with a security/privacy tool using defensive
| techniques because of ... the lack of fiber in Africa?
| PaulHoule wrote:
| My backyard but people will take Africa more seriously than
| anywhere in the US 2 miles from the end of cable.
| gucci-on-fleek wrote:
| I agree with your general point that most companies/projects do
| a terrible job optimizing for slow computers/networks, but
| OpenSSH is from the OpenBSD people, who are well-known for
| supporting _ancient_ hardware [0]. Picking a random
| architecture, they fully support a system with only 64MB of
| memory [1], and the base install includes SSH. So I suspect
| that OpenSSH is fairly well tested on crappy computers
| /networks.
|
| [0]: https://www.openbsd.org/plat.html
|
| [1]: https://www.openbsd.org/landisk.html#hardware
| lokar wrote:
| The openssh team does not owe you anything.
|
| If you want a "1990s" mode, add it yourself or pay some to do
| it for you.
| bergen wrote:
| You just opened a huge nostalgia portal, never thought that
| Dreamweaver would still be around, I used that somewhere around
| 2003 I believe. Good memories
| PaulHoule wrote:
| Frankly I wish there was an HTML editor that delivers on what
| it promised. I mean, markdown is almost as rife with edge
| cases as YAML and somehow the link syntax still eludes me. If
| we could "just" template by merging at the DOM level and had
| decent HTML editors the world would be a different place. But
| yeah, Adobe probably thinks Dreamweaver isn't worth
| maintaining just as they seem to think Photoshop is barely
| worth maintaining (they keep adding AI features that sorta
| work but the foundations seem to be much worse than
| Illustrator)
| layer8 wrote:
| > One thing you notice if you have ADSL
|
| This is funny to me, because ADSL used to be the fast thing, as
| opposed to dialup modems.
| idontwantthis wrote:
| If security doesn't matter then why not use telnet or something
| else besides ssh instead of forking a security library?
| layer8 wrote:
| Telnet nowadays typically isn't available by default for
| security reasons, and OP wants people to be able to play the
| game just by typing "ssh thegamehost".
| AceJohnny2 wrote:
| > _Telnet nowadays typically isn't available by default for
| security reasons_
|
| And with good reason. This CVE is from _yesterday_ :
|
| https://nvd.nist.gov/vuln/detail/CVE-2026-24061
|
| > telnetd in GNU Inetutils through 2.7 allows remote
| authentication bypass via a "-f root" value for the USER
| environment variable.
| layer8 wrote:
| Telnetd is the server though, and OP wouldn't be using
| that.
| kenmacd wrote:
| @eieio: whatever email protection you're running is triggering on
| the extension info. For example I see:
|
| > And they're sent to servers that advertise the availability of
| the [email protected] extension. What if we just...don't
| advertise [email protected]?
| eieio wrote:
| Is it possible that this is on your end?
|
| The extension is "ping@openssh.com." It shows up in the blog
| reliably for me across several browsers and devices.
| wizzwizz4 wrote:
| No, it's Cloudflare munging the HTML. Cloudflare then
| provides JavaScript to un-munge it, but that's not reliable.
| eieio wrote:
| TIL! I'll see if I can change that.
| Animats wrote:
| _In 2023, ssh added keystroke timing obfuscation. The idea is
| that the speed at which you type different letters betrays some
| information about which letters you're typing. So ssh sends lots
| of "chaff" packets along with your keystrokes to make it hard for
| an attacker to determine when you're actually entering keys._
|
| Now that's solving the problem the wrong way. If you really want
| that, send all typed characters at 50ms intervals, to bound the
| timing resolution.
| adgjlsfhk1 wrote:
| Typing with an extra 50ms latency will be fairly unpleasant.
| braiamp wrote:
| Also considering ssh tunnels.
| Animats wrote:
| Average is 25ms. Just put sending on a clock.
| omoikane wrote:
| > send all typed characters at 50ms intervals
|
| Wouldn't this just change the packet interval from 20ms to
| 50ms? Or did you mean a constant stream of packets at 50ms
| intervals, nonstop?
|
| I think the idea behind the current implementation is that the
| keystrokes are batched in 20ms intervals, with the optimization
| that a sufficiently long silence stops the chaff stream, so the
| keystroke timing is obfucated with an increased error bar of
| 20ms multiplied by number of chaff packets.
| mystraline wrote:
| Whatever hair-brained suck-u-rity idiot thought this was
| great is an idiot.
|
| A trick Ive used to find hidden wireless cameras is to load
| up a wifi pineapple or similar, and log how many encrypted
| packets there are.
|
| Now, move around. If you see an encrypted spike of packets,
| guess what? You found a hidden wifi camera!
|
| Same thing with ssh. If you see 33 byte spam, guess what -
| nobody's typing. You see different or more than the norm?
| Someone's typing! GASP THE HORROR! TYPING!
|
| Now where's the real exploit here, with knowing someone's
| typing? Sounds like CVE inflation.
| frotaur wrote:
| The problem is not knowing whether someone is typing, as
| far as I understand. But that you may extract some
| information about what keys are being typed, based on the
| small differences in timings between them.
| davidhyde wrote:
| I wonder if this is the same reason why Microsoft's Remote SSH
| plugin on VS Code is so flaky even with a decent internet
| connection. Every couple of months I try to give it another go
| and give up due to the poor keyboard latency I inevitably
| experience. And the slow reconnects whenever I glance away from
| my computer monitor briefly. This is on a fiber connection with a
| 20ms ping to the remote machine.
| WesolyKubeczek wrote:
| You surely mean the latency in its embedded terminal and not
| the code editor, right? I use VSCode's remote SSH specifically
| so that code editing doesn't suck. It really does not.
| davidhyde wrote:
| You're right, the latency is in the embedded terminal.
| Perhaps it is trying to run SSH inside SSH. Still, the
| disconnects are a pain too.
| JohnLeitch wrote:
| The reliance on LLMs is unfortunate. I bet this mystery could
| gave been solved much quicker by simply looking at the packet
| capture in Wireshark. The Wireshark dissectors are quite mature,
| SSH is covered fairly well.
| MrDarcy wrote:
| How much are you staking on that bet?
| mystraline wrote:
| Sigh.
|
| I'm still waiting for a systems engineering tool that can log
| every layer, and handle SSL the whole pipe wide.
|
| Im covering everything from strafe and ltrace on the machine,
| file reads, IO profiling, bandwidth profiling. Like, the
| whole thing, from beginning to end.
|
| Theres no tool that does that.
|
| Hell, I can't even see good network traces within a single
| Linux app. The closest you'll find is
| https://github.com/mozillazg/ptcpdump
|
| But especially with Firefox, good luck.
| fragmede wrote:
| Real talk though, how much would such a tool be worth to
| you? Would you pay, say, $3,000/license/year for it? Or,
| after someone puts in the work to develop it, would you
| wait for someone else to duct tape something together
| approximately similar enough using regexps that open source
| but 10% as good, and then not pay for the good proprietary
| tool because we're all a bunch of cheap bastards?
|
| We have only ourselves to blame that there aren't better
| tools (publicly) available. If I hypothetically (really!)
| had such a tool, it would be an advantage over every other
| SRE out there that could use it. Trying to sell it directly
| comes with more headaches than money, selling it to
| corporations has different headaches, open-sourcing it
| don't pay the bills, nevermind the burnout (people don't
| donate for shit). So the way to do it is make a pitch deck,
| get VC funding so you're able to pay rent until it gets
| acquired by Oracle/RedHat/IBM (aka the greatest hits for
| Linux tool acquisition), or try and charge money for it
| when you run out of VC funding, leading to accusations of
| "rug pull" and development of alternatives (see also:
| docker) just to spite you.
|
| In the base case you sell Hashimoto and your bank account
| has two (three!) commas, but worst case you don't make rent
| and go homeless when instead you could've gone to a FAANG
| and made $250k/yr instead of getting paid $50k/yr as the
| founder and burning VC cash and eating ramen that you have
| to make yourself.
|
| I agree, that would be an awesome tool! Best case scenario,
| a company pays for that tool to be developed internally,
| the company goes under, it gets sold as an asset and
| whomever buys it forms a compnay and tries to sell it
| directly and then that company goes under but that whomever
| finally open sources it because they don't want it to slip
| into obscurity but if falls into obscurity anyway because
| it only works on Linux 5.x kernels and can't be ported to
| the 6.x series that we're on now easily.
| JohnLeitch wrote:
| Well, I spent a good part of my career reverse engineering
| network protocols for the purpose of developing exploits
| against closed source software, do I'm pretty sure I could do
| this quickly. Not that it matters unless you're going to pay
| me.
| pbar wrote:
| Unfortunately with SSH specifically, the dissectors aren't very
| mature - you only get valid parsing up to the KeX completion
| messages (NEWKEYS), and after that, even if the encryption is
| set to `none` via custom patches, the rest of the message flow
| is not parsed.
|
| Seems because dumping the session keys is not at all a common
| thing. It's just a matter of effort though - if someone put in
| the time to improve the SSH story for dissectors, most of the
| groundwork is there.
| JohnLeitch wrote:
| Interesting, I thought it was possible to decrypt SSH in
| Wireshark a la TLS, but it seems I'm mistaken. It still would
| have been my first goto, likely with encryption patched out
| as you stated. With well documented protocols, it's generally
| not too difficult deciphering the raw interior bits as needed
| with the orientation provided by the dissected pieces. So let
| me revise my statement: this probably would have been a
| fairly easy task with protocol analysis guided code review
| (or simply CR alone).
| turtlebits wrote:
| Way to gatekeep. God forbid people use tools to help them
| investigate instead of knowing the exact approach to take.
| kkkqkqkqkqlqlql wrote:
| My thoughts exactly. The OP used AI to get a starting point
| to their investigation, then used their skills to improve
| their game, with actual (I guess according to the article
| itself) proof of that, as opposed to just approving changes
| from the LLM.
|
| This looks like an actual productivity boost with AI.
| JohnLeitch wrote:
| What I suggested (mistakenly so, see my revised suggested
| approach in response to one of your siblings) is the exact
| opposite of gate keeping.
| danudey wrote:
| I'm anti-LLM in most cases, but:
|
| > I bet this mystery could gave been solved much quicker by
| simply looking at the packet capture in Wireshark.
|
| For some people who are used to using Wireshark and who know
| what to look for, probably yes. For the vast majority of even
| technical people, probably not.
|
| In my case, I did a packet capture of a single keystroke using
| tcpdump and imported it into Wireshark and I get just over 200
| 'Client: encrypted packet' and 'Server: encrypted packet'
| entries. Nothing useful there at all. If I tcpdump the entire
| SSH connection setup from scratch I get just as much useful
| information - nothing - but, oddly, fewer packets than my one
| keystroke triggered.
|
| So yeah, I dislike LLMs entirely and dislike the reliance on
| LLMs that we see today, but in this case the author learned a
| lot of interesting stuff and shared it with us, whereas without
| LLMs he might have just shrugged and moved on.
| mystraline wrote:
| And thats a huge downside when people howl about "Encryption
| everywhere! ".
|
| Try debugging that shit. Thats right, debugging interfaces
| aren't safe, by some wellakshually security goon.
|
| You want a real fun one to debug, is a SAML login to a
| webapp, with internal Oauth passthrough between multiple
| servers. Sure, I can decrypt client-server stuff with tools,
| but server-server is damn near impossible. The tools that
| work break SSL, and invalidate validation of the ssl.
|
| Yes, Esri products suck. Bad.
| tonymet wrote:
| obviously OPs empirical and analytical rigor are top notch. He
| applied LLMs in the best way possible: fill gaps with clumsy
| command line flags or protocol implementations. Those aren't
| things one needs to keep in their head all the time.
| fragmede wrote:
| Asking an LLM about SSH (hint: the two S-es stand for security)
| would tell you why only having packet capture in Wireshark
| isn't going to reveal shit.
| sureglymop wrote:
| Wireshark can decrypt it, so I don't understand what you
| mean?
| JohnLeitch wrote:
| Not even remotely accurate. While the dissector is not as
| mature as I thought and there's no built-in decryption as
| there is for TLS, that doesn't matter much. Hint: every
| component of the system is attacker controlled in this
| scenario.
| flumpcakes wrote:
| I don't see how Claude helped the debugging at all. It seemed
| like the author knew what to do and it was more telling Claude to
| think about that.
|
| I've used Claude a bit and it never speaks to me like that
| either, "Holy Cow!" etc. It sounds more annoying than interacting
| with real people. Perhaps AIs are good at sensing personalities
| from input text and doesn't act this way with my terse prompts..
| AceJohnny2 wrote:
| Even if the chatbot served only as a Rubber Ducky [1], that's
| already valuable.
|
| I've used Claude for debugging system behavior, and I kind of
| agree with the author. While Claude isn't always directly
| helpful (hallucinations remain, or at least outdated
| information), it helps me 1) spell out my understanding of the
| system (see [1]) and 2) help me keep momentum by supplying
| tasks.
|
| [1] https://en.wikipedia.org/wiki/Rubber_duck_debugging
| H8crilA wrote:
| AIs are exceptional at sensing personalities from text. Claude
| nailed it here, the author felt so good about the "holy cow"
| comments that he even included them in the blog post. I'm not
| just poking this, but saying that the bots are fantastic
| sycophants.
| fragmede wrote:
| > I am working on a high-performance game that runs over ssh.
|
| Step one, run https://www.psc.edu/hpn-ssh-home/introduction/
| instead Step two, tune TCP/IP stack Step... much later: write
| your own "crypto". (I'm using quotes because, before someone
| points out the obvious, packets-per-keystroke isn't, itself, a
| cryptographic algorithm, but because it's being done to protect
| connections from being decrypted/etc, mess with it at your own
| peril.)
| whiterook6 wrote:
| Tell me more about this game!
| markhahn wrote:
| fwiw, I tcdumped between two systems running fedora43 and saw no
| chaff. (one packet out, one reply, one tcp ack.)
___________________________________________________________________
(page generated 2026-01-22 23:00 UTC)