[HN Gopher] How is GNU `yes` so fast? (2017)
___________________________________________________________________
How is GNU `yes` so fast? (2017)
Author : zbentley
Score : 104 points
Date : 2022-06-04 11:20 UTC (11 hours ago)
(HTM) web link (old.reddit.com)
(TXT) w3m dump (old.reddit.com)
| asicsp wrote:
| Past discussion:
|
| "How is GNU `yes` so fast?"
| https://news.ycombinator.com/item?id=14542938 _(872 points | 5
| years ago | 334 comments)_
| ryukafalz wrote:
| For those wondering why they may have done this, there's a
| relevant section in the GNU Coding Standards:
|
| https://www.gnu.org/prep/standards/html_node/Reading-Non_002...
|
| This appears to be a case where they went for speed rather than
| simplicity.
| dc-programmer wrote:
| Is this for legal or product reasons or both? It's hard to tell
| from the link.
| ryukafalz wrote:
| Legal reasons; it's in a section titled "Legal Issues". The
| idea is if you're cloning an existing program, it needs to be
| clear that you're not copying code from the original. This
| may be less of an issue with the other free implementations
| we have now, but a lot of these were clones of proprietary
| UNIX programs, and then it's really important that you can
| tell it's not derived from the proprietary version.
| bauruine wrote:
| Thanks for the link. That was what I was thinking about when
| writing my response about "complex code to do trivial things is
| kind of a meme." but i couldn't find it.
| pixelbeat__ wrote:
| I did this because it runs significantly faster with relatively
| extra complexity. It's useful for quickly generating test data
| etc
| yakubin wrote:
| _> It looks like we can 't outdo C nor GNU in this case.
| Buffering is the secret, and all the overhead incurred by the
| kernel throttles our memory access, pipes, pv, and redirection is
| enough to negate 1.5 GiB/s._
|
| On my computer reading from _/ dev/zero_ (16.5GiB/s) is still
| significantly faster than piping _yes_ (7.11GiB /s), so I'd be
| interested in an exploration of why that is. BPF would probably
| be useful here. Maybe when I find a moment, I'll write a follow-
| up, doing just that... unless someone else does it faster. ;)
| DonHopkins wrote:
| I always thought the device minor number of /dev/zero should
| control which number you get, so you can also make useful
| devices like /dev/seven with minor number 7 for an infinite
| source of beeps, or /dev/newline with a minor number 10 for an
| infinite source of newlines, and /dev/rubout with minor number
| 127 for an infinite source of rubouts.
| jwilk wrote:
| I see a great need for /dev/yes.
| 3-cheese-sundae wrote:
| fmajid wrote:
| The throughput of GNU yes may be higher, but the simpler
| unbuffered implementations probably have a shorter latency to
| first line output, and since in most cases the program on the
| other end of the pipe only reads a handful of line, or more liely
| just one, this means they will actually be faster.
| bonzini wrote:
| Using puts, fputs or fwrite (which is what the other
| implementations do) also buffers, so all the GNU implementation
| does is special case the buffer to remove the overhead of
| stdio.
| rbanffy wrote:
| Is the performance gain worth the less legible
| implementation?
| jcelerier wrote:
| I have never read the source of yes but used it so for me
| it could be 20 megabytes of source for what I know - as
| long as it is efficient why would I care ?
| MichaelBurge wrote:
| I use "yes > file" to benchmark IO, knowing that this is
| pretty much the "speed of light" for writes.
|
| I could use "cp /dev/urandom file" but then it's hard to
| tell if the filesystem or kernel optimize a copy different
| from generating your own writes.
| wodenokoto wrote:
| Tangibly related, but yes is one of my favorite apps. I use it
| quite often as it's easier to just spam "yes" than dig through
| the man page to figure out how to make the command not ask
| question.
|
| I always feel like it is the wrong way of doing it, but at the
| same time I just love the idea of a core utility that does
| nothing but spam yes.
| krallja wrote:
| I use it to be REALLY sure I've cleared my scrollback buffer.
| gigatexal wrote:
| GNU yes is not doing anything crazy imo why are the BSDs not
| using GNU yes if it's this much faster?
| yung_steezy wrote:
| BSD devs tend to prioritise clean code over speed. I'm not
| certain of the implementation details here but their approach
| does make openbsd coreutils a more pleasant read than the GNU
| coreutils.
| trelane wrote:
| Do the various BSD equivalents have to also work on other,
| esoteric OSen, or do they generally tend to fork a new
| version only to run on just on that flavour of BSD?
| pxc wrote:
| Idk about the stuff that's normally 'included' in the
| distribution (base install of the OS), but NetBSD's package
| manager supports a ton of platforms. I've used it on Linux
| and macOS.
| bauruine wrote:
| Not exactly an answer but OpenBSD sometimes has different
| portable versions of their software. e.g.
| https://www.openssh.com/portable.html
| rahen wrote:
| Also clean code usually results in compact code, which used
| to be effectively the same as fast code for the machines BSD
| was initially developed for.
|
| I wouldn't be surprised if the BSD coreutils were more
| performant on older machines with small memory, buffers and
| prefetch queues (sometimes just a few bytes) than their GNU
| counterpart.
| masklinn wrote:
| I would be very surprised. An 8k buffer is hardly a large
| imposition, and less syscalls has always been hugely
| beneficial.
| masklinn wrote:
| Because BSDs don't use the GNU userland?
|
| As to why they don't use similar techniques, probably because
| they historically didn't really care enough about the
| performances of yes(1) to bother with it.
|
| FreeBSD has since added output buffering to yes(1), you can see
| the difference between openbsd yes(1)[0] which remains utterly
| naive and freebsd yes(1)[1] which uses an 8k internal buffer.
|
| [0]:
| https://github.com/openbsd/src/blob/master/usr.bin/yes/yes.c
|
| [1]:
| https://svnweb.freebsd.org/base/head/usr.bin/yes/yes.c?view=...
| mtlmtlmtlmtl wrote:
| Amusingly, the OpenBSD one remains completely naive, but
| still uses pledge for privilege dropping.
| midislack wrote:
| It's not naive, it's _correct._ GNU 's focus is on
| features, kitchen sink style, performance tuning, etc. If
| that's what you want, it's for you. OpenBSD is focused on
| correctness. It's not correct to use hacks and buffering in
| tools to increase performance. That should be handled at
| the syscall interface or in the kernel or whatever.
| dizzant wrote:
| Can you expand on why you think performance hacking
| degrades correctness? As I understand it, "yes" (with no
| arguments) is supposed to output "y\n" continuously until
| terminated. Throughout/latency are unspecified, admitting
| various implementations with various concerns, like GNU's
| focus on throughout vs BSD's focus on latency/simplicity.
| Even an implementation that took 10 seconds between each
| output could be desirable for some use cases! Is that
| version of "yes" less correct?
| GuB-42 wrote:
| I didn't try it but I think it would be relatively
| straightforward to compile GNU tools on BSD.
|
| But generally GNU tends to focus on performance and features
| while BSD focuses on simplicity. GNU having ridiculously
| complex code to do trivial things is kind of a meme.
|
| And also, most people don't need a ridiculously fast "yes".
| Usually, when you want a fast stream of bytes, for example to
| fill up some space, /dev/zero is a better option.
| bauruine wrote:
| >GNU having ridiculously complex code to do trivial things is
| kind of a meme.
|
| One of the reasons is that GNU had to reimlmement UNIX tools
| as GPL code. How do you reimplement something trivial without
| it looking as the original to avoid copyright claims? One of
| the solutions is to implement it as complicated as possible.
| rahen wrote:
| For the trivia, GNU "true" is an abomination which even
| includes a cornercase that returns "false".
|
| Meanwhile, its BSD version essentially sums up to "return 1".
|
| Same for much of the GNU code:
|
| - echo: https://gist.github.com/fogus/1094067
|
| - cat: http://9front.org/img/longcat.png
| kzrdude wrote:
| When does the speed of yes matter?
| DonHopkins wrote:
| There's a whole best-selling book about it:
|
| https://en.wikipedia.org/wiki/Getting_to_Yes
| gmfawcett wrote:
| Well played. :) Interestingly, if I listed their
| negotiation principles here in a blind test, you might
| think they were about software development:
| - separate the people from the problem - focus on
| interests, not positions - invent options for
| mutual gain - insist on using objective criteria
|
| I'd hire that engineer!
| chongli wrote:
| It doesn't, 99.9999% of the time. But there's probably
| someone, somewhere, who uses it in some crazy huge obscure
| script, who appreciates the work people put into optimizing
| it.
| lupire wrote:
| There is a program in the real world that consumes a line
| of input and processes it, faster than a naive yes can emit
| it?
| DonHopkins wrote:
| yes | not
| Brian_K_White wrote:
| ding ding ding
|
| But then again, maybe the purpose is to swamp the input,
| maybe for testing, so the more the better, so, you never
| know and it's wrong to decide what all users don't need.
| fnordpiglet wrote:
| When you're really impatient with prompts.
| liftm wrote:
| I've misused it as a load generator before: yes
| '{"some":"json"}' | kafkacat -P -t bleh (Though in my case,
| ~100 MB/s were enough.)
| bauruine wrote:
| Somehow related. FizzBuzz at 40+ GiB/s
| https://codegolf.stackexchange.com/a/236630
| vlovich123 wrote:
| Someone fixed the bottleneck in pv to hit 123 GB/s [1].
|
| Someone else managed to double yes performance [2] via vmsplice.
|
| [1]
| https://www.reddit.com/r/unix/comments/6gxduc/how_is_gnu_yes...
|
| [2]
| https://www.reddit.com/r/unix/comments/6gxduc/how_is_gnu_yes...
| mrb wrote:
| Removing a bottleneck in pv to get it to 123 GB/s?! I thought:
| cool! This kind of work is right up my alley! I'm curious,
| let's see how this person did it. I read that reddit post and I
| see the guy is mrb_... My own work, from 4 years ago. I
| completely did not remember :)
| junon wrote:
| This made me chuckle. I've done this a handful of times in
| the form of SO answers. Always a fun feeling to discover
| you've helped yourself somehow.
| gilleain wrote:
| Hah. That's so much better than the opposite - 'who wrote
| this terrible, buggy, badly formatted code??!' then finding
| it was you several years ago.
| fnordpiglet wrote:
| I'm a little surprised there's no references to the gnu coreutil
| yes source. I love speculation as much as the next guy but a lot
| of it could be satisfied by reading the source code. It's
| fascinating that the BSD code is linked but not the primary
| subject!
|
| Source for yes which does all the buffer malarkey:
| https://github.com/coreutils/coreutils/blob/master/src/yes.c
|
| Source for the actual output:
| https://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=blob_pla...
| gjm11 wrote:
| Maybe the OP was edited in response to this, but right now it
| _does_ link to exactly the same thing as you do. Search for
| "Let's see how yes does it".
| alpaca128 wrote:
| Can 4 year old posts still be edited?
| jwilk wrote:
| No idea, but the link was there back in 2020:
|
| https://web.archive.org/web/20200122231049/https://old.redd
| i...
___________________________________________________________________
(page generated 2022-06-04 23:01 UTC)