[HN Gopher] Make Ubuntu packages 90% faster by rebuilding them
___________________________________________________________________
Make Ubuntu packages 90% faster by rebuilding them
Author : jeffbee
Score : 525 points
Date : 2025-03-18 23:55 UTC (23 hours ago)
(HTM) web link (gist.github.com)
(TXT) w3m dump (gist.github.com)
| yjftsjthsd-h wrote:
| > What happens if you grab the jq source code from Launchpad,
| then configure and rebuild it with no flags at all? Even that is
| about 2-4% faster than the Ubuntu binary package.
|
| Why wouldn't that be identical?
| jeffbee wrote:
| Because instead of using the Debian build, I just used
| `./configure; make`
| Aachen wrote:
| How does that explain it?
| zetalyrae wrote:
| -march=native maybe?
| cjbprime wrote:
| Could be measurement noise..
| motoboi wrote:
| I tough about that. The output of hiperfine mention nothing
| chi-test or anything about testing distributions...
| taeric wrote:
| Surprised it would be 90%. Are the packages all debug symbols?
|
| I recall trying with march=native and seeing some improvement,
| but not enough to care at a system level.
| jeffbee wrote:
| > Are the packages all debug symbols?
|
| Upstream is stripped actually. My final build here is 43x
| larger than the distro binary, partly due to having plentiful
| debug info.
| taeric wrote:
| Sorry, I didn't mean symbols, I meant optimization level.
| Hilarious that they have symbols stripped, but still
| outclassed.
|
| Love to see why they wouldn't have more optimizations.
|
| I'm assuming this isn't the case for more packages? Feels too
| good.
| dan-robertson wrote:
| Debug symbols do not need to be paged in so shouldn't make
| much/any difference. Compiling with -O3 can increase code
| size a lot due to inlining, which can be a little bad.
| pests wrote:
| Mario 64 gets flak for some files not being compiled with
| optimizations on. There is a YouTuber, whose name I
| forget at the moment, who has been doing optimization and
| improvements on it and constantly talks about how this is
| for the better due to the low instruction cache size and
| slow speed on the console. Compiling with optimizations
| explodes code size (unrolling links, inlining, etc) to
| the point it's a net loss after factoring in the
| increased load on the IC.
| chasil wrote:
| The -Os flag optimizes for size.
|
| What impact does that have?
| rurban wrote:
| The IC is the instruction cache. Bloated code needs more
| slow loads from the IC, and even if the performance is
| faster per cache line, the 50x overhead of added cache
| loads diminishes all -O3 advantages.
| tjalfi wrote:
| Kaze Emanuar[0] is the YouTuber you're thinking of.
|
| [0] https://www.youtube.com/@KazeN64
| taeric wrote:
| Right, my humor there was that I'm assuming people
| specifically did non-debug flags for the build as an
| optimization. Only for them to still fall short.
|
| And understood a little on -O3 possibly increasing code
| size. I had thought that was more of a concern for tight
| environments than for most systems? Of course, I'd have
| assumed that -march=native would be more impactful, but
| the post indicates otherwise.
|
| I said in a top level, but it seems the allocator makes
| the biggest impact for this application? Would be
| interesting to see which applications should use
| different allocators. Would be amazing to see a system
| where the more likely optimal allocator was default for
| different applications, based on their typical allocation
| patterns.
| turol wrote:
| > Right, my humor there was that I'm assuming people
| specifically did non-debug flags for the build as an
| optimization.
|
| It used to be the case that the presence of debug symbols
| would affect GCC code generation. Nowadays that should be
| fixed. I think it still affects the speed of compilation
| so if you're building the whole system from source you
| might want to avoid it.
| SkiFire13 wrote:
| > And understood a little on -O3 possibly increasing code
| size. I had thought that was more of a concern for tight
| environments than for most systems?
|
| Increasing code size too much can result in hot functions
| not fitting in the icache, and that ultimately can make
| your program slower.
| AceJohnny2 wrote:
| > _Hilarious that they have symbols stripped, but still
| outclassed._
|
| Debug symbols have 0 runtime penalty, just storage. They're
| just another section of the binary, referenced by
| debuggers, and which the loader skips.
|
| In any case, all distros break out the symbols into
| separate files so that they can have their (storage) cake
| and (debug) eat it too.
| taeric wrote:
| Right, I meant the humor to be that folks to not use
| debug flags, with my naive presumption that they did this
| as an optimization, only for that not to really help.
| wongarsu wrote:
| Ubuntu is still building for CPUs with the x86-64-v1
| feature set. On experimental images built for x86-64-v3
| (roughly the feature set of 10 year old CPUs, a bit newer
| on low-power CPUs) they have observed 60% improvement on
| some benchmarks (and much less in some others). Some
| distros have switched to -v3, Ubuntu is still holding out
| to support older hardware. The author is compiling to the
| actual feature set of their specific CPU, which is nearly
| guaranteed to be even more than -v3.
|
| Another thing the article adds is LTO, which in my
| experience also makes a huge difference: it makes your
| software a lot faster in certain cases, but also makes
| build times a lot worse. Spending a bit more time in the
| build process should be an easy call, but might be harder
| at the scale of a distro.
| taeric wrote:
| I was under the, evidently naive, understanding that
| there were some shenanigans done in the libc to pick
| optimal code paths based on the architecture. This would
| obviously have a startup cost on the first run, but
| presumably the dynamic linking would work to this
| advantage by having it in a fast state for most calls?
|
| Not sure where I got that idea, though. :(. Will have to
| look into that later.
| vlovich123 wrote:
| Even if libc does it, other packages wouldn't. The
| compiler is what generates code and they generally don't
| try to automatically generate for different instruction
| sets because they don't know what code is what hotpath
| and worth optimizing for. You could probably try to build
| for multiple different CPU features at the top level and
| do a universal executable like Apple did for their CPU
| transitions but that's a lot of expense to pay both in
| terms of compilation time AND in terms of size.
| kazinator wrote:
| Using preload on the stock Ubuntu binary, to give it mimalloc,
| they got a 44% speedup.
|
| By rebuilding the binary with different compiler options, but
| not changing malloc, they got an 20% speedup.
|
| If we naively multiply these speedups, we get 1.78: 78% faster.
|
| How it goes to 1.9 is that when you speed up only the program,
| but leave malloc the same, malloc matters to its performance a
| lot more.
|
| When the faster malloc is applied to a program that is compiled
| better, it will make a better contribution than the 44% seen
| when the allocator was preloaded into the slower program.
|
| To do the math right, we would have to look into how much time
| was saved with just the one change, and how much time was saved
| with the other. If we add those times, and subtract them from
| the original, slow time, we should get a time that is close to
| the 1.9 speedup.
|
| Original time: 4.631
|
| Better compiler options alone: 3.853 (-0.778)
|
| Better allocator alone (preload): 3.209 (-1.422)
|
| Add time saved from both: 2.200
|
| Projected time: 4.631 - 2.200 = 2.431
|
| Projected speedup from both: 4.631/2.431 = 1.905
|
| Bang on!
| kazinator wrote:
| This is the same reason why a faster garbage collector makes
| a huge difference to compiled code compared to interpreted.
|
| When an AST interpreted languages gets a VM, gets native
| code, each step reveals GC to be slow.
|
| You might go from 1% time spent in the GC to 15% to 60%
| (numbers of out thin air).
| biohazard2 wrote:
| It's not a 90% speedup, it's ~50% (still quite impressive). The
| author seems to be confused, because the original jq is 1.9x
| slower than the optimized one.
| drakythe wrote:
| That depends on how you're representing the speedup.
|
| To travel 10 miles, at 60 MPH, takes 10 minutes. Make it 100%
| faster, at 120 MPH, and that time becomes 5 minutes. Travel
| just as far in 50% of the time. Or travel just as far 100%
| faster. The 90% speedup matches the reduction of the time it
| takes to nearly half (a 90% (projected) speedup, or about a
| 45% time reduction, as mathed out by kazinator `Projected
| speedup from both: 4.631/2.431 = 1.905`). Your claim that its
| closer to 50% is correct from a total time taken perspective,
| just coming at it from the other direction.
| kstrauser wrote:
| (Well, rebuilding them with a different allocator that benchmarks
| well on their specific workflow.)
| dan-robertson wrote:
| Is it even known what workloads the glibc malloc is good for?
| searealist wrote:
| Using all your memory on multi-threaded workflows.
| rurban wrote:
| It's using the least additional memory, compared to all
| others. Whilst being the slowest by far of all modern
| allocators.
| loeg wrote:
| Everything outperforms glibc malloc. It's essentially
| malpractice that distros continue to use it instead of mimalloc
| or jemalloc.
| throw16180339 wrote:
| It's been awhile since I looked into this, but it's not
| necessarily an easy change. glibc malloc has debugging APIs;
| a distro can't easily replace it without either emulating the
| API or patching programs that use it.
| loeg wrote:
| No need to even patch it out. It's relatively easy to
| change the default, rebuild the world (distros have a flow
| for this), and restore glibc for the tiny, tiny handful of
| individual programs that actually rely on glibc debugging
| APIs.
|
| From a end developer perspective: I have no particular
| familiarity with mimalloc, but I know jemalloc has pretty
| extensive debugging functionality (not API compatible with
| glibc malloc of course).
| code-faster wrote:
| Could you release a fork of jq whose make does all this
| fanciness? I use jq A LOT so would love a turnkey solution to
| make it faster.
| jeffbee wrote:
| If I had the time to fork jq, I would convert the relevant part
| to C++ so it doesn't spend literally all of its time
| dynamically allocating strings it is just about to discard.
| jeffparsons wrote:
| Why does that require C++? A quick search showed up plenty of
| material about "pure C" arena allocators, etc.
| internetter wrote:
| https://github.com/01mf02/jaq?tab=readme-ov-file#performance
|
| jaq runs 5x faster on my machine in some cases
| jeffbee wrote:
| jaq is nice. It just loses, performance-wise, to the final
| step in this article, and it can't do the second mentioned
| workload (yet) so I didn't include it.
| electromech wrote:
| Which workload can't it do? I've had good success with jaq
| performance.
| jeffbee wrote:
| It bombs out on the jq program I use for the 2nd corpus
| that I mentioned. On further investigation, the show-
| stopping filter is strftime. In the jaq readme this is
| the only not-yet-checked box in the compatibility list,
| so perhaps some day soon.
| tehnub wrote:
| Not a fork but a similar program I've been using:
| https://github.com/tidwall/jj?tab=readme-ov-file#performance
| Thaxll wrote:
| Sounds like Gentoo...
| Maxious wrote:
| --funroll-loops for lyfe
| leoh wrote:
| Personally I like to use -O8, it's just even faster that way
| 01HNNWZ0MV43FF wrote:
| --vtech just kicked in
| JoshTriplett wrote:
| -Omg -Wtf -Bbq
| genpfault wrote:
| --funsafe-math-optimizations
|
| Enables fun & safe math optimizations!
| taeric wrote:
| Looks like the main gem here is mimalloc? Love to know more about
| that. Good places to look?
| gz09 wrote:
| https://github.com/microsoft/mimalloc
|
| we're using it in feldera and it's been giving us (slightly)
| better performance than jemalloc which is what we've used
| before mimalloc
| taeric wrote:
| Apologies, I did my first post on my phone, so was rather
| quick. I had found the repository and it is a good read. What
| I'd love to see is a comprehensive dive on the different
| options and a good general discussion on when each is
| superior.
|
| I realize, of course, that this could just be overall
| optimal. Feels more likely that the allocation patterns of
| the application using it will impact which is the better
| allocator?
| marcodiego wrote:
| Or, if you want to do it pain free, install guix on it.
| otsukare wrote:
| How would you do the things mentioned in the article using
| Guix?
|
| You could write your own custom package definitions, extending
| the default to change up compile flags and allocators, but then
| you need to do this for every single package (and maintain them
| all). I'm not sure Guix gives you much here, though maybe
| that's fine for one or two packages.
|
| The most pain-free option I can think of is the --tune flag
| (which is similar to applying -march=native), but packages have
| to be defined as tunable for it to work (and not many are).
|
| Is there another option?
| ergonaught wrote:
| O3 is a bad idea if the package is important. While that's a
| dated opinion based on older experiences (Gentoo, various server
| software, etc), I suspect it's still the easiest way to get
| exposed to compiler bugs.
| motoboi wrote:
| Is a bad ideia if you shove that into people expecting
| stability. But for self compiling is ok because you'll have a
| clear culprit in case of failure.
| jeffbee wrote:
| If you're going to offer unsupported FUD, I would prefer you
| direct the FUD at NDEBUG and the fear that it could have
| disabled a load-bearing assertion.
| wombatpm wrote:
| Load bearing assertions.
|
| Even Design by Contract in Eiffel turns off assertions in
| production.
| groby_b wrote:
| Just because it's turned off by default doesn't mean it
| isn't load bearing. If you didn't ever encounter an ASSERT
| with side effects that were crucial to proper functioning,
| you just haven't seen enough ASSERT statements.
|
| Rule #1 of programming: If it can go wrong, it will.
| eru wrote:
| Also, you might just run into a new bug, and the assert
| might caught it, and at least told you about the bug.
| fc417fc802 wrote:
| That's before we even consider instances where assert was
| used to check a condition that should always be verified.
| In my personal projects I define a "require" macro that I
| use liberally.
| ergonaught wrote:
| If acknowledging my opinion was dated based on older
| experiences constitutes "unsupported FUD", well, have fun
| with the life experiences that attitude produces for you.
| ac130kz wrote:
| First and foremost, an easy way to grab a bunch of undefined
| behavior.
| eru wrote:
| That's a bit sloppy.
|
| The undefined behaviour is already in the C standard.
|
| But you are right that enabling -O3 is generally going to
| make your compiler more aggressive about exploiting the
| undefined behaviour in the pursuit of speed.
|
| In general, if you want to avoid being the guy to run into
| new and exciting bugs, you want to run with options that are
| as close as possible to what everyone else is using.
| atonse wrote:
| I'm almost more amazed that someone figured out jq's syntax and
| got some use out of it.
|
| In all seriousness though, are you sure some of this isn't those
| blocks being loaded into some kind of file system cache the
| second and third times?
|
| How about if you rebooted and then ran the mimalloc version?
| matt123456789 wrote:
| The benchmarking tool being used in this post accounts for that
| using multiple runs for each invocation, together with a warmup
| run that is not included in the result metric.
| atonse wrote:
| Ah missed that, sorry.
| behnamoh wrote:
| jq has point-free programming. it's intuitive once you wrap
| your head around it.
|
| See this: https://en.wikipedia.org/wiki/Tacit_programming#jq
| mmastrac wrote:
| I wish I had seen this earlier. My mental model was close but
| not quite there to the point where I needed to think too hard
| about how to solve problems.
|
| This is much more intuitive now.
| atonse wrote:
| Yeah after this post I sought a couple youtube videos
| explaining it. It's starting to make a bit more sense now.
| But the lightbulb hasn't gone off just yet. Appreciate the
| link.
| Flimm wrote:
| Thank you! That's helpful. I got the examples in that
| Wikipedia section to run using the `--null-input` flag:
| $ jq --null-input '[1, 2] | add' 3
| penguin_booze wrote:
| I think jq's syntax is pretty sweet, once you get used to it
| (or are already familiar with point-free style, as it's in
| Haskell). The man page is subpar, however.
| ltbarcly3 wrote:
| Misleading title, it's 90% of the faster time. It's about 45%
| faster.
|
| It's actually a little bit interesting, if you are interested in
| how we use language. You could argue that now you now get 90%
| more work done in the same amount of time, and that would align
| with other 'speed' units that we commonly use (miles per hour,
| words per minute, bits per second). However, the convention in
| computer performance is to measure time for a fixed amount of
| work. I would guess that this is because generally we have a
| fixed amount of work and what might vary is how long we wait for
| it (and that is absolutely true in the case of this blog post) so
| we put time in the numerator.
|
| It's a very interesting post and very well done, but it's not 90%
| faster.
| odo1242 wrote:
| I feel like using units of rate (90% faster) and not units of
| time makes more sense here.
|
| Plus, if you were using units of time, you wouldn't use the
| word "faster." "Takes 45% less time" and "45% faster" are very
| different assertions, but they both have meaning, both in
| programming and outside it.
| ltbarcly3 wrote:
| It comes down to convention. When talking about proportional
| differences, we can fix the unitary example to be the
| smaller, larger, earlier, or later, object or subject.
|
| I think, generally, we fix on the earlier when talking about
| the change over time of a characteristic. "This stock went up
| 100%, this stock went down 50%". In both cases it's the
| earlier measurement that is taken as the unit. That makes
| this a 45% reduction in time to do the work, and that's
| actually what they measured.
|
| When talking about comparisons between two things that aren't
| time dependent it depends on if we talk in multiples or
| percents I think. A twin bed is half as big as a king bed. A
| king bed is twice as big as a twin bed. Both are idiomatic. A
| king bed is 100% bigger than a twin bed. Yes, you could talk
| like this. A twin bed is 100% smaller than a king bed. Right
| away you say wait, a twin bed isn't 0 size! Because we don't
| talk in terms of the smaller thing when talking about
| decreasing percents, only increasing. A twin bed is 50%
| smaller than a king bed (iffy). A twin bed is 50% as big as a
| king bed. There, that's idiomatic again.
| xmprt wrote:
| I think you're right. I think you could say the package (as in
| the code) is 45% faster or that the package increases parsing
| rate by 90%. But mixing the two is confusing.
| Daneel_ wrote:
| I think it's an interpretation issue. When you say it's 45%
| faster I interpret that as "the new package handles work at a
| rate of 145% when compared to the original, that is, 1.45
| times as fast".
|
| I would rephrase your comment as "the package takes 45% less
| time to process a given data set".
| msm_ wrote:
| Thanks for this, as an average HN user I didn't click the link
| and just skimmed the comments thinking how is it possible that
| they reduced the runtime to 10% of the original. This post
| clarifies that for me (now on to actually read the blog post).
| alpaca128 wrote:
| The title doesn't imply that at all though. 100% faster means
| doubled speed. 10% runtime means 1000% faster.
| ltbarcly3 wrote:
| I think it's not idiomatic to talk that way, especially
| because it's ambiguous.
| tom_ wrote:
| Related: https://randomascii.wordpress.com/2018/02/04/what-we-
| talk-ab...
| Daneel_ wrote:
| Great read. I hadn't even considered that people might
| interpret "90% faster" as "10 times as fast", i.e., it will
| take 100-90=10% of the original time. It seems like a
| completely incorrect interpretation to me, but obviously
| there are people who read it with this understanding. Huh.
| narmiouh wrote:
| It is unfortunate (because how do you interpret 100%
| faster) but a common interpretation of X% faster implies it
| takes x% less time than before. One easy way is to have
| chatgpt give a numeric example for a statement like this,
| and in all cases I tried it gives an example of sorts if
| the job took 100 seconds before it takes 10 seconds now (a
| 10x increase), and I'm assuming it is a representation of
| common usage based on how much data is used to train it.
| gblargg wrote:
| More misleading is that it implies that all packages can be
| made 90% faster. This is one particular package.
| ltbarcly3 wrote:
| After thinking about it more, I think I know why it seems
| misleading. They are talking about the change in the larger
| value as a % of the smaller value. That's what is misleading.
| They are saying it is reduced by 90% (of some other thing).
| When you say "We reduced THING by N%" it's just assumed that
| the N% is N% of THING, not OTHER THING.
| emmelaich wrote:
| I think parsing once into a faster format (sqlite3 or parquet)
| would be more beneficial.
|
| https://simdjson.org/
| llimllib wrote:
| For kicks, I downloaded the file and compared duckdb with the
| spatial extension against jq. jq was about 2x as fast for
| interactive use, but if you had multiple queries to run, paying
| a small cost to create a duckdb database and then querying it
| would be vastly faster
| datacynic wrote:
| I also was nerdsniped into trying this and found that after
| extracting the features array into a newline delimited json
| file, DuckDB finishes the example query in 500 ms (M1 Mac),
| querying the 1.3 GB json file directly with read_json!
| wiredfool wrote:
| Yeah, any of GeoPackage (sqllite db with spatial extensions),
| FlatGeoBuf, or GeoArrow/GeoParquet would be far better for
| querying.
| rlpb wrote:
| Note that if you do this then you will opt out of any security
| updates not just for jq but also for its regular expression
| parsing dependency onigurama. For example, there was a security
| update for onigurama previously; if this sort of thing happens
| again, you'd be vulnerable, and jq is often used to parse
| untrusted JSON.
|
| > * SECURITY UPDATE: Fix multiple invalid pointer dereference,
| out-of-bounds write memory corruption and stack buffer overflow.
|
| (that one was for CVE-2017-9224, CVE-2017-9226, CVE-2017-9227,
| CVE-2017-9228 and CVE-2017-9229)
| topspin wrote:
| This is certainly true. Also, by replacing the allocator and
| changing compiler flags, you're possibly immunizing yourself
| from attacks that rely on some specific memory layout.
| internetter wrote:
| That is, if you are a believer in security via obscurity
| frontfor wrote:
| Are you arguing that ASLR is "security via obscurity"?
| topspin wrote:
| I would, and there is no shame in it, as far as I'm
| concerned.
|
| I don't need to outrun the bear. I just need to outrun
| you.
| msm_ wrote:
| ASLR is not security through obscurity though. It forces
| attacker to get a pointer leak before doing almost
| anything (even arbitrary read and arbitrary write
| primitives are useless without a leak with ASLR). As
| someone with a bit of experience in exploit dev, it makes
| a world of a difference and is one of the most
| influential hardenings, next to maybe stack cookies and
| W^X.
| ryao wrote:
| ASLR obscures the memory layout. That is security by
| obscurity by definition. People thought this was okay if
| the entropy was high enough, but then the ASLR[?]Cache
| attack was published and now its usefulness is
| questionable.
| usefulcat wrote:
| I'm genuinely curious what was so undesirable about this
| sibling comment that it was removed:
|
| "ASLR obscures the memory layout. That is security by
| obscurity by definition. People thought this was okay if
| the entropy was high enough, but then the ASLR[?]Cache
| attack was published and now its usefulness is
| questionable."
|
| Usually when a comment is removed, it's pretty obvious
| why, but in this case I'm really not seeing it at all. I
| read up (briefly) on the mentioned attack and can confirm
| that the claims made in the above comment are at the very
| least plausible sounding. I checked other comments from
| that user and don't see any other recent ones that were
| removed, so it doesn't seem to be a user-specific thing.
|
| I realize this is completely off-topic, but I'd really
| like to understand why it was removed. Perhaps it was
| removed by mistake?
| minitech wrote:
| I assume people downvoted it because "ASLR obscures the
| memory layout. That is security by obscurity by
| definition" is just wrong (correct description here:
| https://news.ycombinator.com/item?id=43408039). It does
| say [flagged] too, though, so maybe that's not the whole
| story...?
| bigstrat2003 wrote:
| No, that other definition is the incorrect one. Security
| by obscurity does not require that the attacker is
| ignorant of the fact you're using it. Say I have an IPv6
| network with no firewall, simply relying on the
| difficulty of scanning the address space. I think that
| people would agree that I'm using security by obscurity,
| even if the attacker somehow found out I was doing this.
| The correct definition is simply "using obscurity as a
| security defense mechanism", nothing more.
| minitech wrote:
| No, I would not agree that you would be using security by
| obscurity in that example. Not all security that happens
| to be weak or fragile and involves secret information
| somewhere is security by obscurity - it's specifically
| the _security measure_ that has to be secret. Of course,
| there's not a hard line dividing secret information
| between categories like "key material" and "security
| measure", but I would consider ASLR closer to the former
| side than the latter and it's certainly not security by
| obscurity "by definition" (aside: the rampant misuse of
| that phrase is my pet peeve).
|
| > The correct definition is simply "using obscurity as a
| security defense mechanism", nothing more.
|
| This is just restating the term in more words without
| defining the core concept in context ("obscurity").
| fc417fc802 wrote:
| I'm inclined to agree and would like to point out that if
| you take a hardline stance that any reliance on the
| attacker not knowing something makes it security by
| obscurity then things like keys become security by
| obscurity. That's obviously not a useful end result so
| that can't be the correct definition.
|
| It's useful to ask what the point being conveyed by the
| phrase is. Typically (at least as I've encountered it)
| it's that you are relying on secrecy of your internal
| processes. The implication is usually that your processes
| are not actually secure - that as soon as an attacker
| learns how you do things the house of cards will
| immediately collapse.
| ryao wrote:
| What is missing from these two representations is the
| ability for something to become trivially bypassable once
| you know the trick to it. AnC is roughly that for ASLR.
| vlovich123 wrote:
| ASLR is not purely security through obscurity because it
| is based on a solid security principle: increasing the
| difficulty of an attack by introducing randomness. It
| doesn't solely rely on the secrecy of the implementation
| but rather the unpredictability of memory addresses.
|
| Think of it this way - if I guess the ASLR address once,
| a restart of the process renders that knowledge
| irrelevant implicitly. If I get your IPv6 address once,
| you're going to have to redo your network topology to
| rotate your secret IP. That's the distinction from ASLR.
| fc417fc802 wrote:
| I don't like that example because the damaged cause by
| and the difficulty of recovering from a secret leaking is
| not what determines the classification. There exist keys
| that if leaked would be very time consuming to recover
| from. That doesn't make them security by obscurity.
|
| I think the key feature of the IPv6 address example is
| that you need to expose the address in order to
| communicate. The entire security model relies on the
| attacker not having observed legitimate communications.
| As soon as an attacker witnesses your system operating as
| intended the entire thing falls apart.
|
| Another way to phrase it is that the security depends on
| the secrecy of the implementation, as opposed to the
| secrecy of one or more inputs.
| vlovich123 wrote:
| You don't necessarily need to expose the IPv6 address to
| untrusted parties though in which case it is indeed quite
| similar to ASLR in that data leakage of some kind is
| necessary. I think the main distinguishing factor is that
| ASLR by design treats the base address as a secret and
| guards it as such whereas that's not a mode the IPv6
| address can have because by its nature it's assumed to be
| something public.
| fc417fc802 wrote:
| Huh. The IPv6 example is much more confusing that I
| initially thought. At this point I am entirely unclear as
| to whether it is actually an example of security through
| obscurity, regardless of whatever else it might be (a
| very bad idea to rely on it for one). Rather ironic given
| that the poster whose claims I was disputing provided it
| as an example of something that would be universally
| recognized as such.
| vlovich123 wrote:
| I think it's security through obscurity because in ASLR
| the randomized base address is a protected secret key
| material whereas in the ipv6 case it's unprotected key
| material (eg every hop between two communicating parties
| sees the secret). It's close though which is why IPv6
| mapping efforts are much more heuristics based than ipv4
| which you can just brute force (along with port #)
| quickly these days.
| fc417fc802 wrote:
| I'm finding this semantic rabbit hole surprisingly
| amusing.
|
| The problem with that line of reasoning is that it
| implies that data handling practices can determine
| whether or not a given scheme is security through
| obscurity. But that doesn't fit the prototypical example
| where someone uses a super secret and utterly broken home
| rolled "encryption" algorithm. Nor does it fit the
| example of someone being careless with the key material
| for a well established algorithm.
|
| The key defining characteristic of that example is that
| the security hinges on the secrecy of the blueprints
| themselves.
|
| I think a case can also be made for a slightly more
| literal interpretation of the term where security depends
| on part of the design being different from the
| mainstream. For example running a niche OS making your
| systems less statistically likely to be targeted in the
| first place. In that case the secrecy of the blueprints
| no longer matters - it's the societal scale analogue of
| the former example.
|
| I think the IPv6 example hinges on the semantic question
| of whether a network address is considered part of the
| blueprint or part of the input. In the ASLR analogue, the
| corresponding question is whether a function pointer is
| part of the blueprint or part of the input.
| vlovich123 wrote:
| > The problem with that line of reasoning is that it
| implies that data handling practices can determine
| whether or not a given scheme is security through
| obscurity
|
| Necessary but not sufficient condition. For example, if
| I'm transmitting secrets across the wire in plain text
| that's clearly security through obscurity even if you're
| relying on an otherwise secure algorithm. Security is a
| holistic practice and you can't ignore secrets management
| separate from the algorithm blueprint (which itself is
| also a necessary but not sufficient condition).
| ryao wrote:
| If someone can guess the randomization within 1 second
| using the AnC attack, you can restart as much as you
| want, but it will not improve security.
| hgomersall wrote:
| In the limit, an encryption key falls to the same logic.
| You simply rely on the difficulty of scanning all
| possibly keys.
| dotancohen wrote:
| > The correct definition is simply "using obscurity as a
| security defense mechanism", nothing more.
|
| Also stated as "security happens in layers", and often
| obscurity is a very good layer for keeping most of the
| script kiddies away and keeping the logs clean.
|
| My personal favorite example is using a non-default SSH
| port. Even if you keep it under 1024, so it's still on a
| root-controlled port, you'll cut down the attacks by an
| order of magnitude or two. It's not going to keep the NSA
| or MSS out, but it's still effective in pushing away the
| common script kiddies. You could even get creative and
| play with port knocking - that keeps under-1024 ports
| logs clean.
| ryao wrote:
| I use non-standard SSH ports too. It does not improve
| theoretical security, but it does improve quality of life
| from generating smaller logs.
| bigstrat2003 wrote:
| Some people use the "flag" button as a "disagree" button
| or even a "fuck this guy" button. Unfortunately,
| constructive but unpopular comments get flagged to death
| on HN all the time.
| usefulcat wrote:
| I had thought that flagging was basically a request for a
| mod to have a look at something. But based on this case I
| now suspect that it's possible for a comment to be
| removed without a mod ever looking at it if enough people
| flag it.
| buttercraft wrote:
| Not removed but hidden. You can turn on showdead in your
| profile to see them.
| usefulcat wrote:
| My point was more that, at least in this case, it looks
| like a post was hidden without any moderator
| intervention.
|
| If this is indeed what happened, it seems like a bad
| thing that it's even possible. Since many, perhaps most
| people probably don't have showdead enabled, it means
| that the 'flag' option is effectively a mega-downvote.
| worthless-trash wrote:
| I believe most people see security through obscurity has
| an attempt to hide an insecurity.
|
| ASLR/KASLR intends to make attackers lives harder by
| having non consistent offsets of known data structures.
| Its not obscuring a security flaw, instead its raises an
| attacks 'single run' effectivness.
|
| The ASLR attack that i believe is being referenced is
| specific to abuse within the browser, and running with a
| single process. This single attack vector does not mean
| that KASLR globally is not effective.
|
| Your quote has some choice words, but its contextually
| poor.
| ryao wrote:
| That attack does not require a web browser. The web
| browser being able to do it showed it was higher severity
| than you would think than if the proof of concept had
| been in C, since web browsers run untrusted code all of
| the time.
| throw16180339 wrote:
| I downvoted because the poster doesn't understand what
| security by obscurity means.
| bigstrat2003 wrote:
| ASLR is _by definition_ security through obscurity. That
| doesn 't make it useless, as there's nothing wrong with
| using obscurity as one layer of defenses. But that
| doesn't change what it fundamentally is: obscuring
| information so that an attacker has to work harder.
| greiskul wrote:
| Is having a secret password security by obscurity? What
| about a private key?
|
| Security by obscurity is about the bad practice of
| thinking that obscuring your mechanisms and
| implementations of security increases your security. It's
| about people that think that by using their nephew's own
| super secret unpublished encryption they will be more
| secure than by using hardened standard encryption
| libraries.
| robertlagrant wrote:
| Security by obscurity is keeping your security algorithms
| and design secret, not your data at runtime secret.
| classichasclass wrote:
| Except in this case there can be a whole bunch of
| parallel bears.
| nickysielicki wrote:
| it's a total distortion of what the phrase means.
|
| Security through obscurity is when you run your sshd
| server on port 1337 instead of 22 without actually
| securing the server settings down, because you don't
| think the hackers know how to portscan that high.
| Everyone runs on 22, but you obscurely run it elsewhere.
| "Nobody will think to look."
|
| ASLR is nothing like that. It's not that nobody thinks to
| look, it's that they have no stable gadgets to jump to.
| The only way to get around that is to leak the mapping or
| work with the handful of gadgets that are stable. It's
| analogous to shuffling a deck of cards before and after
| every hand to protect against card counters. Entire
| cities in barren deserts have been built on the real
| mathematical win that comes from that. It's real.
| BoingBoomTschak wrote:
| Scrambling != Obscuring. Obscuring to me means that
| there's a fixed something to hide that can be discovered
| and exploited.
| ryao wrote:
| ASLR is technically a form of security by obscurity. The
| obscurity here being the memory layout. The reason nobody
| treated it that way was the high entropy that ASLR had on
| 64-bit, but the ASLR[?]Cache attack has undermined that
| significantly. You really do not want ASLR to be what
| determines whether an attacker takes control of your
| machine if you care about having a secure system.
| pdpi wrote:
| The defining characteristic of security through obscurity
| is that the effectiveness of the security measure depends
| on the attacker not knowing about the measure at all.
| That description doesn't apply to ASLR.
| ryao wrote:
| It produces a randomization either at compile time or run
| time, and the randomization is the security measure,
| which is obscured based on the idea that nobody can
| figure it out with ease. It is a poor security measure
| given the AnC attack that I mentioned. ASLR randomization
| is effectively this when such attacks are applicable:
|
| https://web.archive.org/web/20240123122515if_/https://www
| .sy...
| graymatters wrote:
| You are confusing randomization, a legitimate security
| mechanism, with security by obscurity. ASLR is not
| security by obscurity. Please spend the time on
| understanding the terminology rather than regurgitating
| buzz words.
| ryao wrote:
| I understand the terminology. I even took a graduate
| course on the subject. I stand by what I wrote. Better
| yet, this describes ASLR when the AnC attack applies:
|
| https://web.archive.org/web/20240123122515if_/https://www
| .sy...
| eru wrote:
| Why? You could advertise publicly what your flags are.
| smarx007 wrote:
| There are also many flags that should be enabled by default
| for non-debug builds like ubsan, stack protection, see
| https://news.ycombinator.com/item?id=35758898
| ryao wrote:
| UBSAN is usually a debug build only thing. You can run it
| in production for some added safety, but it comes at a
| performance cost and theoretically, if you test all
| execution paths on a debug build and fix all complaints,
| there should be no benefit to running it in production.
| smarx007 wrote:
| I think it's time for the C/C++ communities to consider a
| mindset shift and pivot to having almost all protectors,
| canaries, sanitizers, assertions (e.g. via
| _GLIBCXX_ASSERTIONS) on by default and recommended for
| use in release builds in production. The opposite (i.e,
| the current state of affairs) should be discouraged and
| begrudginly accepted in select few cases.
|
| https://www.youtube.com/watch?v=gG4BJ23BFBE is a
| presentation that best represents my view on the kind of
| mindset that's long overdue to become the new norm in our
| industry.
| estebarb wrote:
| By hardwiring the allocator you may end up with binaries that
| load two different allocators. It is too fun to debug a
| program that is using jemalloc free to release memory
| allocated by glibc. Unless you know what you are doing, it is
| better to leave it as is.
| ryao wrote:
| A userland package manager like Gentoo Prefix could be used to
| install a custom build of this and still get security updates.
| rlpb wrote:
| Indeed, there are many methods to have a custom build and
| still get security updates, including at least one method
| that is native to Ubuntu and doesn't need any external
| tooling. However my warning refers to the method presented in
| the article, where this isn't the case.
| alextingle wrote:
| > including at least one method that is native to Ubuntu
| and doesn't need any external tooling.
|
| Can you explain a little more? Search has failed me on this
| one.
| Onavo wrote:
| What about PGO?
| ryao wrote:
| https://en.wikipedia.org/wiki/Profile-guided_optimization
| jeffbee wrote:
| This is generally true but specifically false. The builds
| described in the gist are still linking onigurama dynamically.
| It is in another package, libonig5, that would be updated
| normally.
| rlpb wrote:
| The gist uses the orig tarball only, so skips the distro
| patch that selects the distro packaged libonig in favor of
| the "vendored" one. At least that's how it appears to me. I
| only skimmed the situation.
|
| Or do you see something deeper that ensures that the distro
| libonig is actually the one that gets used?
| jmward01 wrote:
| But isn't there still the kernel of an idea here for a package
| management system that intelligently decides to build based on
| platform? Seems like a lot of performance to leave on the
| table.
| stkdump wrote:
| Rebuilding from scratch also takes longer than installing a
| prebuilt package. So while it might be worth it for a heavily
| used application, in general I doubt it.
|
| Also I think in earlier days the argument to build was so you
| can optimize the application for the specific capabilities of
| your system like the supported SIMD instruction set or
| similar. I think nowadays that is much less of a factor.
| Instead it would probably be better to do things like that on
| a package or distribution level (i.e. have one binary
| distribution package prebuilt by the distribution for
| different CPU capabilities).
| taeric wrote:
| I'm curious how applicable these are, in general? Feels like
| pointing out that using interior doors in your house misses out
| on the security afforded from a vault door. Not wrong, but
| there is also a reason every door in a bank is not a vault
| door.
|
| That is, I don't want to devalue the CVE system; but it is also
| undeniable that there are major differences in impact between
| findings?
| skrebbel wrote:
| > I don't want to devalue the CVE system
|
| You could, though. It's 99.9% stuff like this!
| lpapez wrote:
| In my experience, most CVEs are reports about ice cream
| trucks lacking nuclear-proof bank vault doors.
| rlpb wrote:
| Sure, but jq is very much a "front door" in your analogy.
| You'd have to look at each individual CVE to assess the risk
| for your specific case, but for jq, claimed security
| vulnerabilities are worth paying attention to.
| guappa wrote:
| The normal way is to use dpkg to rebuild and patch, and use dch
| to increase the patch version with a .1 or something similar,
| so that the OS version always takes precedence, and then
| rebuild.
| teitoklien wrote:
| Gentoo linux is essentially made specifically for people like
| this, to be able to optimize one's own linux rig for one's
| specific usecase.
|
| After initial setup, it's pretty simple and easy to use, I
| remember making a ton of friends at matrix's Gentoo Linux
| channel, was fun times.
|
| https://www.gentoo.org/
|
| Fun fact, initial ChromeOS was basically just custom Gentoo Linux
| install, I'm not sure if they still use Gentoo Linux internally.
| shanemhansen wrote:
| So ChromeOS and also the OS for GKE are still basically built
| this way.
| 6SixTy wrote:
| Afaik the Gentoo based ChromeOS is being replaced by Android.
| mycall wrote:
| It will be interesting if Gentoo supported Fuchsia next.
| surajrmal wrote:
| What would that even mean?
| sham1 wrote:
| Portage on top of Fuchsia, probably. There's not all that
| much more to Gentoo all things considered.
| nickysielicki wrote:
| Zircon is miles apart from linux. It's like saying you'd
| run Portage on top of Windows. You and what army?
| chasil wrote:
| Windows was built with a POSIX subsystem from the very
| beginning. WSL 1 uses something analogous.
|
| https://en.m.wikipedia.org/wiki/Microsoft_POSIX_subsystem
|
| The Zircon kernel does not support signals, so basic C is
| not going to work well.
|
| "It is heavily inspired by Unix kernels, but differs
| greatly. For example, it does not support Unix-like
| signals, but incorporates event-driven programming and
| the observer pattern."
|
| https://en.m.wikipedia.org/wiki/Fuchsia_(operating_system
| )#K...
| 6SixTy wrote:
| As long as there's Bash and Python support, Gentoo Prefix
| could as well. That's kind of the only things that are
| hard needed for Gentoo as Portage is written in Python
| and Ebuilds are Bash scripts. Bigger issue would be that
| the Fuchsia kernel likely isn't POSIX complete.
| ryao wrote:
| Gentoo Portage had support for running on top of Interix,
| so it did run on Windows at some point (over 15 years
| ago).
| yjftsjthsd-h wrote:
| Gentoo already has prefix support for non-Linux systems and
| used to have at least some interest in a full
| Gentoo/kFreeBSD, so it's plausible.
| eru wrote:
| I used Gentoo for a while, but the temptation to endlessly
| fiddle with everything always let me to eventually break the
| system. (It's not Gentoo's fault, it's mine.)
|
| Afterwards I moved to ArchLinux, and that has been mostly fine
| for me.
|
| If you are using a fairly standard processor, then Gentoo
| shouldn't give you that much of an advantage?
| ryao wrote:
| Gentoo lets you do all of the tweaks mentioned here within
| the system package manager, so you still get security updates
| for your tweaked build. You can also install Gentoo on top of
| another system via Gentoo Prefix for use as a userland
| packages manager:
|
| https://wiki.gentoo.org/wiki/Project:Prefix
| eru wrote:
| > Gentoo lets you do all of the tweaks mentioned here
| within the system package manager, so you still get
| security updates for your tweaked build.
|
| Yes, Gentoo is great. I'm just saying that for me it was
| too much of a temptation.
| serbuvlad wrote:
| I'd like to bring attention to the ALHP repos[1].
|
| These are the Arch packages built for x86-64-v2, x86-64-v3
| and x86-64-v4, which are basically names for different sets
| of x86-64 extensions. Selecting the highest level supported
| by your processor should get you most of the way to
| -march=native, without the hassle of compiling it yourself.
|
| It also enables -O3 and LTO for all packages.
|
| [1]: https://github.com/an0nfunc/ALHP
| eru wrote:
| Nice, I'll try them out!
|
| LTO is great, but I have my doubts about -O3 (vs the more
| conservative -O2).
|
| UPDATE: bah, ALHP repos don't support the nvidia drivers.
| And I don't want to muck around with setting everything up
| again.
|
| Another update: I moved to nvidia-open, so now I can try
| the suggested repos.
| darkwater wrote:
| All in under 2 hours. You were totally nerd-sniped.
| eru wrote:
| Well, I moved to the new repos. But I had to change to
| the dkms drivers.
|
| Seems to work so far, but nothing 'feels' faster either.
| alessandroberna wrote:
| You should be able to use the regular arch repos as a
| fallback (and even repos from alhp with a lower feature
| level https://somegit.dev/ALHP/ALHP.GO/issues/255#issueco
| mment-333...)
|
| doing this should allow you to use as many optimized
| packages as possible while still being able to install
| packages not supported by the alhp
| globular-toast wrote:
| I've broken Arch but never broken Gentoo. I think this more
| due to the fact I ran Arch first and you then Gentoo first,
| rather than any real difference between them.
|
| Gentoo is more stable than Arch by default, though. It's not
| actually a bleeding edge distro, but you can choose to run it
| that way if you wish. Gentoo is about choice.
| eru wrote:
| > I think this more due to the fact I ran Arch first and
| you then Gentoo first, rather than any real difference
| between them.
|
| I can believe that.
|
| > Gentoo is more stable than Arch by default, though. It's
| not actually a bleeding edge distro, but you can choose to
| run it that way if you wish. Gentoo is about choice.
|
| I actually had way more trouble with stuff breaking with
| Ubuntu. That's because every six months, when I did the
| distro upgrade, lots of stuff broke at once and it was hard
| to do root cause analysis.
|
| With a rolling distribution, it's usually only one thing
| breaking at a time.
| johnisgood wrote:
| Ubuntu et al. are notorious for breaking stuff. If you
| want to remove a package, it sometimes want to delete
| your Linux kernel. :|
| johnisgood wrote:
| I have used Gentoo and Arch. I still favor Arch, however. I
| like pacman (C), I don't like Portage (Python).
|
| I found Arch Linux to be more stable than Gentoo, but that
| is just my own experience.
| donio wrote:
| > Gentoo linux is essentially made specifically for people like
| this, to be able to optimize one's own linux rig for one's
| specific usecase.
|
| That's true but worth noting that "optimize" here doesn't
| necessarily refer to performance.
|
| I've been using Gentoo for 20 years and performance was never
| the reason. Gentoo is great if you know how you want things to
| work. Gentoo helps you get there.
| kennysoona wrote:
| If it wasn't for performance, what was gained in using it
| over something like Slackware and building only the packages
| you needed to?
| xlii wrote:
| Long time ago when I was using it I preferred Gentoo
| because of ergonomics and better exposition to supply
| chain.
|
| Slackware was very manual and some bits were drowned in its
| low level and long command chains. Gentoo felt easy but
| highlighted dependencies with a hard cost associated with
| compilation times.
|
| Being a newb back then I enjoyed user friendliness with
| access to the machinery beneath. Satisfaction of a 1s boot
| time speedu, a result of 48h+ compilation, was
| unparalleled, too ;)
| CamouflagedKiwi wrote:
| USE flags. You can build packages with specific features
| enabled or disabled, which can further reduce your
| dependency tree.
| kennysoona wrote:
| For building specific packages with particular flags,
| wouldn't slackbuilds have been sufficient?
| stakhanov wrote:
| Reducing the dependency tree gets a bit more complicated
| once you consider that now you have to satisfy not only
| runtime dependencies for all packages but also build-time
| dependencies. There may be ways of cleaning that up after
| a build, but next time you want to emerge a new package
| you'll just end up having to re-build the build-time
| dependencies, so in practice you'll just end up leaving
| them there. There is an ability to emerge packages to a
| separate part of the filesystem tree (ROOT="/my/chroot"
| emerge bla), so that you have one build-time system act
| as a kind of incubator for a runtime system that gets to
| be minimal. But you'll end up encountering problems that
| most other Gentoo users wouldn't encounter, having to do
| with the separation between build-time dependencies and
| runtime dependencies not being correctly made in the
| recipes. Personally, I had been relying on this feature
| for roughly the last 10 years, but there has been steady
| deterioration there over the years and I eventually gave
| up late last year.
| cb321 wrote:
| This is a good point. I've been using Gentoo since early
| 2004 (the dreaded Pentium IV era, Lol). Lately, I run
| into this with dev-lang/tcl only being need to build dev-
| db/sqlite. I actually think it's pretty weird that
| software intended to be as widely used as sqlite with as
| much of a free base of supporting devs doesn't just do
| the extra effort to use a Makefile.
| 0xbadcafebee wrote:
| > If it wasn't for performance, what was gained
|
| A new hobby
| kennysoona wrote:
| That's very fair, I've certainly tinkered with stuff for
| the fun of it when there were easier or more suitable
| alternatives.
| vasco wrote:
| Never seen the HN version of the 'install gentoo' meme before,
| more sophisticated definitely.
|
| > The goal of Gentoo is to have an operating system that builds
| all programs from source, instead of having pre-built binary
| packages. While this does allow for advanced speed and
| customizability, it means that even the most basic components
| such as the kernel must be compiled from source. It is known
| through out the Linux community as being a very complex
| operating system because of its daunting install process. The
| default Gentoo install boots straight to a command prompt, from
| which the user must manually partition the disk, download a
| package known as a "Stage 3 tarball", extract it, and build the
| system up by manually installing packages. New or inexperienced
| users will often not know what to do when they boot in to the
| installer to find there is no graphical display. Members of /g/
| will often exaggerate the values of Gentoo, trying to trick new
| users in to attempting to install it.
| toyg wrote:
| Where does that blurb come from, chatgpt? I don't think it's
| true anymore, last time I checked I think Gentoo had a
| "normal" liveCD installation for the base system, which you
| could then recompile on your own if wanted.
| Vilkku wrote:
| Seems to be from https://knowyourmeme.com/memes/install-
| gentoo
| _joel wrote:
| GRP (Gentoo packages) existed at least 20 years ago, from
| my memory, as that's the last time I really used it in
| anger. I remeber packages being available and not having to
| rice everything, for sure.
| ndesaulniers wrote:
| They still use the build system. Woof.
| stakhanov wrote:
| I had had Gentoo continuously in use since 2003, and only very
| recently moved off of it (late 2024) when I tried Void Linux.
| On Void, buildability from source by end users is not a
| declared goal nor architectural feature, but you have a pretty
| decent chance of being able to make it work. You can expect one
| or two hiccups, but if you have decent all-round Linux
| experience, chances are you'll be able to jump into the build
| recipes, fix them, make everything work for what you need it to
| do, and contribute the fixes back upstream. This is what you
| get from a relentless focus on minimalism and avoiding
| overengineering of any kind. It's what I had been missing in
| Gentoo all those years. With Gentoo, I always ended up having
| to fiddle with use flags and package masks in ways that
| wouldn't be useful to other users. The build system is so
| complex that it had been just too difficult for me, over all
| these years, to properly learn it and learn to fix problems at
| the root cause level and contribute them upstream. Void should
| also be an ideal basis for when you don't want to build the
| entire system from source, but you just want to mix & match
| distro-provided binaries with packages you've built from source
| (possibly on the basis of a modified build recipe to better
| match your needs or your hardware).
| boobsbr wrote:
| Obligatory "Gentoo is rice" mention:
|
| https://www.shlomifish.org/humour/by-others/funroll-loops/Ge...
| jsight wrote:
| I tried that once, but it is still compiling. :)
|
| Kidding... honestly that was a pretty fun distribution to play
| around with ~20 years ago. The documentation was really good
| and it was a great way to learn how a lot of the pieces of a
| Linux distribution fit together.
|
| I was never convinced that the performance difference was
| really noticeable, though.
| wantless wrote:
| Gentoo was the primary source of heating for my living
| quarters back in the early 2000s. My tower was highly
| constrained on memory, and I was on a relentless quest to
| pare out any modules or dependencies I wasn't actually using.
| Performance gains were primarily from being able to stay out
| of slow HDD swap space memory. I doubt there were any gains
| once amortizing the compilation times, but I ran my compile
| batches at night, and they kept me nice and warm.
| ac130kz wrote:
| -march=native + mimalloc (or jemalloc) should be sufficient
| without causing significant undefined behavior like -O3 or most
| extra optimization related compiler arguments.
| eru wrote:
| Yes, -O3 is seldom an improvement for most C compilers.
| loeg wrote:
| -O3 doesn't cause undefined behavior.
| MyOutfitIsVague wrote:
| No, but it can often expose undefined behavior that doesn't
| stop a program from "working" on lower optimization levels.
|
| Which is why I'm in support of always building your own
| software with -O3.
| vlovich123 wrote:
| Are you sure about that? I'm not aware of any additional UB
| exploitation enabled by O3 vs O2 or Os.
| MyOutfitIsVague wrote:
| Nope, I'm not sure about it. I remember when I was using
| Gentoo about 10 years ago, this was the common reason
| given for using -O2 instead of -O3 in your build flags,
| and I'm just speaking from that memory.
| vlovich123 wrote:
| Yeah there's lots of cargo culting FUD like that that
| happens (see other posters in this thread that claim that
| glibc is a more stable allocator )
| DrNosferatu wrote:
| Sometimes comes to mind:
|
| Can't one recompile the same exact Ubuntu packages you already
| have on your system with optimal flags for your specific
| hardware?
|
| Plus, the configuration could be automated - just push a button
| to (lazy, in background) recompile.
|
| Priority for the ones profiled to be performance critical.
| eru wrote:
| > Can't one recompile the same exact Ubuntu packages you
| already have on your system with optimal flags for your
| specific hardware?
|
| Well, in principle yes. But you'd also need to figure out what
| you mean by 'optimal' flags? The flags might differ between
| different packages, and not all packages will be compatible
| with all the flags.
|
| Even worse, in the example in the article they got most of
| their speedup out of moving to a different allocator, and
| that's not necessarily compatible with all packages (nor an
| improvement for all packages).
|
| However, if you still want to do all these things, then Gentoo
| Linux is the distribution for you. It supports this tinkering
| directly and you'll have a much happier time than trying to
| bend Ubuntu to your will.
| zX41ZdbW wrote:
| ClickHouse will be faster for processing large JSON data files.
| The example from the gist will look like follows:
| ch -q "WITH arrayJoin(features) AS f SELECT
| f.properties.SitusCity WHERE f.properties.TotalNetValue < 193000
| FROM 'data.json'"
|
| Reference: https://jsonbench.com/
| jeffbee wrote:
| I'm willing to believe that will execute in less than 2
| seconds, but it doesn't work as given.
| rustc wrote:
| Can you post a link to the JSON file?
|
| Try this (the placement of FROM was incorrect):
| ch "WITH arrayJoin(features) AS f SELECT
| f.properties.SitusCity FROM 'a.json' WHERE
| f.properties.TotalNetValue < 193000"
| jeffbee wrote:
| It's a public data file accessible at https://data.acgov.or
| g/datasets/2b026350b5dd40b18ed7a321fdcd...
| 77pt77 wrote:
| If you decide to go down this road just use Gentoo.
| benatkin wrote:
| Not a bad idea. Arch also comes to mind, and its support on
| https://copy.sh/v86/ is a testament to its flexibility.
| m000 wrote:
| I haven't kept up with Gentoo. How long does it take these days
| to bootstrap a functional desktop from source?
|
| I remember when it was my main driver back in early '00s,
| running on a horribly underpowered PII at 266MHz. It would
| often be compiling 24/7 to keep up with the updates.
| 77pt77 wrote:
| Haven't used it in a looong time, but by 2010 it was no
| longer that bad.
|
| My first install was 2002 and it took my a good 24 hours to
| get X to boot.
|
| It did catch a faulty memory DIM later on because the linking
| kept failing intermittently.
| electromech wrote:
| I'd be curious how the performance compares to this Rust jq
| clone:
|
| cargo install --locked jaq
|
| (you might also be able to add RUSTFLAGS="-C target-cpu=native"
| to enable optimizations for your specific CPU family)
|
| "cargo install" is an underrated feature of Rust for exactly the
| kind of use case described in the article. Because it builds the
| tools from source, you can opt into platform-specific
| features/instructions that often aren't included in binaries
| built for compatibility with older CPUs. And no need to clone the
| repo or figure out how to build it; you get that for free.
|
| jaq[1] and yq[2] are my go-to options anytime I'm using jq and
| need a quick and easy performance boost.
|
| [1] https://github.com/01mf02/jaq
|
| [2] https://github.com/mikefarah/yq
| saghm wrote:
| As a bonus that people might not be aware of, in the cases
| where you do want to use the repo directly (either because
| there isn't a published package or maybe you want the latest
| commit that hasn't been released), `cargo install` also has a
| `--git` flag that lets you specify a URL to a repo. I've used
| this a number of times in the past, especially as an easy way
| for me to quickly install personal stuff that I throw together
| and push to a repo without needing to put together any sort of
| release process or manually copy around binaries to personal
| machines and keep track of the exact commits I've used to build
| them.
| oguz-ismail wrote:
| > I'd be curious how the performance compares to this Rust jq
| clone
|
| Every once in a while I test jaq against jq and gojq with my jq
| solution to AoC 2022 day 13 https://gist.github.com/oguz-
| ismail/8d0957dfeecc4f816ffee79d...
|
| It's still behind both as of today
| alberth wrote:
| Isn't this what ClearLinux essentially is (way over generalizing)
| rgmerk wrote:
| It's a while since I had to deal with this kind of thing, but my
| memory was that as soon as you go beyond the flags that the
| upstream developers use (just to be clear, I mean the upstream
| developers, not the distro packagers) you're buying yourself
| weird bugs and a whole lot of indifference if they occur.
|
| I haven't used a non-libc malloc before but I suspect the same
| applies.
| taeric wrote:
| I've been building my own emacs for a long time, and have yet
| to hit any weird bugs. I thought that as long as you avoid any
| unsafe optimizations, you should be fine? Granted, I also
| thought that -march=native was the main boost that I was
| seeing. This post indicates that is not necessarily the case.
|
| I also suspect that any application using floats is more likely
| to have rough edges?
| pertymcpert wrote:
| Complex software usually has some undefined behavior lurking
| that at higher or even just _different_ optimization levels
| can trigger the compiler to do unexpected things to the code.
| It happens all the time in my line of work. If there 's an
| extensive test suite you can run to verify that it still
| works mostly as expected then it's easier.
| taeric wrote:
| This is one where I suspect we don't disagree. But "all the
| time" can have a very different feel between people.
|
| It also used to happen that just changing processors was
| likely to find some problems in the code. I have no doubt
| that still happens, but I'd also expect it has reduced.
|
| Some of this has to be a convergence on far fewer compilers
| than we used to encounter. I know there are still many c
| compilers. Seems there are only two common ones, though.
| Embedded, of course, is a whole other bag of worms.
| Y_Y wrote:
| Did you try -ffast-math? IIRC that used to break emacs in
| some subtle way, while providing no extra speed.
| taeric wrote:
| I thought touching the math optimizations directly was in
| the "unsafe" bucket. Really the only optimization I was
| aiming for was -march=native. That and the features like
| native compilation that have made it to the release.
|
| I do think I saw improvements. But I never got numbers, so
| I'm assuming most of my feel was wishful thinking. Reality
| is a modern computer is hella fast for something like
| emacs.
|
| I did see compilation mode improve when I trimmed down the
| regexes it watches for to only the ones I knew were
| relevant for me. That said, I think I've stopped doing
| that; so maybe that is a lot better?
| Brian_K_White wrote:
| Two opposing things are both true at the same time.
|
| If you as an individual avoid being at all different, then you
| are in the most company and will likely have the most success
| in the short term.
|
| But it's also true that if we all do that then that leads to
| monoculture and monoculture is fragile and bad.
|
| It's _only_ because of people building code in different
| contexts (different platforms, compilers, options, libraries,
| etc...) that code ever becomes at all robust.
|
| A bug that you mostly don't trigger because your platform or
| build flags just happens to walk just a hair left of the hole
| in the ground, was still a bug and the code is still better for
| discovering and fixing it.
|
| We as individuals all benefit from code being generally robust
| instead of generally fragile.
| notpushkin wrote:
| On the other hand, _if_ your optimization helps consistently
| across platforms, you could convince upstream developers to
| implement it directly. (Not necessarily across _all_ platforms
| - a sizable performance gain on just a single arch might still
| be enough to tweak configuration for that particular build).
| john-tells-all wrote:
| If you actually do this, just get Ubuntu to download the exact
| source package you want. In this case: apt-get
| source jq
|
| Then go into the package and recompile to your heart's content.
| You can even repackage it for distribution or archiving.
|
| You'll get a result much closer to the upstream Ubuntu, as
| opposed to getting lots of weird errors and misalignments.
| smallstepforman wrote:
| Engineering is a compromise. The article shows most gains come
| from specialising the memory allocater. The thing to remember is
| that some projects are multithreaded, and allocate in one thread,
| use data in another and maybe deallocate in a 3rd. The allocator
| needs to handle this. So a speedup for one project may be a crash
| in another.
|
| Also, what about reallocation strategy? Some programs preallocate
| and never touch malloc again, others constantly release and
| acquire. How well do they handle fragmentation? What is the
| uptime (10 seconds or 10 years)? Sometimes the choice of
| allocators is the difference between long term stability vs short
| term speed.
|
| I experimented with different allocators devoloping a video
| editor testing 4K videos that caches frames. 32Mb per frame, at
| 60fps, thats almost 2Gb per second per track. You quickly hit
| allocator limitations, and realise that at least vanilla glibc
| allocator offers the best long term stability. But for short
| running benchmarks its the slowest.
|
| As already pointed out, engineering is a compromise.
| vlovich123 wrote:
| Mimalloc is a general purpose allocator like JEMalloc /
| TCMalloc. Glibc is known to have a pretty bad allocator that
| modern allocators like MIMalloc & the latest TCMalloc (not the
| one available by default in Ubuntu) run laps around. While of
| course speedup may be variable, generally the benchmarks show
| an across the board speedup (whether that matters for any given
| application is something entirely different). As for crashes,
| these are all general purpose multi-thread allocators and
| behave no differently from glibc (modulo bugs that can exist
| equally in glibc).
| smallstepforman wrote:
| Agree for most short running apps. I updated my comment to
| reflect issues with apps that are constantly reallocating,
| and running for longer that 60 seconds. But you are
| absolutely correct for most short running apps, 99%
| recommended to replace glibc. However, there is an app or two
| where glibc stability doesnt trigger a pathological use
| cases, and you have no choice. Hence why its the default,
| since there are less crashes in pathaligical cases. And the
| devs are exhausted dealing with crashing bugs which can be
| eliminated by using the slower allocator.
| vlovich123 wrote:
| Looked at your updated post and it looks like you're
| operating under wildly incorrect assumptions.
|
| 1. Fragmentation: MIMalloc and the newest TCMalloc
| definitely handle this better than glibc. This is well
| established in many many many benchmarks.
|
| 2. In terms of process lifetime, MIMalloc (Microsoft Cloud)
| and TCMalloc (Google Cloud) are designed to be run for
| massive long-lived services that continually
| allocate/deallocate over long periods of time. Indeed, they
| have much better system behavior in that allocating a bunch
| of objects & then freeing them actually ends up eventually
| releasing the memory back to the OS (something glibc does
| not do).
|
| > However, there is an app or two where glibc stability
| doesnt trigger a pathological use cases, and you have no
| choice.
|
| I'm going to challenge you to please produce an example
| with MIMalloc or the latest TCMalloc (or heck - even any
| real data point from some other popular allocators vs vague
| anectodes). This just simply is not something these
| allocators suffer from and would be major bugs the projects
| would solve.
| Y_Y wrote:
| Challenge yourself to produce some numbers first. If
| there are many many many benchmarks it shouldn't be too
| difficult to link one. Just saying something is "well
| established" doesn't really help without some other
| context.
| scns wrote:
| https://github.com/microsoft/mimalloc?tab=readme-ov-
| file#per...
| Y_Y wrote:
| Thanks for that. I have an innate skepticism of
| benchmarks from a company who wants to show their
| solution is best, and it seems their latest results are
| from 2021, but I couldn't find a better comparison myself
| either.
|
| I do note that rpmalloc (old), Hermes (not public?
| doesn't compare with mimalloc) and also snmalloc (also
| Microsoft) have benchmarks of their own showing
| themselves to be best in some circumstances.
|
| https://github.com/mjansson/rpmalloc-benchmark
|
| https://arxiv.org/abs/2109.02922
|
| https://github.com/SchrodingerZhu/bench_suite/blob/master
| /ou...
| Tuna-Fish wrote:
| More than that, based on the basic architecture of
| JEmalloc, mimalloc and tcmalloc, you should always expect
| their fragmentation behavior to be better for long-
| running software than the glibc malloc. (At the expense
| of consuming substantially more memory for very small
| programs with only a few allocations of a given size).
| The glibc malloc has nearly pessimal fragmentation
| behavior, you are very confused here.
| Y_Y wrote:
| > you are very confused here
|
| Is this addressed to me? If so would you do me the
| kindness of attempting to disabuse me of my confusion?
| Tuna-Fish wrote:
| Wait, no, sorry. That reply was supposed to go to a
| different post.
| jeffbee wrote:
| Oddly enough I've written about those results before:
|
| https://www.dropbox.com/scl/fi/evnn6yoornh9p6l7nq1t9/Irre
| pro...
| vlovich123 wrote:
| but we all can agree that glibc is trash right? :)
|
| I'd be careful extrapolating from just one benchmark but
| generally if I had to choose I'd pick the new tcmalloc if
| I could. It seems to be a higher quality codebase.
| jeffbee wrote:
| The problem with the current tcmalloc, which I agree is
| very good, is the difficulty of integrating it with
| random builds. It works well for large systems and
| especially those that are already bazelized, but I
| couldn't whip up a quick way to link it to `jq` in this
| gist.
| jcelerier wrote:
| I tried to use mimalloc and snmalloc and in both cases
| got crashes I don't get with glibc when interoperating
| with other libraries (libusb, jack, one that I suspect to
| be in the Nvidia driver) :(
| vlovich123 wrote:
| If you are not properly overriding the allocator
| consistently for everything within an executable, that's
| entirely possible (eg linking against 1 allocator and
| then linking with a dynamic library that's using a
| different one). Without a specific repro it's hard to
| distinguish PEBCAK from legit bug. Also it certainly
| can't be the Nvidia driver since that's not running
| anything in your process.
| kllrnohj wrote:
| > Also it certainly can't be the Nvidia driver since
| that's not running anything in your process.
|
| A huge chunk of a modern GPU driver is part of the
| calling process, loaded like a regular library. Just spot
| checking Chrome's GPU thread, there's dozens of threads
| created by a single 80+mb nvidia DLL. And this isn't
| unusual, every GPU driver has massive libraries loaded
| into the app using the GPU - often including entire
| copies of LLVM for things like shader compilers.
| blagie wrote:
| The explanation here is usually simpler. Any major change
| like this can lead to crashes.
|
| If I have three libraries, biz, bar, and bif, and each
| has bugs. You've used biz. When the system was unstable,
| you'd debug until it works (either by finding the real
| bug, or by an innocuous change).
|
| If you switch libraries, the bugs which go away are the
| ones which aren't affecting you, since your software
| works. On the other hand, you have a whole new set of
| bugs.
|
| This comes up when upgrading libraries too. More bugs are
| usually fixed than introduced, but there's often a debug
| cycle.
| akoboldfrying wrote:
| > This just simply is not something these allocators
| suffer from and would be major bugs the projects would
| solve.
|
| Not OP, but the following logic shows why this claim is
| bogus. In short: If two non-garbage-collecting memory
| allocators do _anything_ differently -- other than behave
| as perfect "mirror images" of each other, so that
| whenever one allocates byte i, the other allocates byte
| totalMem-i -- then there exists a program that crashes on
| one but not the other, and vice versa.
|
| In detail:
|
| If 2 allocators do not allocate exactly the same blocks
| of memory to the same underlying sequence of malloc() or
| free() calls, then there exists a program which, if built
| twice, once using each allocator, and then each
| executable is run with the same input, will after some
| time produce different patterns of memory fragmentation.
|
| The first time this difference appears -- let's say,
| after the first n calls to either malloc() or free() --
| the two executables will have the same total number of
| bytes allocated, but the specific _ranges_ of allocated
| bytes will be different. The nth such call must be a
| malloc() call (since if it were a free() call, and
| allocated ranges were identical after the first n-1 such
| calls, they would still be identical after the first n,
| contradicting our assumption that they are different).
| Then for each executable, this nth malloc() call either
| allocates a block at or some distance past the end, or it
| subdivides some existing free block. We can remove the
| latter possibility (and simplify the proof) by assuming
| that there is no more memory available past the end of
| the highest byte thus far allocated (this is allowed,
| since a computer with that amount of memory could exist).
|
| Now have both programs call free() on every allocated
| block except the one allocated in operation n. Let the
| resulting free range at the start of memory (before the
| sole remaining allocated block) have total length s1 in
| executable 1 and s2 in executable 2, and let the
| resulting free range at the end of memory (after that
| sole remaining allocated block) have length e1 in
| executable 1 and e2 in executable 2. By assumption,
| s1[?]s2 and e1[?]e2. Now have both executables call
| malloc() twice, namely, on s1 and e1 _in descending
| order_. Then, unless s1=e2, executable 1 can satisfy both
| malloc()s, but executable 2 can satisfy only the first.
| Similarly, calling malloc() on s2 and e2 in decreasing
| order will succeed in executable 2 but not executable 1,
| again unless s1=e2 holds.
|
| What if s1=e2 does hold, though? This occurs when, say,
| one executable allocates the block 100 bytes from the
| start of memory, while the other allocates it 100 bytes
| from the end. In this case, all we need is to keep some
| second, symmetry-breaking block around at the end in
| addition to the block allocated by operation n -- that
| is, a block for which it does _not_ hold that one
| allocator allocates the mirror-image memory range of the
| other. (If no such block exists, then the two allocators
| are perfect mirror images of each other.)
| vlovich123 wrote:
| I really have no idea what you're getting at here. This
| isn't embedded where there's a fixed pool to allocate out
| of. If there's insufficient space it'll get more virtual
| memory from the OS.
|
| Also, nothing you've said actually says that the other
| allocator will be the worse one. Indeed, glibc is known
| to hold onto memory longer and have more fragmentation
| than allocators like mimalloc and tcmalloc so I'm still
| at a loss to understand how even if what you wrote is
| correct (which I don't believe it is) that it follows
| that glibc is the one that won't crash. If you're
| confident in your proof by construction, please post a
| repro that we can all take a look at.
| akoboldfrying wrote:
| > If there's insufficient space it'll get more virtual
| memory from the OS.
|
| Swap space is finite too.
|
| > Also, nothing you've said actually says that the other
| allocator will be the worse one.
|
| I'm not claiming that either is worse. I'm showing
| mathematically that for _any_ two allocators that behave
| differently _at all_ (with the one tiny exception of a
| pair of allocators that are perfect mirror images of each
| other), it 's possible to craft a program that succeeds
| on one but fails on the other.
|
| I didn't say so explicitly as I thought it was obvious,
| but the upshot is: _It 's never completely safe to just
| change the allocator_. Even if 99% if the time, one works
| better than the other, there's _provably_ a corner case
| where it will fail but the other does not.
| akoboldfrying wrote:
| I should have been more explicit about the assumptions I
| make about an allocator:
|
| 1. If malloc() is called when there exists a contiguous
| block of free memory with size >= the argument to
| malloc(), the call will succeed. I think you'll agree
| that this is reasonable.
|
| 2. Bookkeeping (needed at least for tracking the free
| list, plus any indexes on top) uses the same amount of
| _malloc()able_ memory in each allocator. I.e., if
| malloc(x) at some point in time reduces the number of
| bytes that are available to future malloc() calls by y
| >= x bytes under allocator 1, it must reduce the number
| of bytes available to future malloc() calls by y under
| allocator 2 if called at the same point in time as well.
| This may not hold exactly in practice, but it's a very
| good approximation -- it's possible to store the free
| list "for free" by using the first few bytes as next and
| prev pointers in a doubly linked list.
|
| To head one another possible objection off at the pass:
| If the OS allocates lazily (i.e., it doesn't commit
| backing store at malloc() time, instead waiting till
| there is an actual access to the page, like Linux does),
| this doesn't change anything: Address space (even 64-bit
| address space) is still finite, and that is still being
| allocated eagerly. In practice, you could craft the
| differentially crashing program to crash much faster if
| you call memset() immediately on every freshly malloc()ed
| block to render this lazy commit ineffective -- then you
| would only need to exhaust the physical RAM + swap,
| rather than the complete 64-bit virtual address space.
| vlovich123 wrote:
| Swapping out allocators won't cause some programs to
| crash and others to not crash unless your program is
| already using up all available RAM or your program has a
| bug and the allocation pattern happens to be more likely
| to trigger invalid memory access in a way that crashes.
|
| This allocation pattern idea is unlikely to show up in
| any real application except at the absolute limit where
| your exhausting RAM and the OOM killer gets involved.
| Even then I think you're going to not see the allocator
| be much of a differentiating factor.
| yxhuvud wrote:
| Funny, cause the situations where I've had to replace glibc
| is always that it is a long running server that allocates
| often. Glibc: Ballooning memory, eventually crash.
| jemalloc: Stable as a rock.
| grandempire wrote:
| That's why it's a bad idea to use one allocator for everything
| in existence . It's terrible that everyone pays the cost of
| thread safety even for single threaded applications - or even
| multithreaded applications with disciplined resource
| management.
| yxhuvud wrote:
| While I do agree with the general sentiment, I think the
| default should be to use the safer ones that can handle multi
| threaded usage. If someone wants to use a bad allocator like
| glibc that doesn't handle concurrency well, then they should
| certainly be free to switch.
| grandempire wrote:
| So now you're using std vector and it's taking a lock for
| no reason, even though push_back on the same vector across
| threads isn't thread safe to begin with.
|
| Haphazard multithreading is not a sane default.
|
| I understand a million decisions have been made so that we
| can't go flip that switch back off, but we've got to learn
| these lessons for the future.
| yxhuvud wrote:
| Whatever. Allocation is slow enough that I don't care
| about a noncontended lock. Make things work by default
| and if you want to gain performance by not allowing
| multithreading then that should be possible and easy. But
| safety first, as in most cases it really don't matter.
| When it comes to mainstream general purpose allocators it
| isn't really a tradeoff anyhow, as all of them are
| nominally threadsafe.
|
| Even glibc claims to be multithreading safe even if it
| tends to not return or reuse all freed memory.
| silisili wrote:
| This is a common pain point.
|
| Write in a language that makes sense for the project. Then
| people tell you that you should have used this other language,
| for reasons.
|
| Use a compression algo that makes sense for your data. Then
| people will tell you why you are stupid and should have used
| this other algo.
|
| My most recent memory of this was needing to compress specific
| long json strings to fit in Dynamo. I exhaustively tested every
| popular algo, and Brotli came out far ahead. But that didn't
| stop every passerby from telling me that zlib is better.
|
| It's rather exhausting at times...
| silisili wrote:
| note: in the above, I meant 'zstd', not 'zlib'.
| 1dom wrote:
| Sorry, I'm struggling to make sense of this comment. I don't
| know C or C compilers very well at all, but I read the full
| gist and felt I learned a bunch of stuff and got a lot of value
| from it.
|
| But then when I read this top comment, it makes me concerned
| I've completely misunderstood the article. From the tone of
| this comment, I assume that I shouldn't ever do what's talked
| about in this gist and it's a terrible suggestion that
| overlooks all these complexities that you understand and have
| referenced with rhetorical-looking questions.
|
| Any chance you could help me understand if the original gist is
| good, makes any legitimate points, or has any value at all?
| Because I thought it did until I saw this was the top comment,
| and it made me realise I'm not smart enough to be able to tell.
| You sound like you're smart enough to tell, and you're telling
| me only bad things.
| alextingle wrote:
| I'll have a go at explaining: The process described in the
| article isn't a simple recipe that you can apply to any
| program to achieve similar results.
|
| `jq` is a command-line program that fires up to do one job,
| and then dies. For such a program, the only property we
| really want to optimise is execution speed. We don't care
| about memory leaks, or how much memory the process uses
| (within reason). `jq` could probably avoid freeing memory
| completely, and that would be fine. So using a super-stupid
| allocator is a big win for `jq`. You could probably write
| your own and make it run even faster.
|
| But for a program with different runtime characteristics, the
| results might be radically different. A long-lived server
| program might need to avoid memory bloat more than it needs
| to run fast. Or it might need to ensure stability more than
| speed or size. Or maybe speed does matter, but it's
| throughput speed rather than latency. Each of those cases
| need to be measured differently, and may respond better to
| different optimisation strategies.
|
| The comment that confused you is just trying to speak a word
| of caution about applying the article's recipe in a
| simplistic way. In the real world, optimisation can be quite
| an involved job.
| 1dom wrote:
| > The comment that confused you is just trying to speak a
| word of caution about applying the article's recipe in a
| simplistic way. In the real world, optimisation can be
| quite an involved job.
|
| I think that's what confused and irritated me. There's a
| lot of value and learning in the gist - I've used JQ in my
| previous jobs regularly, this is the real world, and
| valuable to many. But the top comment (at the time I
| responded) is largely rhetorically trashing the submission
| based on purely the title.
|
| I get that the gist won't make _everything_ faster: but I
| struggle to believe that any HN reader would genuinely
| believe that's either true, or a point that the author is
| trying to make. The literal first sentence of the
| submission clarifies the discussion is purely about JQ.
|
| Anyone can read a submission, ignore any legitimate value
| it in, pick some cases the submission wasn't trying to
| address, and then use those cases to rhetorically talk it
| down. I'm struggling to understand why/how that's bubbling
| to the top in a place of intellectual curiosity like HN.
|
| Edit: I should practice what I preach. Conversation and
| feedback which is purely cautionary or negative isn't a
| world that anyone really wants! Thanks for the response, I
| really appreciated it:) It was helpful in confirming my
| understanding that this submission does genuinely improve
| JQ on Ubuntu. Cautionary is often beneficial and necessary,
| and I think the original comment I responded to could make
| a better world with a single sentence confirming that this
| gist is actually valuable in the context it defines.
| scottlamb wrote:
| > I experimented with different allocators devoloping a video
| editor testing 4K videos that caches frames. 32Mb per frame, at
| 60fps, thats almost 2Gb per second per track. You quickly hit
| allocator limitations, and realise that at least vanilla glibc
| allocator offers the best long term stability. But for short
| running benchmarks its the slowest.
|
| I also work with large (8K) video frames [1]. If you're talking
| about the frames themselves, 60 allocations per second is
| nothing. In the case of glibc, it's slow for just one reason:
| each allocation exceeds DEFAULT_MMAP_THRESHOLD_MAX (= 32 MiB on
| 64-bit platforms), so (as documented in the mallopt manpage),
| you can not convince glibc to cache it. It directly requests
| the memory from the kernel with mmap and returns it with munmap
| each time. Those system calls are a little slow, and faulting
| in each page of memory on first touch is in my case slow enough
| that it's impossible to meet my performance goals.
|
| The solution is really simple: use your own freelist (on top of
| the general-purpose allocator or mmap, whatever) for just the
| video frames. It's a really steady number of allocations that
| are exactly the same size, so this works fine.
|
| [1] in UYVY format, this is slightly under 64 MiB; in I420
| format, this is slightly under 48 MiB.
| hedora wrote:
| I'd be shocked if jemalloc or tcmalloc had issues with that
| workload.
|
| Do you have a minimal reproducing example?
| scottlamb wrote:
| Sure, in plain C because why not. <https://gist.github.com/
| scottlamb/459a3ce6230be67bf4ceb1f1a8...>
|
| There's one other element I didn't mention in my previous
| comment, which is a thread handoff. It may be significant
| because it trashes any thread-specific arena and/or because
| it introduces a little bit of variability over a single
| malloc at a time.
|
| For whatever reason the absolute rate on my test machine is
| much higher than in my actual program (my actual program
| does other things with a more complex threading setup, has
| multiple video streams, etc.) but you can see the same
| effect of hitting the mmap, munmap, and page fault paths
| that really need not ever be exercised after program start.
|
| In my actual (Rust-based) program, adding like 20 lines of
| code for the pooling was a totally satisfactory solution
| and took me less time than switching general-purpose
| allocator, so I didn't try others. (Also, my program
| supports aarch64 and iirc the vendored jemalloc in the
| tikv-jemallocator crate doesn't compile cleanly there.)
| jeffbee wrote:
| Buffers of that size are also in tcmalloc's "whatever" class,
| right? It just does a smarter[1] job of not unmapping them.
|
| 1: for a certain point of view
| brucehoult wrote:
| Since the test data is not available for us to experiment
| ourselves, I've asked the poster to try it with Boehm GC compiled
| as a transparent malloc() replacement.
| coolgoose wrote:
| Reminds me how I learned Linux by bootstrapping Gentoo from
| Knoppix live cd's on my Athlon xp 1600+ and using God knows how
| many compilations flags
| dlcarrier wrote:
| For a lot of packages, installing the .deb file instead of the
| Snap package can in and of itself result in several times faster
| performance.
| sedatk wrote:
| When I was using Linux in 1995, installing a package meant
| downloading the sources and running "make install". Have we come
| full circle?
| sneak wrote:
| I love the part where he showed us the commands to rebuild the
| packages with different compilers and LDFLAGS.
| rubatuga wrote:
| Chimera Linux defaults to using clang + mimalloc
|
| Read more here: https://chimera-linux.org/about/
| polskibus wrote:
| This is gentoo resurrected! Great memories building entire Linux
| from scratch on install ;)
| cyounkins wrote:
| Why is glibc malloc() not more performant? Are tcmalloc/mimalloc
| making tradeoffs that maintainers are unwilling to make in glibc?
| citrin_ru wrote:
| AFAIK ptmalloc (on which glibc is based) was created decades
| ago and both multi-threaded application and multi CPU systems
| where rare back then (at least in the Linux world) so multi-
| threaded performance didn't matter. Some improvement in glibc
| were made since then but I don't think it's possible to
| significantly improve glibc malloc without rewriting it more or
| less fully. At that point it would make more sense to import
| some existing malloc implementation.
|
| And we are speaking about trafeoff the default number of arenas
| in glibc malloc is 8 times CPU which is a terrible tradeoff -
| on many workloads it cause heap fragmentation and memory usage
| (RSS) many times higher than allocated memory size, that's why
| it is common to find advice to set MALLOC_ARENA_MAX to 1 or 2.
| But probably such high number of areans allows glib to look
| less bad on synthetic benchmakrs.
|
| Jemalloc, tcmalloc, mimalloc all were created with focus on
| multi-threaded applications from the beginning and while they
| don't work better than glibc malloc for single threaded
| application they don't work worse for this use case either.
| Probably the main disadvantage of using je/tc/mi mallocs for a
| single threaded app is large code size.
| dgan wrote:
| what makes me jealous,is not the double performance ,but the fact
| that he can handle 13000 files in 2 seconds. i am at my
| ${corporate} job now, and have about 40'000 json files stored
| locally on the windows laptop. when I need to load a them, it
| takes _minutes_. i am still not sure if i read files wrong or
| NTFS is a pile of trash when comes to reading many smallish files
| nurettin wrote:
| Does ${corporate} prevent you from merging your jsons to a
| single file with a decent format like parquet?
| dgan wrote:
| they don't but that defeats the purpose. i want to be able to
| inspect the insides, so binary formats are out, i have to
| easily open & read the content
| nurettin wrote:
| I don't get how you can't put the content within the merged
| file and query it but good luck.
| Hackbraten wrote:
| There might me some kind of malware scanner at play.
|
| Maybe it's time to rebuild those with mimalloc, too.
| HdS84 wrote:
| Try excluding that directory from the virus scanner. Also a dev
| drive might come in handy for this workload
| dgan wrote:
| you were right. it didn't occur to me that antivirus would
| scan everything I open. Moved all files into an excluded
| directory, reading files is now almost 10 times faster.
| Thanks kind stranger !
| SkiFire13 wrote:
| NTFS is definitely very slow when it comes to reading many
| small files. Windows Defender or whatever antivirus you might
| be using can also further slow this down.
| v1ne wrote:
| NTFS is slow, especially when you operate on a lot of tiny
| files (nobody in the Windows world would do that, you'd always
| put your tiny data blobs into a bigger container file, e.g.
| asset files in games), but from my corporate experience, it's
| mostly the _multiple_ "endpoint security" solutions that bog
| file system performance down.
|
| It's the reason I so far use a Mac at work, which has its own
| issues, and a lot of them.
| scoopr wrote:
| Could be perhaps worth the trouble to try dev-drive/ReFS? [0]
|
| [0] https://learn.microsoft.com/en-us/windows/dev-drive/
| globular-toast wrote:
| No GCC march=native?
| penguin_booze wrote:
| Wouldn't that be the default?
| rurban wrote:
| He added it of course
| globular-toast wrote:
| With clang. GCC is known to do better with march=native, or
| at least it was.
| yxhuvud wrote:
| What i wonder is when any main stream distro will wake up and
| change their default allocator away from the very dated glibc.
| teekert wrote:
| Ah yes, this is nice but it sounds like Gentoo with extra steps.
| Probably also some extra benefits. But there is a reason we don't
| all run Gentoo. Although given a large and seamless enough cache,
| maybe we would.
| rurban wrote:
| He is still missing profile guided optimizations (with his
| testcase), and bolt.
|
| And in some/most cases -Os is faster than -O3
|
| Boehm GC is usually slower than glibc with lots of allocs. MPS
| would be better, but needs lots of rewriting.
| https://github.com/Ravenbrook/mps
| roeles wrote:
| > And in some/most cases -Os is faster than -O3 This surprises
| me. Can you please elaborate?
| withinboredom wrote:
| If the whole program can fit in the L1/L2 cache, it can speed
| up things quite a lot!
| rurban wrote:
| A IC cache miss costs 50x more on average. If your insn and
| data are small, the overhead of doing more work is less than
| the overhead of cache misses
| jeffbee wrote:
| BOLT is only good for ~2% in this scenario, which is about what
| I would have expected from glancing at the profiles.
|
| -Os is much slower, as I would have expected. Nothing in the
| perf data suggested high L1i miss rate. The .text of the "size
| optimized" program is, for some reason, 16x larger.
| hyperman1 wrote:
| Reading this, I wonder a few things that seem distro quick wins:
|
| 1) Why don't distros replace the glibc allocator with one of the
| better ones?
|
| 2) Why don't distros allow for making server-specific builds for
| only a few packages? You don't have to pay the compilation cost
| for everything, just a list of 2 or 3 packages.
| vkazanov wrote:
| Glibc is a reliable generalist while most other ones specialise
| in something: single thread, multiple threads, slab, arena
| style, etc ..
|
| Some of the other things disabled might be useful for distros
| in general, like ndebug assertions or compiling with debig
| symbols.
| hyperman1 wrote:
| Is glibc being generalist still true? I might be wrong here,
| but I had the impression it was open and free but behind most
| others,and e.g. tcmalloc is better on most criteria. Do we
| have benchmarks and comparisons for the generalist case?
| hinkley wrote:
| Is the glibc allocator still that bad? I thought they replaced
| that ages ago.
| nottorp wrote:
| Make a specific ubuntu package 90% faster by rebuilding it after
| changing the memory allocator, that is.
|
| Should probably work for Debian and RedHat too. For this
| particular package.
|
| Edit: based just on the title i initially thought this is an
| article about turning Ubuntu into Linux From Scratch.
| Aachen wrote:
| Reading that such a simple change can get such a big speedup, my
| first thought is to let the authors of jq know. Maybe there's a
| caveat to be aware of, maybe they'll test it and end up making it
| faster for everyone. Useful to drop a quick note pretty much no
| matter the result, I think?
|
| The article doesn't seem to even consider that option and I don't
| see any comment here mentioning this either. Am I missing
| something?
| chasil wrote:
| I would be curious if Intel Clear Linux had similar gains, by
| using newer opcodes in the instruction set.
|
| I wonder if the glibc allocator is standard there.
|
| https://en.m.wikipedia.org/wiki/Clear_Linux_OS
| jbverschoor wrote:
| On FreeBSD doing a make world / make buildworld used to be so
| common
| wengo314 wrote:
| "Make one Ubuntu package 90% faster by rebuilding it and
| switching the memory allocator"
|
| i wish i could slap people in the face over standard tcp/ip for
| clickbait. it was ONE package and some gains were not realized by
| recompilation.
|
| i have to give it to him, i have preloaded jemalloc to one
| program to swap malloc implementation and results have been very
| pleasant. not in terms of performance (did not measure) but in
| stabilizing said application's memory usage. it actually fixed a
| problem that appeared to be a memory leak, but probably wasn't
| fault of the app itself (likely memory fragmentation with
| standard malloc)
| dietr1ch wrote:
| To me it's obviously a scam because there's no way such an
| improvement can be achieved globally with a single post
| explanation. 90% faster is a micro-benchmark number.
| looofooo0 wrote:
| True that, I mean it is still interesting, that if you have a
| narrow task, you might achieve some significant speed up from
| rebuilding them. But this is a very niche application.
| wengo314 wrote:
| true, i saw a thread recently on reddit where guy hand-
| tuned compilation flags and did pgo profiling for a video
| encoder app that he uses on video encode farm.
|
| In his case, even a gain of ~20% was significant. It
| calculated into extra bandwidth to encode a few thousand
| more video files per year.
| arghwhat wrote:
| This is neither a micro-benchmark nor a scam, but it is
| click-bait by not mentioning jq specifically.
|
| Micro-benchmarks would be testing e.g. a single library
| function or syscall rather than the whole application. This
| is the whole application, just not one you might care that
| much for the performance of.
|
| Other applications will of course see different results, but
| stuff like enabling LTO, tuning THP and picking a suitable
| allocator are good, universal recommendations.
| FooBarWidget wrote:
| I did research into the glibc memory allocator. Turns out this
| is not memory fragmentation, but per-thread caches that are
| never freed back to the kernel! A free() call does not actually
| free the memory externally unless in exceptional circumstances.
| The more threads and CPU cores you have, the worse this problem
| becomes.
|
| One easy solution is setting the "magic" environment variable
| MALLOC_ARENA_MAX=2, which limits the number of caches.
|
| Another solution is having the application call malloc_trim()
| regularly, which purges the caches. But this requires
| application source changes.
|
| https://www.joyfulbikeshedding.com/blog/2019-03-14-what-caus...
| glandium wrote:
| The glibc memory allocator DOES have pathological cases that
| lead to what can look like memory leaks. See
| https://glandium.org/blog/?p=3698
| https://glandium.org/blog/?p=3723
| https://sourceware.org/bugzilla/show_bug.cgi?id=23416
| (despite being rather old, it's also still a problem)
| wengo314 wrote:
| FWIW i had it with icinga2. so now they actually preload
| jemalloc in the service file to mitigate the issue, this
| may very well be what you're talking about
|
| in case someone is interested:
| https://github.com/Icinga/icinga2/issues/8737
|
| (basically using jemalloc was the big fix)
|
| https://icinga.com/docs/icinga-2/latest/doc/15-troubleshoot
| i...
| blablabla123 wrote:
| True, I also believed it for a second. But it's also easy to
| blame Ubuntu for errors. IMHO they are doing a quite decent job
| with assembling their packages. In fact they are also compiled
| with Stack fortifications. On the other hand I'm glad they are
| _not_ compiled with the possibly buggy _-O3_. It can be nice
| for something performance critical but I definitely don 't want
| a whole system compiled with -O3.
| kllrnohj wrote:
| > with the possibly buggy -O3
|
| -O3 isn't buggy on either GCC or Clang. Are you thinking of
| -Ofast and/or -ffast-math that disregard standards
| compliance? Those aren't part of -O3.
| BoingBoomTschak wrote:
| -O3 itself isn't "buggy", but since it uses more
| optimizations, it can reveal issues in them. Other Gentoo
| users know: e.g.
| https://bugs.gentoo.org/show_bug.cgi?id=941208
| https://bugs.gentoo.org/show_bug.cgi?id=940923 (search O3
| in the bugzilla).
| kllrnohj wrote:
| strict-aliasing, which is what caused that bug to
| manifest, is enabled at O2.
| BoingBoomTschak wrote:
| Yep, it's what caused the bug to manifest, but who knows
| if that UB would have caused -O2 optimizations to mangle
| the result as well.
|
| EDIT: first one is -funswitch-loops, though
| kllrnohj wrote:
| Probably would have. There's no shortage of UB bugs
| revealed by O2 after all, including major security
| issues.
| margana wrote:
| I thought it would be something like recompiling to utilize
| AVX512 capabilities or something.
| tremon wrote:
| Vector operations like AVX512 will not magically make common
| software faster. The number of applications that deal with
| regular operations on large blocks of data is pretty much
| limited to graphical applications, neural networks and bulk
| cryptographic operations. Even audio processing doesn't
| benefit that much from vector operations because a codec's
| variable-size packets do not allow for efficient
| vectorization (the main exception being multi-channel effects
| processing as used in DAW).
| isotypic wrote:
| Vector operations are widely used in common software. Java
| uses AVX512 for sorting. glibc uses SIMD instructions for
| string operations.
| cratermoon wrote:
| I wonder how many prepackaged binary distributions are built
| with the safest options for the os/hardware and don't achieve
| the best possible performance. I bet most of them, tbh. Many
| years ago I started building Mozilla and my own linux kernels
| to my preferences, usually realizing modest performance gains.
| The entire purpose of the Gentoo Linux distribution, e.g., is
| performance gains possible by optimized compilation of
| _everything_ from source.
| tonymet wrote:
| the title is clickbait, but it's good to encourage app
| developers to rebuild. esp when you are cpu bound on a few
| common utitilities e.g. jq, grep, ffmpeg, ocrmypdf -- common
| unix utils built build targets for general use rather than a
| specific application
| UncleEntity wrote:
| Or, if I understand TFA correctly, don't release debug builds
| in your release packages.
|
| Reminds me of back in the day, when I was messing around with
| blender's cmake config files quite a bit, I noticed the
| fedora package was using the wrong flag -- some sort of debug
| only flag intended for developers instead of whatever they
| thought is was. I mentioned this to the package maintainer,
| it was confirmed by package sub-maintainer (or whomever) and
| the maintainer absolutely refused to change it because the
| spelling of the two flags was close enough they could just
| say "go away, contributing blender dev, you have no idea what
| you're talking about." Wouldn't doubt the fedora package
| still has the same mistaken flag to this day and all this
| occurred something like 15 years ago.
|
| So, yeah, don't release debug builds if you're a distro
| package maintainer.
| mooktakim wrote:
| Was big fan of Gentoo. Would spend hours and hours optimizing the
| compile flags to maximise performance. Fun times.
| lifeinthevoid wrote:
| People will run into subtle bugs and weird behavior when starting
| to compile things themselves and replacing libraries I think. Not
| worth the effort for me, fun as an exercise, but would only do it
| if it's really (really) needed for performance.
| p0w3n3d wrote:
| I wonder why my brain calculates `90% faster' as 10% of the time
| (i.e. t1=0.1*t0) it used to be working. However, this makes sense
| only if we calculate the recompiled application running time as
| t1=t0/1.90
| withinboredom wrote:
| "by 190%" instead of "90% faster" is far clearer.
| freep1zza wrote:
| Even Canonical themselves sobered up and abandoned their v3
| recompilation effort as it showed no serious benefits.
|
| Distros who went this path is going the way of the dinousaurs,
| together with their confused user base.
| sheepscreek wrote:
| In the late 1990s, the most common method for distributing a
| package was by providing its source code as a tarball. You
| essentially had to run ./configure and make on every package. I
| believe ./configure does some of what the OP did in a clever way,
| checking for locally installed alternatives, etc. It's still
| pretty cool, though. I didn't think a malloc replacement would
| significantly improve performance.
| mbar84 wrote:
| Why would you give your product a quite specific name that's
| exactly the same as another product? "Core 2 Duo" immediately
| brings an Intel CPU to mind and will probably make it harder to
| search for anything related to the watch.
| tpoacher wrote:
| Obviously the title / article is clickbait.
|
| But in general I have often found great benefit in many cases in
| obtaining packages from source and compiling them, or downloading
| official binaries, even if the repository does have the latest
| version in theory.
|
| (shameless plug incoming): My only obstacle with such "Manually
| Installed [and/or] Source-Compiled" (MISC) packages was checking
| for updates. So I made this: https://sr.ht/~tpapastylianou/misc-
| updater/
|
| Works great, and I've been slowly improving it to add 'upgrader'
| commands for some of the packages too.
|
| It's not a full replacement for a package manager, nor is it
| meant to be. But it's made my ability to use MISC packages
| instead of repository ones a lot more streamlined and easier.
| kedmi wrote:
| Please rebuild Chrome
| jeffbee wrote:
| Chrome is optimized right up to its eyebrows. If you are using
| Google distributions of Chrome binaries, I wouldn't expect
| improvements from build tweaks.
| simlevesque wrote:
| Isn't GeoJSON a really bad tool for what you are trying to do ?
| Just use GeoParquet:
|
| Converter: https://geoparquet.org/convert/
|
| It'll scan way less data and be 90% faster than your 90% faster
| jq.
| jeffbee wrote:
| GeoJSON is an interchange format that a government agency is
| using to provide this data, and it updates continuously, so at
| some point I will be interested in the efficiency of the ETL
| even if I start doing all my queries against an optimized
| format.
| mannyv wrote:
| What you should take away from this is that there are things you
| can do to speed up your app that are outside of normal code
| optimization.
| rajaravivarma_r wrote:
| It is funny, how being a corporate-Rails programmer taught me
| this early in my career. Because building Ruby from source was
| the only way you could install latest Ruby versions (using ruby-
| build) and installing Ruby from source means its dependencies
| like openssl, zlib had to be installed from source too, to match
| the required versions.
|
| And for a long time using Jemalloc was the only was to keep the
| memory usage constant with multi-threaded ruby programs like Puma
| and Sidekiq. This was either achieved through compiling Ruby with
| jemalloc or modifying the LD_LIBRARY_PATH.
|
| Some developers also reported 5% or so reduction in response
| times with jemalloc, iirc.
|
| The problem with this approach is though, when a package has a
| lot of dependencies like ImageMagick which relies on jpeg, png,
| ghost and a lot of other libraries, you have to take a trial and
| error approach until it succeeds. Fortunately fixing the
| dependency errors are the easiest, sometimes building Python from
| source would throw errors from headers which are impossible to
| understand. If you find a Stackoverflow solution then you are
| good, or you have to go down the rabbit hole coming either
| successful or empty handed based on your level of expertise.
___________________________________________________________________
(page generated 2025-03-19 23:02 UTC)