[HN Gopher] How simultaneous multithreading works under the hood
___________________________________________________________________
How simultaneous multithreading works under the hood
Author : rbanffy
Score : 154 points
Date : 2024-07-28 15:35 UTC (7 hours ago)
(HTM) web link (blog.codingconfessions.com)
(TXT) w3m dump (blog.codingconfessions.com)
| mhh__ wrote:
| Good summary overall, although seemed a little muddled in places.
|
| Would love to know some of the tricks of the trade from insiders
| (not relating to security at least)
| behnamoh wrote:
| Tangent: Ever since I became familiar with Erlang and the
| impressive BEAM, any other async method seems subpar and
| contrived, and that includes Python, Go, Rust, etc.
|
| It's just weird how there's a correct way to do async and
| parallelism (which erlang does) and literally no other language
| does it.
| lawn wrote:
| I'm a huge fan of the BEAM, but I wonder if you're not
| overselling it a little? Surely there are trade-offs here that
| sometimes aren't worth it and alternative ways are better?
|
| For example, the BEAM isn't optimized for throughput and if
| that's a high priority for you then you might want to choose
| something else (maybe Rust).
| behnamoh wrote:
| > For example, the BEAM isn't optimized for throughput...
|
| weird take, given that Erlang powers telecom systems with
| literally millions of connections at a time.
| deagle50 wrote:
| that's not throughput
| jjtheblunt wrote:
| but the traffic on each connection doesn't need bandwidth
| in the throughput sense mentioned
| JackSlateur wrote:
| I worked in telecoms, no erlang found. Perhaps we were too
| modern ?
| immibis wrote:
| I worked in telecoms. We had a third party component
| written in Erlang. I'd say it was about the second most
| reliable component of the system. It was susceptible to
| some memory leaks, but they usually turned out to be
| caused by misuse of the C client library.
|
| The most reliable component was written in C, with what
| must have been a space shuttle level of effort behind it.
| No memory allocation, except in code paths that can
| return an error to the user who asked for something that
| caused the system to need more memory (we probably got
| this wrong on our side of the API, and didn't test out of
| memory scenarios, and they probably would have just
| resulted in OOM kills anyway). Every line of code
| commented, sometimes leading to the infamous "//add 1 to
| i" but most times showing deep design thought. State
| machines documented with a paragraph for every state
| transition explaining non-obvious concerns.
| toast0 wrote:
| The original use that Erlang was built for was controlling
| telecom switches. In that application, Erlang supervises
| phone lines, responding to on/off hook and dialed numbers
| etc, but does not touch the voice path at all --- it only
| controls the relays/etc to connect the voice path as
| desired. That's not a high throughput job at all.
|
| I used Erlang at WhatsApp. Having a million chat
| connections was impressive, but that's not high throughput
| either. Most of those connections were idle. As we added
| more things to the chat connection, we ended up with a
| significantly lower target per machine (and then at
| Facebook, the servers got a lot smaller and the target
| connections per machine was way less).
|
| We did have some Erlang machines that pushed 20gbps for
| https downloads, but I don't think that's that impressive;
| serving static files with https at 20gbps with a 2690v4
| isn't hard to do if you have clients to pull the files.
|
| IMHO, Erlang is built for reliability/fault tolerance
| first. Process isolation happens to be good for both fault
| tolerance and enabling parallel processing. I find it to be
| a really nice environment to run lots of processes in, but
| it's clearly not trying to win performance prizes. If you
| need throughput, you need to at least process the data
| outside of Erlang (TLS protocol happens in Erlang, TLS
| crypto happens in C), and sometimes it's better to keep the
| data out of Erlang all together. Erlang is better suited
| for 'control plane' than 'data plane' applications; but
| it's 2024 and we have an abundance of compute power, so you
| can shoehorn a lot into a less than high performance
| environment ;)
| JackSlateur wrote:
| Could you share more intels about this ? Links or whatever
|
| I'd like to learn more
| lastofus wrote:
| Other languages do sometimes implement this at the library
| level. Clojure's core.async comes to mind (though there are
| subtle differences). There's downsides to this approach though.
|
| The data going into each "mailbox" either needs to be
| immutable, or deep copied to be thread safe. This obviously
| comes at a cost. Sometimes you just have a huge amount of state
| that different threads need to work on, and the above solution
| isn't viable. Erlang has ets/dets to help deal with this. You
| will notice ets/dets looks nothing like the mailbox/process
| pattern.
|
| Erlang is great, but it is hardly the "one true way". As with
| most things, tradeoffs are a thing, and usually the right
| solution comes down to "it depends".
| wongarsu wrote:
| > The data going into each "mailbox" either needs to be
| immutable, or deep copied to be thread safe
|
| Or moved. The mailbox/process pattern works great in Rust
| because you can simply move ownership of a value. Kind of
| like if in C you send the pointer and then delete your copy.
|
| Of course doing this across threads doesn't work with every
| type of value (what Rust's type system encodes as a value
| being `Send`). For example you can't send a reference-counted
| value if the reference counter isn't thread-safe. But that's
| rarely an issue and easily solved with a good type system.
| OnlyMortal wrote:
| What are you talking about?
|
| In C++ you have ASIO (Boost) that's mostly used for IPC but can
| be used as a general purpose async event queue. There is
| io_uring support too. You can sit a pool of threads as the
| consumers for events if you want to scale.
|
| C++ has had a defacto support for threads for ages (Boost) and
| it has been rolled into the standard library since 2011.
|
| If you're using compute clusters you also have MPI in Boost.
| That's a scatter/gather model.
|
| There's also OpenMP to parallelize loops if you're so inclined.
| bee_rider wrote:
| I should look this up but I'm lazy.
|
| I'm familiar with MPI in Fortran/C, and IIRC they had some
| MPI C++ primitives a that never really got a ton of traction
| (that's just the informal impression I got skimming their
| docs, though).
|
| How's MPI in C++ boost work? MPI communicates big dumb arrays
| best I think, so maybe they did all the plumbing work for
| translating Boost objects into big dumb arrays, communicating
| those, and then reconstructing them on the other side?
| OnlyMortal wrote:
| It's still a dumb way to do computation. The Boost library
| hides the implementation away though, on Linux and
| Macintosh, it requires gomp.
|
| Really, it's a wrapper but makes it "easier" for scientists
| to use who are, in general, not good coders.
|
| Source: I've seen CERN research code.
| bee_rider wrote:
| What's a dumb way to do computation? Using objects? I'm
| generally suspicious of divergence from the ideal form of
| computation (math, applied to a big dumb array) but C++
| is quite popular.
| mrkeen wrote:
| The actor model (share-nothing) is one way to address the
| problem of shared, mutable state.
|
| But what if I want to have my cake and eat it too? What if I
| want to have thread-safe, shared, mutable state. Is it not
| conceivable that there's a better approach than share-nothing?
| moffkalast wrote:
| As much as I hate to say it, I think Java probably has the
| best eaten cake implementation by far. Volatile makes sure
| variables stay sane on the memory side, if you only write to
| a variable from one thread and read it from others, then it
| just sort of magically works? Plus the executors to handle
| thread reuse for async tasks. I assume C# has the same
| concepts given that it's just a carbon copy with title case
| naming.
|
| Python can't execute in on two cores at once, so it
| functionally has no multithreading, JS can share data between
| threads, but must convert it all to string because pointless
| performance penalties are great to have. Golang has that
| weird FIFO channel thing (probably sockets in disguise for
| these last two). C/C++ has a segfault.
| mrkeen wrote:
| > Volatile makes sure variables stay sane on the memory
| side
|
| This doesn't get you from shared-mutable-hell to shared-
| mutable-safe, it gets you from shared-mutable-
| relaxedmemorymodel-hell to shared-mutable-hell. It's the
| kind of hell you don't come across until you start being
| too smart for synchronisation primitives and start taking a
| stab at lockfree/lockless wizardry.
|
| > if you only write to a variable from one thread and read
| it from others, then it just sort of magically works
|
| I'm not necessarily convinced by that - but either way
| that's a huge blow to 'shared' if you are only allowed one
| writer.
|
| > Plus the executors to handle thread reuse for async
| tasks.
|
| What does this solve with regard to the shared-mutable
| problem? This is like "Erlang has BEAM to handle the
| actors" or something - so what?
| moffkalast wrote:
| Well it doesn't get you there because shared-mutable-safe
| doesn't exist, at least I doubt it can without major
| tradeoffs. You either err on the side of complete safety
| with a system that is borderline unusable for anything
| practical, or you let people do whatever they want and
| let them deal with their problems once they actually have
| them.
|
| > either way that's a huge blow to 'shared' if you are
| only allowed one writer
|
| Yeah for full N thread reading and editing you'd need N
| vars per var which is annoying, but that kind of every-
| thread-is-main setup is something that is exceedingly
| rare. There's almost always a few fixed main ones and
| lots running specific tasks that don't really need to
| know about all the other ones.
| throwitaway1123 wrote:
| > JS can share data between threads, but must convert it
| all to string
|
| To be more precise, you can send data to web workers and
| worker threads by copying via the structured clone
| algorithm (unlike JSON this supports almost all data
| types), and you can also move certain transferable objects
| between threads which is a zero-copy (and therefore much
| faster) operation.
| moffkalast wrote:
| Ah yeah dataviews, but you still need to convert from
| json to those and that takes about as much overhead, plus
| they're much harder to deal with complexity-wise being
| annoying single type buffers and all. For any other
| language it would work better, but because JS mainly
| deals with data arriving from elsewhere it means it needs
| to be converted every single time instead of just
| maintaining a local copy for thread comms.
| throwitaway1123 wrote:
| > Ah yeah dataviews, but you still need to convert from
| json to those and that takes about as much overhead
|
| You don't necessarily need to have an intermediate JSON
| representation. Many of the built in APIs in Node and
| browsers return array buffers natively. For example:
| const buffer = await fetch('foo.wav').then(res =>
| res.arrayBuffer()) new
| Worker('worker.js').postMessage(buffer, [buffer])
|
| This completely transfers the buffer to the worker
| thread, after which it is detached (unusable from the
| sending side) [1][2].
|
| [1] https://developer.mozilla.org/en-
| US/docs/Web/API/Worker/post...
|
| [2] https://developer.mozilla.org/en-
| US/docs/Web/JavaScript/Refe...
| neonsunset wrote:
| > I assume C# has the same concepts given that it's just a
| carbon copy with title case naming.
|
| Better not comment than look like an idiot. Moreover, this
| applies the use of volatile keyword in Java as well.
| imtringued wrote:
| Go does something similar and ponylang is basically compiled
| Erlang.
| pavlov wrote:
| Intel's next generation Arrow Lake CPUs are supposed to remove
| hyperthreading (i.e. SMT) completely.
|
| The performance gains were always heavily application-dependent,
| so maybe it's better to simplify.
|
| Here's a recent discussion of when and where it makes sense:
| https://news.ycombinator.com/item?id=39097124
| PaulKeeble wrote:
| Most programs end up with some limitation on the number of
| threads they can reasonably used. When you have a lot less
| Cores than that SMT makes a lot of sense to better utilise the
| resources of the CPU. However once you get to the point where
| you have enough cores SMT no longer makes any sense. I am not
| convinced we are necessarily there yet but the P/E cores Intel
| are using are an alternative towards a similar goal and makes a
| lot of sense on the desktop given how many workloads are
| single/low threaded. I can see the value in not having to deal
| with SMT and E core distinctions in application optimisation.
|
| AMD on the other hand intends to keep mostly homogenous cores
| for now and continue to use SMT. I doubt its going to be simple
| to work out which strategy in practice is the best, its going
| to vary widely by application.
| variadix wrote:
| It is my understanding that SMT should be beneficial
| regardless of core count, as SMT should enable two threads
| that can stall waiting for memory fetches to fully utilize a
| single ALU, i.e. SMT improves ALU utilization in memory bound
| applications with multiple threads by interleaving ALU usage
| when each thread is waiting on memory. Maybe larger caches
| are reducing the benefits of SMT, but it should be beneficial
| as long as there are many threads who are generally bound by
| memory latency.
| t-3 wrote:
| > Maybe larger caches are reducing the benefits of SMT, but
| it should be beneficial as long as there are many threads
| who are generally bound by memory latency.
|
| I thought the reason SMT sometimes resulted in lower
| performance was that it halved the available cache per
| thread though - shouldn't larger caches make SMT _more_
| effective?
| jmb99 wrote:
| My understanding is that a larger cache can make SMT more
| effective, but like usual, only in certain cases.
|
| Let's imagine we have 8 cores with SMT, and we're running
| a task that (in theory) scales roughly linearly up to 16
| threads. If each thread's working memory is around half
| as much as there is cache available to each thread, but
| each working set is only used briefly, then SMT is going
| to be hugely beneficial: while one hyperthread is
| committing and fetching memory, the other one's cache is
| already full with a new working set and can begin
| computing. Increasing cache will increase the allowable
| working set size without causing cache contention between
| hyperthreads.
|
| Alternatively, if the working set is sufficiently large
| per thread (probably >2/3 the amount of cache available),
| SMT becomes substantially less useful. When the first
| hyperthread finishes its work, the second hyperthread has
| to still wait for some (or all) of its working set to be
| fetched from main memory (or higher cache levels if
| lucky). This may take just as long as simply keeping
| hyperthread #1 fed with new working sets. Increasing
| cache in this scenario will increase SMT performance
| almost linearly, until each hyperthread's working set can
| be prefetched into the lowest cache levels while the
| other hyperthread is busy working.
|
| Also consider the situation where the working set is
| much, much smaller than the available cache, but lots of
| computing must be done to it. In this case, a single
| hyperthread can continually be fed with new data, since
| the old set can be purged to main memory and the next set
| can be loaded into cache long before the current set is
| processed. SMT provides no benefit here no matter how
| large you grow the cache (unless the tasks use wildly
| different components of the core and they can be run at
| instruction-level parallelism - but that's tricky to get
| right and you may run into thermal or power throttling
| before you can actually get enough performance to make it
| worthwhile).
|
| Of course the real world is way more complicated than
| that. Many tasks do not scale linearly with more threads.
| Sometimes running on 6 "real" cores vs 12 SMT threads can
| result in no performance gain, but running on 8 "real"
| cores is 1/3 faster. And sometimes SMT will give you a
| non-linear speedup but a few more (non-SMT) cores will
| give you a better (but still non-linear) speedup. So
| short answer: yes, sometimes more cache makes SMT more
| viable, if your tasks can be 2x parallelized, have
| working sets around the size of the cache, and work on
| the same set for a notable chunk of the time required to
| store the old set and fetch the next one.
|
| And of course all of this requires the processor and/or
| compiler to be smart enough to ensure the cache is
| properly fed new data from main memory. This is
| frequently the case these days, but not always.
| hinkley wrote:
| I'd like to see the math for why it doesn't work out to have
| a model where n real cores share a set of logic units for
| rare instructions and a few common ones where say the average
| number of instructions per clock is 2.66 so four cores each
| have 2 apiece and then share 3 between them.
|
| When this whole idea first came up that's how I thought it
| was going to work, but we've had two virtual processors
| sharing all of their logic units in common instead.
| hinkley wrote:
| On common industry benchmarks at least every second generation
| of Intel hyperthreading ended up being slower than turning it
| off. Even when it worked it was barely double digit percent
| improvements, and there were periods when it was worse for
| consecutive generation. Why do they keep trying?
| tedunangst wrote:
| Because the benchmarks don't measure what people do with
| computers.
| YesBox wrote:
| Im creating a game + engine and speaking from personal
| experience/my use case, hyperthreading was less performant than
| (praying to the CPU thread allocation god) each thread
| utilizing its own core. I decided to max out the number of
| threads by using std::thread::hardware_concurrency() / 2 - 1.
| (i.e. number of cores - 1 ).
|
| I'm working with a std::vector
| bee_rider wrote:
| It seems a bit high-level, kind of skimming over a bunch of
| architecture concepts with a couple references to the fact that
| this might be duplicated when hyperthreading, this might not...
|
| IMO a blog post should be more actionable. This isn't a textbook
| chapter. For example we go through the frontend. When discussing
| the trace cache we have:
|
| > ... Instruction decoding is an expensive operation and some
| instructions need to be executed frequently. Having this cache
| helps the processor cut down the instruction execution latency.
|
| ...
|
| > Trace cache is shared dynamically between the two logical
| processors on an as needed basis.
|
| ...
|
| > Each entry in the cache is tagged with the thread information
| to distinguish the instructions of the two threads. The access to
| the trace cache is arbitrated between the two logical processors
| each cycle.
|
| So the threads share a trace cache, but keep track of which
| hyperthread used which instructions--but we don't really know,
| practically, if we prefer threads that are running very similar
| computations or if that is a non-issue (that is, does the fact
| that they share the trace cache mean one thread can benefit from
| things the other has cached? Or does the tagging keep them
| separated?).
|
| In general, often they say "this is split equally between the two
| threads" or "this is shared," which makes me wonder "if I disable
| SMT does the now single-thread get access to twice as much of
| this resource, and are there cases where that matters."
|
| This is somewhat covered in:
|
| > As we have seen, enabling SMT on a CPU core requires sharing
| many of the buffers and execution resources between the two
| logical processors. Even if there is only one thread running on
| an SMT enabled core, these resources remain unavailable to that
| thread which reduces its potential performance.
|
| But this seems a bit fuzzy to me, I mean, we talk about caches
| which are shared dynamically between the two threads so at least
| _some_ resources will be more readily available if only a single
| thread is running.
|
| It also could be interesting--if the author is an expert, perhaps
| they could share their experience as to which pipeline stages are
| often bottlenecks that get tighter with hyperthreads on, and
| which aren't? We have a sort of even focus on each stage without
| many hints as to which practically matter. Or how we can help
| them out. Also it is largely based on a 2002 whitepaper so I
| guess the specific pipeline stages must have evolved a bit since
| then.
|
| Or maybe they could share some battle stories, favorite tools,
| some examples of applications and why they put pressure on
| particular stages, things which surprisingly didn't scale when
| hyperthreads were enabled (I'm not asking for all these things,
| just any would be good).
| shaggie76 wrote:
| A grossly over-simplified argument for SMT that resonated with me
| was that it could keep a precious ALU busy while a thread stalls
| on a cache miss.
|
| I gather in the early days the LPDDR used on laptops was slower
| too and since cores were scarce so this was more valuable there.
| Lately, though, we often have more cores than we can scale with
| and the value is harder to appreciate. We even avoid scheduling
| work on a shared with an important thread to avoid cache-
| contention because we know the single-threaded performance will
| be the bottleneck.
|
| A while back I was testing Efficient/Performance cores and SMT
| cores for MT rendering with DirectX 12; on my i7-12700K I found
| no benefit to either: just using P-cores took about the same time
| to render a complex scene as P+SMT and P+E+SMT. It's not always a
| wash, though: on the Xbox Series X we found the same test
| marginally faster when we scheduled work for SMT too.
| bayindirh wrote:
| Rendering is one of the scenarios which was either same or
| slower with SMT since the beginning. This is because rendering
| is already math heavy, and your FPU is always active, esp.
| dividers (which is always the most expensive operation for
| processors).
|
| SMT shines while waiting for I/O or doing some simple integer
| stuff. If both your threads can saturate the FPU, SMT is
| generally slower because of the extra tagging added to the data
| inside the CPU to note what belongs where.
| hinkley wrote:
| But the way you make rendering embarrassingly parallel is the
| way you make web servers parallel; treat the system as a
| large number of discrete tasks with deadlines you work toward
| and avoid letting them interact with each other as much as
| possible.
|
| You don't worry about how long it takes to render one frame
| of a digital movie, you worry about how many CPU hours it
| takes to render five minutes of the movie.
| rcxdude wrote:
| If you're waiting for IO, you're likely getting booted off
| the processor by the OS anyway. SMT is most useful when your
| code doesn't have enough instruction-level parallelism but is
| still mostly compute bound.
| corysama wrote:
| I believe "I/O" here is referring to data movement between
| DRAM and registers. Not drives or NICs.
| gary_0 wrote:
| I wonder if instead of having SMT, processors could briefly
| power off the unused ALUs/FPUs while waiting for something
| further up the pipeline, and focus on reducing heat and power
| consumption rather than maximizing utilization.
| hinkley wrote:
| Could you, do they, put the "extra" LUs right next to the
| parts of the chip with the highest average thermal
| dissipation to even out the thermal load across the chip?
|
| Or stack them vertically, so the least consistently used
| parts of the chip are farthest away from the heat sink,
| delaying throttling.
| rcxdude wrote:
| They basically do: it's pretty common to clock gate inactive
| parts of the ALU, which reduces their power consumption
| greatly. Modern processor power usage is very workload-
| dependent for this reason.
| usrusr wrote:
| I consider SMT a relic left over from the days when CPU
| design was all about performance per square millimeter. We
| are in the process of substituting that goal with that of
| performance per watt, or in the process of slowly realizing
| that our goals have shifted quite a while ago.
|
| I really don't expect SMT to stay much longer. Even more so
| with timing visibility crosstalk issues lurking and big/small
| architectures offering more parallelism per chip area where
| single thread performance isn't in the spotlight. Or perhaps
| the marketing challenge of removing a feature that had once
| been the pride of the company is so big that SMT stays
| forever.
| hinkley wrote:
| At this point, especially with backside power, I wonder how
| much cache stalls on one processor result in less thermal
| throttling both on that processor and neighboring ones.
|
| Maybe we should just be letting these procs take their little
| naps?
| immibis wrote:
| This leads, in the extreme, to the idea of a huge array of
| very simple cores, which I believe is something that has been
| tried but never really caught on.
| makerofthings wrote:
| Sounds like gpu to me.
| pavlov wrote:
| The Xeon Phi was a "manycore" x86 design with lots of
| tiny CPU cores, something like the original Pentium, but
| with the addition of 512-bit SIMD and hyperthreading:
|
| https://en.m.wikipedia.org/wiki/Xeon_Phi
| orbat wrote:
| That description reminds me of GreenArrays'
| (https://www.greenarraychips.com) Forth chips that have 144
| cores - although they call them "computers" because they're
| more independent than regular CPU cores, and eg. each has
| its own memory and so on. Each "computer" is very simple
| and small - with a 180nm geometry they can cram 8 of them
| in 1mm^2, and the chip is fairly energy-efficient.
|
| Programming for these chips apparently a bit of a nightmare
| though. Because the "computers" are so simple, even eg.
| calculating MD5 turns into a fairly tricky proposition as
| you have to spread out the algorithm to multiple computers
| with very small amounts of memory, so something that would
| be very simple on a more classic processor turns into a
| very low level multithreaded ordeal
| gonzo wrote:
| Intel's hyperthreading is really a write pipe hack.
|
| It's not so much cache misses as allowing the core to run
| something else while the write completes.
|
| This is why some code scales poorly and other code achieves
| near linear speed ups.
| immibis wrote:
| Anecdotally, mkp224o (.onion vanity address miner, supposedly
| compute-bound with little memory access) runs about 5-10%
| faster on my 24-core AMD with 48 threads than with 24 threads.
| However, I haven't tried the same benchmark with SMT disabled
| in firmware.
| jeffbee wrote:
| One of the biggest mistakes users have is a mental model of SMT
| that imagines the existence of one "real core" and one inferior
| one. The threads are coequal in all observable respects.
| mgaunard wrote:
| The whole point of SMT is to maximize utilization of a
| superscalar execution engine.
|
| I wonder if that trend means people think superscalar is less
| important than it used to be.
| superjan wrote:
| What I think is worth knowing is that compute units in GPU's also
| use SMT, usually at a level of 7 to 10 threads per CU. This helps
| to hide latency.
| written-beyond wrote:
| Poor AMD their bulldozer architecture got so much flak for not
| including SMT and now everyone's moving away from it.
|
| Yes yes I know bulldozer had a bunch more issues than just no
| SMT. It actually had the exact opposite with multiple cores
| sharing the same ALU or something like that. But still they could
| have been onto something if they had made it marginally more
| performant.
| sweetjuly wrote:
| > As we have seen, enabling SMT on a CPU core requires sharing
| many of the buffers and execution resources between the two
| logical processors. Even if there is only one thread running on
| an SMT enabled core, these resources remain unavailable to that
| thread which reduces its potential performance.
|
| This isn't true (anymore?). We've seen a variety of SMT cores
| which partition the ROB, fetch/decode bandwidth, etc. when
| running in SMT mode but allow full use when not in SMT mode.
___________________________________________________________________
(page generated 2024-07-28 23:00 UTC)