[HN Gopher] Threadripper 3990X: The Quest To Compile 1B Lines Of...
___________________________________________________________________
Threadripper 3990X: The Quest To Compile 1B Lines Of C++ On 64
Cores
Author : fmxexpress
Score : 156 points
Date : 2021-02-03 17:55 UTC (5 hours ago)
(HTM) web link (blogs.embarcadero.com)
(TXT) w3m dump (blogs.embarcadero.com)
| tester756 wrote:
| Just try to compile LLVM - maybe not 1b of LoC, but that's
| definitely going to be challenging
| andy_ppp wrote:
| Does anyone have reviews of this on their JS test suite. The
| quicker the tests run the better my life, I have around 2000
| quite slow tests... 76s MacBook 15" 2016, 30s M1 Apple Silicon
| Mac Mini, what should I expect with loads more cores like this?
| nevi-me wrote:
| How parallel do the tests run? The Threadrippers have massive
| number of cores, but their per-core performance is lower than
| say a Ryzen 9.
| [deleted]
| smoldesu wrote:
| no, the macos compositor
| Yuioup wrote:
| Embarcadero? Are they still around?
| colejohnson66 wrote:
| They're still selling Delphi, for what it's worth
| bullen wrote:
| In my experience multi-core compilation does not work.
|
| make -j>3 just locks the process and fails.
| jcelerier wrote:
| You just need more ram. I 'ever compile at less than -j$(ncpu).
| Hard with less than 32 GB tho - a single clang instance can
| easily eat upwards of 1gb of ram
| bullen wrote:
| Aha, I only compile on ARM so I got no room to increase
| RAM...
|
| Is it the same with g++? I have 4GB so I should be able to
| compile with 4 cores, but the processes only fill 2-3 cores
| even when I try make -j8 on a 8 core machine and then locks
| the entire OS until it craps out?!
|
| Something is fishy...
| ahepp wrote:
| Why are you compiling on ARM with only 4GB RAM? Wouldn't it
| make more sense to cross compile from a machine with more
| resources, if you cared about build speed? (maybe there's a
| good reason not to do that, idk)
|
| If it's crapping out when you give it -j8, that seems to
| strongly suggest you're running into limited resources
| somewhere.
|
| I'm no expert in the intricacies of parallel builds, but as
| far as I know you can still have dependencies between
| targets that will limit parallelism.
| drmpeg wrote:
| A swap file is your friend.
| PragmaticPulp wrote:
| Fun experiment.
|
| The more pedestrian 5950X or the now bargain 3950X are great for
| anyone doing a lot of compiling. With the right motherboard they
| even have ECC RAM support. Game changer for workstations in the
| $1000-$2000 range.
|
| The more expensive Threadripper parts really shine when memory
| bandwidth becomes a bottleneck. In my experience, compiling code
| hasn't been very memory bandwidth limited. However, some of my
| simulation tools don't benefit much going from 8 to 16 cores with
| regular Ryzen CPUs because they're memory constrained.
| Threadripper has much higher memory bandwidth.
| PartiallyTyped wrote:
| IIRC AMD has EEC support in all x70/x50 motherboards and cpu
| combinations. If I may, what kind of simulations are you
| running?
|
| I am trying to build a system for Reinforcement Learning
| research and seeing many things depend on python, I am not
| certain how to best optimise the system.
| bmurphy1976 wrote:
| Where can you get decent ECC ram for a reasonable price? I was
| on the hunt recently for ECC RAM for my new desktop and I gave
| up and pulled the trigger on low latency non-ECC RAM.
| Availability seems to be pretty terrible at the moment.
| m463 wrote:
| same for everything. 5950? where would you even get one?
|
| wait - seems you can get one, just pay 2x list price.
| 55873445216111 wrote:
| You can get ECC UDIMMs from Supermicro. They are rebranded
| Micron DIMMs. ECC memory is not going to go as high of
| frequencies as you might be looking for. They will only go up
| to the officially validated speed of the CPUs.
| https://store.supermicro.com/16gb-ddr4-mem-
| dr416l-cv02-eu26....
| bmurphy1976 wrote:
| Oh thanks, I didn't even think to check Supermicro! The
| prices are reasonable as well.
| ska wrote:
| I suspect the biggest (build time) benefit to most c++
| workflows and toolchains was the move to ubiquitous SSD. Prior
| to that in my experience excepting expensive RAID array
| dedicated build machines, it was really easy to build a system
| that would always be IO bound on builds. There of course were
| tricks to improve things but you still tended to hit that wall
| unless your CPUs were really under spec.
|
| edit: to be clearer, I'm not thinking of dedicated build
| machines here (hence RAID comment) but over all impact on dev
| time by getting local builds a lot faster.
| trhway wrote:
| i (and some teamates) actually put HDDs on some workstations
| as SSD just die after 2-3 years of active build on them and
| with modern HDDs you have practically unlimited storage while
| you can have only limited number of 400G builds on SSD (the
| org has psychological barriers to having more than 1-2Tb SSD
| in a machine) and the SSD start to have perf issues when at
| 70-80% capacity . With HDD the build time didn't change much
| - the machines have enough memory for the system to cache a
| lot (256-512G RAM).
| EricE wrote:
| A way to have your cake and eat it too - check out
| Primocache. It's pretty inexpensive disk caching software
| (especially for Windows Server which is where I really
| leverage it!).
|
| Pair it with an Optane for L2 cache and it will speed up
| normal SSD use too ;)
| fctorial wrote:
| How does it compare to default linux caching algorithm?
| Dylan16807 wrote:
| The default one that caches disk data in memory? It
| solves a very different problem by caching data from
| slower disks onto faster disks. It can accelerate reads
| and it can also act as a writeback cache.
| outworlder wrote:
| > i actually put HDDs on some workstations as SSD just die
| after 2-3 years of active build on them
|
| That sounds very low for modern SSDs, even consumer-grade.
| Have you tried different vendors?
| winkeltripel wrote:
| If spending hours a day at 100% utility, SSDs will rarely
| last 5 years.
| Retric wrote:
| If your SSD would be at 100% utilization it's going to
| take a lot of HDD to reach that kind of bandwidth. To the
| point where for high bandwidth loads SSD's actually cost
| less even if you have to replace them regularly.
|
| 100% utilization and 30x the bandwidth = 30x as many HDD.
| Alternatively, if HDD's are an option you're a long way
| from 100% utilization.
| trhway wrote:
| SSD have hard time sustaining 200-400Mb/s write where is
| 4 HDD do is easily. Our case isn't that much about IOPS.
|
| Anyway, reasonably available SSDs have up to [1000 x SSD
| size] total write limit, so doing couple of 400G
| builds/day would use up the 1TB drive in 3 years. At
| worst times we had to develop&maintain 5 releases in
| parallel instead of regular 2-3.
| Sesse__ wrote:
| Urm what? A modern NVMe drive will sustain ~2 GB/sec
| write.
|
| (See e.g. https://cdn.mos.cms.futurecdn.net/Ko5Grx7WzFZAX
| k6do4SSf8-128..., from Tom's Hardware)
| adrian_b wrote:
| Few SSDs can sustain such a speed for a long time. After
| they exhaust their temporary SLC cache they drop to much
| lower speeds. SSDs that have accumulated a large amount
| of writes might also make large pauses at random times
| during writing, for garbage collection. Good SSDs remain
| faster than HDDs even in these circumstances, but they
| are nevertheless much slower than when benchmarked for
| short times while they are new.
| Retric wrote:
| You can get drives without such issues. For the kind of
| long sequential writes followed by deletes where HDD are
| are even vaguely comparable, good SSD's can maintain very
| close to their theoretical maximum speeds indefinitely.
|
| They only really slow down in usage patterns that trash
| traditional HDD's, which also have much worse disk
| fragmentation issues.
| nwmcsween wrote:
| 4 HDDs can do 200-400MB/s _sequential_ IO, 1 modern SSD
| can do 150-200MB/s _random_ IO and 400MB/s sequential IO
| while 4 HDDs would have a hard time doing IIRC more than
| 8MB/s random IO
| trhway wrote:
| i don't argue here. It is just random IO isn't that
| important for our use case, especially compare to
| capacity and sequential IO which are important for us and
| which HDD is ok for.
| [deleted]
| kyrra wrote:
| Depends on the kind of SSD. If it's using SLC, the write
| endurance is much much higher. If you're going with cheap
| SSDs (TLC or QLC), your write endurance will suck.
|
| see: https://www.anandtech.com/show/6459/samsung-
| ssd-840-testing-...
| mywittyname wrote:
| SLC seems to be going away pretty quickly, if it hasn't
| already been phased out. It just can't produce the price
| / GB of the better tech. Also, that article you linked is
| almost 10 years old.
|
| You're best bet for long-term reliability is to buy much
| more capacity than you need and try not to exceed >50%
| capacity for high write frequency situations. I keep an
| empty drive around to use as a temp directory for
| compiling, logging, temp files, etc.
|
| Also, my understanding is that consumer-grade drive need
| a "cool down" period to allow them to perform wear
| leveling. So you don't want to be writing to these drives
| constantly.
| pdimitar wrote:
| I recently bought an external 32GB SLC SSD (in the form
| factor of an USB pendrive). Its random read/write speeds
| are quite insane (130+ MB/s both) while consumer SSDs
| like the Samsung 850 Evo barely manage 30 MB/s
| read/write. It's also advertised as very durable.
|
| I plan on using a couple of those as ZFS metadata and
| small block caches for my home NAS, we'll see how it goes
| but people generally and universally praise the SLC SSDs
| for their durability.
|
| > _You 're best bet for long-term reliability is to buy
| much more capacity than you need and try not to exceed
| >50% capacity for high write frequency situations. I keep
| an empty drive around to use as a temp directory for
| compiling, logging, temp files, etc._
|
| That's likely true. I am pondering buying one external
| NVMe SSD for that purpose exactly.
| 1996 wrote:
| > I recently bought an external 32GB SLC SSD (in the form
| factor of an USB pendrive). Its random read/write speeds
| are quite insane (130+ MB/s both)
|
| Which brand/spec?
| pdimitar wrote:
| https://www.aliexpress.com/item/32846479966.html?spm=a2g0
| s.9...
|
| Vtran / eVtran. They have favourable reviews, too.
| d00bianista wrote:
| It's actually not better tech, instead it's more
| complicated, more error prone and less durable ways to
| use the same technology that produces more space for a
| lower price. MLC is pretty much okay but TLC is a little
| too fragile and low performance in my opinion. I prefer
| spinning HDD's over QLC since the spinning drives have
| predictable performance.
| mywittyname wrote:
| Better can mean a lot of different things. From context,
| I was using better to mean higher storage capacity for a
| given price.
| read_if_gay_ wrote:
| What kind of workload will do that?
| leetcrew wrote:
| a build server recompiling multiple branches over and
| over in response to changes.
| dragontamer wrote:
| And logging all of those unit tests associated with all
| of those builds (and rolling over those logs with TRACE
| level debugging).
|
| Every build gets fully tested at maximum TRACE logging,
| so that anyone who looks at the build / test later can
| search the logs for the bug.
|
| 8TBs of storage is a $200 hard drive. Fill it up, save
| everything. Buy 5 hard drives, copy data across them
| redundantly with ZFS and stuff.
|
| 1TBs of SSD storage is $100 (for 4GBps) to $200 (for
| 7GBps). Copy data from the hard drive array on the build
| server to the local workstation as you try to debug what
| went wrong in some unit test.
| officeplant wrote:
| My mind was blown when I had to send a fully logged 5
| minutes of system operation to a friend for diagnostics
| (MacOS 11 on an M1 Mini). He wasn't joking when he said
| don't go over a few minutes because the main 256GB system
| drive almost ran out of space in that time. After getting
| it compressed down from 80GB and sent over I got my mind
| blown again when he explained he had to move to his
| workstation with 512+gb of ram just to open the damn
| file.
| ska wrote:
| Those are pretty beefy workstations, does every developer
| have one or are these really build servers? As I noted you
| could always throw money at this and end up somewhere
| reasonable, but it introduces workflow considerations for
| your devs.
| trhway wrote:
| It is one per dev, and in reality it is at least a couple
| - I have 4 of them for example. The product itself is a
| beast too.
| ska wrote:
| That works! It helps that "plenty-of-ram" is more
| achievable than it used to be, also.
| fctorial wrote:
| What are the signs of an SSD that's about to die?
| trhway wrote:
| anecdotally i'd see filesystem errors and write speed
| dropping, sometimes writes practically hanging,
| especially if the drive is close to 90% capacity.
| fctorial wrote:
| Don't SSDs have a finite TBW? 50GB of writes everyday
| (possible on large projects) will consume that in a couple of
| months.
| frizkie wrote:
| They do but it's really large. The Tech Report did an
| endurance test on SSDs 5-6 years ago [0]. The tests took 18
| months before all 6 SSDs were dead.
|
| Generally you're looking at hundreds of terabytes, if not
| more than a petabyte in total write capacity before the
| drive is unusable.
|
| This is for older drives (~6 years old as I said), and I
| don't know enough about storage technology and where it's
| come since then to say, but I imagine things probably have
| not gotten worse.
|
| [0]: https://techreport.com/review/27909/the-ssd-endurance-
| experi...
| yread wrote:
| > things probably have not gotten worse.
|
| I am afraid they did, consumer SSDs moved from MLC (2
| bits per cell) to TLC (3) or QLC (4). Durability changed
| from multiple petabytes to low hundreds of terabytes.
| Still a lot, but I suspect the test would be a lot
| shorter now.
| Dylan16807 wrote:
| The TLC drive in that test did fine. 3D flash cells are
| generally big enough for TLC to not be a problem. I would
| only worry about write limits on QLC right now, and even
| then .05 drive writes per day is well under warranty
| limits so I'd expect it to be fine. Short-lived files are
| all going to the SLC cache anyway.
| dekhn wrote:
| I've swapped hundreds of terabytes to a terabyte SSD (off
| the shelf cheapie) with no recognizable problems (the
| gigapixel panoramas look fine).
| nine_k wrote:
| SSDs avoid catastrophic write failures by retiring
| damaged blocks. Check the reported capacity; it may have
| shrunk :) Before you ever see bad blocks, the drive will
| expend spare blocks; this means that a new drive you buy
| has 10% more capacity than advertised, and this capacity
| will be spent to replace blocks worn by writes.
| dekhn wrote:
| Now I'm curious how the kernel responds to having swap
| partitions resized under it.
| bluedino wrote:
| Something like the Samsung EVO 960 (typical mid-range SSD)
| will take 400TB of writes in it's lifetime. So that's 8,000
| days of 50GB writes.
| spullara wrote:
| Hmmm. Looks like I need to move my temp dir for GeForce
| instant replay off of my SSD. It records about 1.6GB/5min
| which is 460GB per day. RAM disk would probably be the
| best option.
| reitzensteinm wrote:
| I'm pretty sure it doesn't record the desktop with
| instant unless you have specifically set that up, so
| you'd only be writing to the drive while you're in game.
| lumost wrote:
| You know I wonder how much of an impact this has had on the
| recent move back to statically typed and compiled languages
| vs. interpreted languages. I had assumed most of the
| compilation speedups were due to enhancements to the compiler
| toolchain - but my local laptop moving from 100 IOPS to >
| 100k IOPS and 3GB/s throughput may have more to do with it.
| PragmaticPulp wrote:
| SSDs help, but nothing beats core count X clock speed when
| compiling.
|
| Source code files are relatively small and modern OSes are
| very good at caching. I ran out of SSD on my build server a
| while ago and had to use a mechanical HDD. To my surprise, it
| didn't impact build times as much as I thought it would.
| ska wrote:
| Hmm. Maybe the tradeoff has changed since I last tested
| this (to be fair, a few years ago). But I'm also not
| focused on build servers especially, it's always been
| possible to make those reasonably fast. Unless you have a
| very specific sort of workflow anyway, your devs are doing
| way more local builds than on the server and that sped up a
| ton moving to SSD, in my experience anyway. YMMV of course.
| MayeulC wrote:
| Well, you can always move your code to a ramdisk, I
| suspect any C(++) isn't more than a few GB anyway ?
| [deleted]
| masklinn wrote:
| > The more expensive Threadripper parts really shine when
| memory bandwidth becomes a bottleneck.
|
| Threadripper can be useful for IO, especially for C++ (which is
| famously quite IO intensive) owing to its 128 PCIe lanes, you
| can RAID0 a bunch of drives and have absolutely ridiculous IO.
| coliveira wrote:
| It is a good thing that Embarcadero is keeping alive this
| technology to create desktop apps from the early 2000s that was
| abandoned by MS and other large companies in favor of complex
| Web-based apps.
| cosmotic wrote:
| Someone has to build the native electron wrapper
| dvfjsdhgfv wrote:
| If you mean wrapping native widgets, this wouldn't solve much
| - you would still need some language to take care of the
| logic, like a JavaScript engine. At this point just using
| Electron is simply easier for devs, and as much as we hate
| it, realistically speaking it's still better than nothing.
| dvfjsdhgfv wrote:
| If only they had made Delphi Community Edition available a
| decade earlier...
| phendrenad2 wrote:
| Qt is still around and doing well. Some people still need
| desktop apps.
| ahepp wrote:
| >C++Builder with TwineCompile is a powerful productivity solution
| for multi-core machines compiling 1 million lines of code very
| quickly and can work better than the MAKE/GCC parallel
| compilation Jobs feature due to it's deep IDE integration
|
| You're claiming this plugin has deeper IDE integration than
| `make`? I find that really, really difficult to believe. And if
| it's true, it seems like the solution is to either use a better
| IDE, or improve IDE support for the de facto standard tools that
| already exist, as opposed to writing a plugin for the -j flag.
| klodolph wrote:
| Make forces you to choose between being able to do full
| parallel builds or using recursive make, you can't do both.
| StillBored wrote:
| I might be misunderstanding something, but the common gnumake
| does no such thing.
|
| https://www.gnu.org/software/make/manual/html_node/Job-
| Slots...
| ChuckMcM wrote:
| It may be a Windows thing. I too started reading the post
| and was thinking, why not -j128 or -j64 depending on if HT
| was on and then realized that the author's system wasn't
| one that had been tuned for decades to build software as
| quickly as possible :-).
|
| It would be an interesting academic exercise to create a
| setup on Linux that did the same thing but using the extant
| build tools, and then to do a deep dive into how much of
| the computer was actually compiling code and how much of it
| was running the machinery of the OS/IDE/Linkages. A LONG
| time ago we used to do that at Sun to profile the
| "efficiency" of a machine in order to maximize the work it
| did for the user per unit time, vs other things. That sort
| of activity however became less of a priority as the CPUs
| literally became 1000x faster.
| archi42 wrote:
| We're running our builds and tests on both Windows 10 and
| Linux. At least for Windows 7, the compiler start up time
| had a huge impact on total run time. I think back then we
| had 2500 tests, each compiled, then linked, and then
| dwarfdumped and then some more tools (we build a
| compiler). We moved some stuff to Linux (eg dwarfdump)
| and saved a few program executions by compiling & linking
| single c file tests in one step (yeah, the driver still
| calls everything, but the driver is called once less). I
| was under the impression Windows 10 improved on that (we
| didn't benchmark in detail since then, since it's still
| far enough), but on Linux I don't have to bat an eye on
| eg make -j32.
|
| BTW, the author messed up anyway. Make -j does schedule
| everything at once. How do I know? Only way I got my
| private 128GB build host to oom...
| klodolph wrote:
| I didn't go into great detail of the reasons, but the
| jobserver doesn't address the problem except for the most
| trivial cases--the core problem is that you can't cross to
| a recursive make invocation via multiple edges.
|
| This is fairly common in larger projects, so you end up
| having to do some hackery to manually sequence make
| invocations if you want to use recursive make (which is
| pretty awful).
|
| Honestly, for large projects, Make is an insane choice,
| notwithstanding the fact that people who are _sufficiently
| savvy_ at Make can sometimes make it work. (If your tools
| are bad, you can make up for it with extra staff time and
| expertise.)
| hedora wrote:
| > _the core problem is that you can't cross to a
| recursive make invocation via multiple edges._
|
| I've never had that issue, and used to heavily use
| recursive make. I carefully benchmarked those make files,
| and am sure this wasn't an issue.
|
| I suggest reading the paper "Recursive Make Considered
| Harmful" before attempting to use make for a large
| project. It describes a lot of anti-patterns, and better
| alternatives.
|
| I've found every alternative to make that I've used to be
| inferior, and they almost always advertise a "killer
| feature" that's a bad idea, or already handled better by
| make. It's surprising how many people reimplemented make
| because they didn't want to RTFM.
|
| Anyway, the next build system on my list to check out is
| bazel. Some people that I've seen make good use of make
| say it's actually an improvement.
| klodolph wrote:
| > I've never had that issue, and used to heavily use
| recursive make. I carefully benchmarked those make files,
| and am sure this wasn't an issue.
|
| So, you've never had two different targets that depend on
| something in a subdirectory? If you've just solved this
| by building subdirectories in a specific order, or
| building entire subdirectories rather than the specific
| targets you need, what you're really doing is making an
| incorrect dependency graph in order to work around make's
| limitations. These kind of decisions make sense for full
| builds, but interfere with incremental builds.
|
| Bazel & family (Buck, Pants, Please, etc.) are among the
| few build systems that solve this problem well. It's not
| an accident that they all use package:target syntax for
| specifying targets, rather than just using a path,
| because this allows the build system to determine exactly
| which file contains the relevant build rule without
| having to probe the filesystem.
|
| I would love to simply recommend Bazel but the fact is
| there is a bit of a barrier to entry depending on how
| standard / nonstandard your build rules are and depending
| on how you think that third-party dependencies should be
| pulled in. Depending on your project, you could convert
| to Bazel in an hour just by looking at a couple examples,
| or you could have to dive deep into Bazel to figure out
| how to do something (custom toolchains, custom rules,
| etc.)
|
| As you observed, the alternatives to make are often
| inferior, and it's often because they're solving the
| wrong problems. For example, sticking a more complex &
| sophisticated scripting system in front of make.
| ChuckMcM wrote:
| That is a really excellent observation.
|
| My experience is that "larger projects" to the investment
| in their build infrastructure to maximize parallelism in
| order to reduce build times because it pays dividends in
| terms of turn around time and thus productivity.
|
| One of the joys of working with someone who has been
| building large projects for a while, is that they just
| design the build system from the start to be as parallel
| as practical.
| StillBored wrote:
| Make isn't great, but if you look at the opensource world
| the vast majority of the large projects are make based.
| The backbone of your average distro is a thin layer on
| top of what is mostly automake/autoconf in the individual
| projects. That is because while I can create a toy
| project that builds with make in a half dozen lines, big
| projects can extend it to cover those odd edge cases that
| break a lot of other "better" build systems.
| Particularity, when a project starts including a half
| dozen different languages.
|
| So, while i'm not a make fan, I'm really tired of people
| pissing on solutions (c also comes to mind) that have
| been working for decades because of edge cases or
| problems of their own creation because they don't
| understand the tool.
|
| A well understood tool is one where people know where the
| problems lie and work around them. Most "perfect" tools
| are just project crashing dragons hiding under pretty
| marketing of immature tooling.
| klodolph wrote:
| > ...I'm really tired of people pissing on solutions (c
| also comes to mind) that have been working for decades
| because of edge cases or problems of their own creation
| because they don't understand the tool.
|
| You would have to be extraordinarily cynical to think
| that there's been no progress in making better build
| software in the past forty years or so.
|
| Yes, there are definitely plenty of build systems out
| there that were built from scratch to "fix" make without
| a sufficient understanding of the underlying problem. I'm
| not going to name any; use your imagination. But make was
| never _that good_ at handling large projects to begin
| with, and it was never especially good at ensuring builds
| are correct. This is not "pissing on make", for Chris's
| sake, make is from 1976 and it's damn amazing that it's
| still useful. This is just recognizing make's
| limitations, and one of those limitations is the ability
| for make to handle large projects well _in general,_
| specific cases notwithstanding. Part of that is
| performance issues, and these are tied to correctness. As
| projects which use make grow larger, it becomes more
| likely that they run into problems which either interfere
| with the ability to run make recursively or the ability
| to run make incrementally.
|
| The "poor craftsman who blames his tools" aphorism is a
| nice way to summarize your thoughts about these kind of
| problems sometimes, and it does apply to using tools like
| make and C, but it's also a poor craftsman who chooses
| tools poorly suited to the task.
| zajio1am wrote:
| Why would you do recursive make? That is setup discouraged
| for decades ...
| ahepp wrote:
| (for reference: "Recursive Make Considered Harmful",
| https://accu.org/journals/overload/14/71/miller_2004/)
| klodolph wrote:
| Because it can be hard to maintain non-recursive make
| systems for large projects. Just because recursive make is
| discouraged does not mean that the alternative is without
| its own drawbacks.
| barkingcat wrote:
| there's something much easier to bring 64 cores to its knees -
| chromium takes a loooong time to compile.
| formerly_proven wrote:
| [Not "real" C++ code, benchmark is for compiling 14492754 copies
| of a fairly simple C function]
| jandrese wrote:
| Seems like he tried more complex examples too, but ran into
| roadblocks like a 2GB limit on executables and running into a
| commandline length limit restriction that dates back to early
| DOS days which made it impossible to link.
|
| Both of those problems seemed solvable if he was willing to
| chunk up his application into libraries, maybe 1024 files per
| library then linked to the main application.
| simcop2387 wrote:
| I believe this is one of the reasons for object libraries (or
| archives, the foo.a files on linux/unix), you can then link
| in all of the object files from one of those at link time
| without having to list them all at once. That won't get past
| the 2GB limit on executables but it will get past the command
| line length.
| com2kid wrote:
| This is correct, a .lib file on Windows has a bunch of .obj
| files in it that you can then link together.
|
| You can also use command files[1] to pass options in
| instead of using the command line.
|
| [1] https://docs.microsoft.com/en-
| us/cpp/build/reference/linking... to pass
| [deleted]
| jandrese wrote:
| I was going to suggest that, but he's running on Windows
| and I don't know if they are supported there. I guess they
| probably are since they're a compiler feature.
| jpaul23 wrote:
| Does there exist some kind of random C code generator?
| solinent wrote:
| > 1B Lines of C++
|
| Seems like our code is inflating quite rapidly. I remember when
| 1M was the biggest project. /snark
| renewiltord wrote:
| Hahaha, fuck me, CPUs are _fast_. That 's wicked. 15 mins. A
| billion lines of C. Insane. Wonder if there's some IO speed to be
| gained from ramdisking the inputs.
| titzer wrote:
| 17,300 lines/sec per core. That's embarrassingly slow IMHO.
| bjoli wrote:
| That depends completely on what optimizations are being done.
|
| But alas, I have said for some time that a fast compiler
| should be able to compile about 1MLOC/S with some basic
| optimization work.
| Macha wrote:
| It is pointed out that the threadripper does worse per core
| when under full load than even high core count consumer CPUs
| like the 3950x/5950x. That's the tradeoff you make for huge
| core count CPUs. 4x 3950x might do better, but then you need
| to build 3 other PCs, and for actual processing tasks, co-
| ordinate stuff to run across multiple systems.
| bserge wrote:
| What can perform better?
| mhh__ wrote:
| I can build the D compiler (500k lines?) Warts and all in a
| second in my machine - and that's code that's not
| particularly complicated, but not at all optimized for
| compile times realistically.
| titzer wrote:
| Lots of single-pass compilers can achieve 1MLOC/s. But the
| main problem is that C++ has an O(n^2) compilation model
| due to header explosion. Also, expanding C++ templates is
| very computationally intensive.
| viktorcode wrote:
| Other languages do perform better
| Cyph0n wrote:
| As a huge Rust fan: not Rust :)
| josephg wrote:
| Go, Jai, V8 (weirdly), some hobbyist C compilers.
| moonchild wrote:
| V8 is a JIT compiler, which has a baseline interpreter
| and progressive optimization. It's not surprising that
| interpreters have fast startup time. (Hotspot is the
| primary exception in this case, but a big part of its
| slowdown comes from the fact that it verifies bytecode
| before loading it.)
| titzer wrote:
| Bytecode verification is not a large part of compilation
| time for Java. For HotSpot server, verification is
| dwarfed by building the IR, repeatedly walking and
| transforming it, generating code, and register
| allocation. Keep in mind that HotSpot will very likely
| inline a ton of methods, making the compilation unit
| large. Many passes are at least slightly superlinear, so
| that makes compilation time explode.
|
| That said, I ported C1 (the client compiler) to Java back
| in my Maxine VM days, about 2009. At that time on typical
| x86 processors it was easily hitting 1.8MB/s (of
| bytecode) compilation time. You can work backward to how
| many lines of code a second that is (and sure, you aren't
| paying the cost of the frontend), but yeah, that is in
| the ballpark for 1MLOC/s I think. And that's with an
| optimizing compiler with IR and regalloc!
| josephg wrote:
| I wonder if it'd be possible / useful to implement a rust
| or C compiler that worked that way. Basically instant
| startup time after a code change is a delightful feature.
| moonchild wrote:
| I've been bandying around ideas lately for an extremely
| fast c compiler that, among other things, caches machine
| code for individual function declarations. You wouldn't
| realistically be able to get to the level of speed as js,
| though, because js is a dynamic language. With a
| statically typed language like c or rust, any time you
| change a declaration, you have to re-typecheck everything
| that depends on that declaration. (Though you can
| frequently avoid generating new code, if you implement
| structs using hashtable semantics.) Both languages also
| have a concept of compile-time evaluation, and rust in
| particular has to deal with template expansion (which is
| not trivial, I hear they use an SMT solver for it now).
| steveklabnik wrote:
| Rust does not use an SMT solver.
| czbond wrote:
| 1B Lines? And this is just from a "rails new" command. Had to for
| some levity.
| hikerclimber wrote:
| hopefully the virus continues evolving ever second all over the
| world so we can never have any cure and it never stops. hopefully
| billionaires lose all there money.
| realolokunmama wrote:
| Hello!! I hope my contact brings you inspiration and joy to your
| day! As you can see, I'm a spiritual spell caster i have herbal
| cure for CANCER and more i also cast spells such as Health
| Restoration Spell Dismiss Depression Spell Love spell Ex Back
| spell Lost Love spell Attractive spell Divorce spell Financial
| spell Promotion spell Marriage spell Protection spell pregnancy
| spell Job spell Grow your business spell Lottery spell Fertility
| spell Court Case spell Diabetes Lupus kindly contact me And don't
| forget that problem shared is a problem solved. Email:
| realolokunmama12@gmail.com
| muststopmyths wrote:
| Interesting. It would be cool to compare this against Visual
| Studio + Incredibuild, in my experience the most solid
| distributed C++ compilation tool.
| trhway wrote:
| Lucky sons of gun. We are stuck with Xeons. Have to wait 3 hours
| for our 20M C/C++ on the 2x14cores Xeon machine after a
| pull/rebase. Ryzen/TR would probably be faster 2-3x times for the
| same money, yet it is a BigCo, so no such luck (and our product
| is certified only for Xeons, so our customers can't run AMD too -
| thus we're de-facto part of the Great Enterprise Wall blocking
| AMD from on-premise datacenter).
| AshamedCaptain wrote:
| Industrial software will always grow in size to use all
| available compilation time ... I have seen large Xeon distcc
| farms and the total build walltime was still measured in
| hours...
| maccard wrote:
| I upgraded from 2x 12 core xeons to a 64 core thread ripper -
| compile times dropped from 45m to 12m
| gm wrote:
| That article mentioned Delphi and Object Pascal, and it brought
| back many fond memories. I absolutely LOVED Delphi and Object
| Pascal back in the day. So clean and so fun to program in. If
| Borland hadn't f-ed it up and had stayed around until now, I'd be
| the biggest Delphi fanboy.
|
| Alas, that was not to be. Modern languages are fun and all, but
| not Delphi-back-in-the-day level fun :-).
| nick__m wrote:
| 2 actively maintained version of Delphi still exist, the
| original one maintained by Embarcadero, and an open-source one
| available at https://www.lazarus-ide.org/ .
| einpoklum wrote:
| A Billion lines, eh? int main ()
| { /* _______ _ _ _ _
| |__ __| | (_) (_) | |
| | | | |__ _ ___ _ ___ __ _ | | ___ _ __ __ _ _ __ _
| __ ___ __ _ _ __ __ _ _ __ ___ | | | '_ \| / __|
| | / __| / _` | | |/ _ \| '_ \ / _` | | '_ \| '__/ _ \ / _` |
| '__/ _` | '_ ` _ \ | | | | | | \__ \ | \__ \ | (_| |
| | | (_) | | | | (_| | | |_) | | | (_) | (_| | | | (_| | | | | | |
| |_| |_| |_|_|___/ |_|___/ \__,_| |_|\___/|_| |_|\__, | |
| .__/|_| \___/ \__, |_| \__,_|_| |_| |_|
| __/ | | | __/ |
| |___/ |_| |___/ */
| return 0; }
| dboat wrote:
| After liking this article, I wanted to check out others on the
| site, and am shocked at the terrible usability of their front
| page. I can't finish reading the titles of their articles before
| the page just keeps moving things around on me. It is so
| frustrating, which is unfortunate because I would otherwise have
| been interested to see more of their content. Experience
| completely ruined by awful design judgment.
| dboreham wrote:
| They finally got around to reusing mainframe model numbers.
| zelly wrote:
| On Linux I would just use Bazel. It can burn through 1B lines of
| code on all cores.
___________________________________________________________________
(page generated 2021-02-03 23:00 UTC)