[HN Gopher] Large Unix programs were historically not all that p...
___________________________________________________________________
Large Unix programs were historically not all that portable between
Unixes
Author : zdw
Score : 66 points
Date : 2021-09-02 05:18 UTC (2 days ago)
(HTM) web link (utcc.utoronto.ca)
(TXT) w3m dump (utcc.utoronto.ca)
| truenindb wrote:
| POSIX programs were portable enough that Microsoft had to become
| POSIX compliant before being granted the operating system
| monopoly rent by the free market.
| pjmlp wrote:
| "Portable" until you rely on signals behaviour, file systems,
| threading, file locking and other stuff left for the
| implementations.
|
| Windows NT POSIX subsystem was for ticking check box on DoD
| contracts.
|
| Had Microsoft been half serious about POSIX in NT like they are
| nowadays with WSL, Linux would never had taken off.
| billsmithaustin wrote:
| In the 90's I worked for a company whose C/C++ code needed to run
| on at least 7 different Unix versions: SunOS, Solaris, two
| different AIX versions, HP-UX, DGUX, and whatever Motorola's Unix
| was. The slogan around the office was, "There is no such thing as
| portable code, just ported code."
| toolslive wrote:
| A bit more than a decade ago, I was on the receiving end of a C
| level (CEO/CTO that is) decision to target Linux, BSD, Solaris
| and ... Windows, for a C++ application.
|
| I think I can imagine your pain.
| drewg123 wrote:
| In the 2000s, I worked for a company that shipped an HPC NIC
| (think precursor to infiniband). We had userspace software _AND
| DEVICE DRVIVERS_ that supported Linux, FreeBSD, Solaris, DEC
| OSF /1 (Tru64), AIX, Windows and Mac OSX.
|
| The device driver code had a shared component that supported
| the hardware, via an OS abstraction layer over top of the
| various OS specific ways to do things. The driver did fun
| things, like pin userspace memory for RDMA, which was often
| outside of supported driver interfaces.
|
| The Windows, and the closed source commercial UNIXes were a
| PITA to bring up initially (AIX was the WORST). But once
| supported, they were easy to maintain because the interfaces
| never changed. Linux was fairly easy to write, but a PITA to
| maintain because the kernel interfaces changed so much. We had
| a configure style shell script to probe the kernel headers and
| detect how many args certain functions took on this particular
| linux version, etc. That script by itself was _larger_ than the
| entire FreeBSD driver. (and FreeBSD was as easy as linux to
| write, and as easy as a commercial UNIX to maintain, since the
| interfaces we used didn 't change).
|
| One interesting thing that fell out of it is that we used ioctl
| interfaces everywhere (even Windows and MacOS). Since this was
| an HPC device, latency was critical and the Mac ioctl routines
| were painfully slow (like 4x slower than Linux/FreeBSD on the
| same hardware). So I tried switching to the Mach based IOKit
| IPC that Apple wants you to use, and that was twice as slow as
| ioctls!
| pinko wrote:
| A couple decades ago I ran an NSF-funded project to develop and
| run a public, open-source automated built-and-test platform for
| ~50 distinct versions of a dozen or so unix OSes (plus a token
| Windows version or two), long before modern CI platforms made
| this commonplace. We needed it for our own software and
| "productized" it for the wider scientific community. It was a ton
| of fun, and a beast. The diverse hardware in our test lab was
| like a zoo. Linux consolidated the industry and made the whole
| thing unnecessary shortly after I left.
| dboreham wrote:
| Historically? What changed? Everything is Javascript now? There's
| only one Unix now?
| tyingq wrote:
| >There's only one Unix now?
|
| Not quite that, but there are certainly a lot fewer with
| notable amounts of use. Which means, in turn, fewer compilers
| with unique flags...you can get away with supporting just
| gcc/clang.
|
| And there are fewer machines that are big endian, or have
| unusual byte alignment, or have a buggy built-in malloc, or
| don't have off_t defined, or have unusual flags to link to
| pthreads. And so on...all the stuff things like Configure test
| for.
| rwmj wrote:
| Portability libraries like glib are better at hiding the
| differences, at least for simpler C/C++ programs. Also (sorry
| BSDs), there is kind of only one "Unix", which is Linux. Even
| if you consider Linux + BSDs + macOS, that's a lot fewer
| combinations than in the good old days when you might have
| considered a port to Solaris, HP-UX, SCO, AIX, OSF/1, ... all
| of which were very different.
|
| Porting programs to Windows still remains a challenge.
| tyingq wrote:
| >Solaris, HP-UX, SCO, AIX, OSF/1
|
| Just to make the problem more clear, add:
|
| Irix, Ultrix, SunOS, DG/UX, Pyramid OSX, NCR Unix, Xenix,
| A/UX, Bull DPX, EP/IX, ISC, MachTen, NeXT, Dynix, UnixWare,
| ConvexOS, Coherent, DomainOS and more...
|
| See the supported platforms for something like Kermit or Pine
| to see this isn't just piling on. Some real and popular
| software had to support huge lists like this. Perl's CPANTS
| testing network was neat for stuff like this, you could write
| a module (including C code), and watch all the various ways
| your code would bomb on other platforms without having direct
| access to them.
|
| http://www.columbia.edu/kermit/unix.html
|
| http://www.mit.edu/afs.new/athena/astaff/source/src-9.0/thir.
| ..
| gpderetta wrote:
| Autoconfig exists for a reason!
| toolslive wrote:
| it works pretty well if you stick to C and target Linux and
| want to support a multitude of distributions. As soon as you
| throw something else into the mix (C++, or OSX or ...) you're
| heading for a world of pain.
| masklinn wrote:
| Sounds like you're saying it doesn't work?
|
| But as the essay notes:
|
| > they gave you the tools that you could use to sort out how
| to achieve that, and to automatically detect various things
| you needed to do to adopt to the local Unix
|
| autoconf doesn't do much on its own, it's more of a toolbox,
| if you don't leverage the tools it _gives you_ , you won't
| achieve any sort of significant portability.
| toolslive wrote:
| No. It works, but it's painful. Let's say m4 and sh are not
| the most comfortable tools to work with. These days, there
| are better ways. (CMake or Scons fe)
| pjmlp wrote:
| I would say the not popular opinion around here that
| applies to most tools on UNIX, and I have used plenty of
| flavours since 1993, when I got introduced to it via
| Xenix.
| b0rsuk wrote:
| Isn't it true for all programs? Not just _unix_ programs?
|
| Firefox is portable and it's a large program, but a LOT of effort
| went into that.
| masklinn wrote:
| > Isn't it true for all programs? Not just _unix_ programs?
|
| That's irrelevant whataboutism.
|
| TFA is very specifically in response to one pining about older
| unix ecosystems and seeing them as havens of portability, and
| explains that that was not the case at all.
| 0xbadcafebee wrote:
| I don't know who ever thought Unixes were portable? None of
| them worked the same... If you were lucky maybe they'd be
| running the same architecture.
| [deleted]
| masklinn wrote:
| Are commenters on this site functionally unable to click
| links anymore or something? Literally the first link in
| TFA's body text is the take they're replying to.
| jhgb wrote:
| > seeing them as havens of portability
|
| Was that in absolute, or relative terms? Since portability
| between OS/360 and TOPS-10, for example, was non-existent.
| TheOtherHobbes wrote:
| Not only. The PDP-11 had 5 (6? 7?) different and
| incompatible OS options available for the same instruction
| set and hardware.
|
| Which - of course - varied somewhat different across
| different models.
| cratermoon wrote:
| I built rn on a number of different systems at the time, and I
| was fascinated by the configure/build's messages while it tried
| to figure out what os & hardware it was running on.
|
| Also "This program posts news to thousands of machines throughout
| the entire civilized world. You message will cost the net
| hundreds if not thousands of dollars to send everywhere. Please
| be sure you know what you are doing."
| smackeyacky wrote:
| I was kinda excited when I first saw autoconf. In the "good old
| days" porting software that academics wanted on the local Unix
| box was a genuine exercise. When you see all the differences
| that the "configure" command comes up with it gives a little
| twinge of nostalgia for when even things like the version of
| libc available wasn't in the expected place. Trying to get
| stuff running on an Apollo or a DEC 3100 or a Pyramid was
| fascinating and frustrating at the same time.
| bloat wrote:
| I remember this too. First year of university in 1991, me and
| my friends who'd all gone to different places used to write
| actual letters on paper and post them to each other. I was on
| the internet for the first time, and was excited to be able
| to download pictures! I could print them out and include them
| in the letters. If I remember correctly I needed to compile
| some kind of image viewer/manipulator on my account on the
| DEC Mips machine we CS students had access to. Anyway, as you
| say it was amazing watching the autoconf script feeling its
| way around and discovering its environment.
| ggm wrote:
| I wasn't. I have always felt autoconf was the wrong solution
| because it gives in to the problem.
|
| Uname -a having a build hash, and an index of hash to
| settings would have done better. Think of all the wasted CPU
| cycles computing how this chip orders a long long and testing
| POSIX compliance one function at a time.
|
| Autoconf only fixes the apparent problem. Not the underlying
| disease.
|
| At least X11 tried encoding the known patterns.
| Hello71 wrote:
| > uname -a
|
| so whenever you upgrade the kernel or change CPU all your
| programs need manual patching? you may answer, "well uname
| -a was a joke, you would actually check the libc version".
| this worked alright in a very specific time period with a
| couple of big vendors with nigh-unmodifiable toolchains
| with well-defined release versions. it became increasingly
| awful when people started making their own toolchains with
| free software and had to patch every single piece of
| software to properly detect their new libc. sitting through
| an hour of autoconf sucks, but slogging through ten hours
| of patching poorly written bespoke configure scripts is a
| hundred times worse.
|
| edit: not to mention cross-compiling where you can't even
| _run_ uname -a, and which was almost universally broken
| before autoconf, which, despite being painful, made cross-
| compiling _mostly_ work everywhere.
| R0b0t1 wrote:
| Hindsight is 20/20.
| graphviz wrote:
| It's not that it can't be done, it's that the community
| went in a different direction, and there's a strong
| cultural bias. Bell Labs internally had a "new make"
| written by Glenn Fowler, that went full bore into
| tracking dependencies, probing local and global
| resources, recording configuration state at build time,
| that we adopted in Graphviz. It was powerful and concise.
| It was also a heroic effort for Glenn to maintain and
| develop, though others were assigned.
| https://ieeexplore.ieee.org/document/6771042
|
| At one point in our project we had a lot of arguments
| about build systems and software repositories (there were
| times when we had to stop talking about it), then at some
| point John Ellson said, look, everyone else is using
| autotools, so do you want them to use your software? We
| realized he was right.
|
| Individuals in the software community are capable of
| building a lot of amazing tools. The difficulty is
| persuading the community and teaching non-specialists to
| use them. They already know how to run make, or
| ./autogen.sh; ./configure --prefix=$HOME
|
| Despite this Graphviz still has 3 or 4 build systems:
| autotools, Visual Studio (probably 90% of our audience is
| on Windows though we try not to think about it), Xcode
| (needed for an under-maintained MacOS app), and Magnus
| Jacobsson and Matthew Fernandez have a lot of a Cmake
| build working, though not yet complete.
|
| Apologies for rambling a little off topic.
| R0b0t1 wrote:
| That's true, now we can go back and simplify, I was more
| speaking to the origination of all of these horrible
| things. Nobody knew when they picked 2048 instead of 1024
| it was going to matter, but then suddenly it did and
| nobody wrote down a good way to probe that API.
| vidarh wrote:
| If there's one thing you never have to apologise for on
| HN it's going off topic into trivia like this.
| jefftk wrote:
| That would move a lot of work onto whoever first stumbled
| against a new combination. The magic of autoconf, when it
| works correctly, is that you don't have to have a
| configuration that has been seen before.
| smcameron wrote:
| I can't count how many times I end trying to run autoconf
| only for it to tell me I have the wrong version of
| autoconf.
| jijji wrote:
| The word "portable" refers to the distinction of using assembly
| language programming versus a language like C, which with little
| modification can compile clean on the target OS...
|
| I agree that to take a program from one unix and make it work on
| another is a tedious task, but the difference is minimal when you
| compare portability of architecture specific assembly language
| code. It is the reason "portable" languages like C were created.
| pjmlp wrote:
| Of course not, POSIX portability leaves lot undefined across UNIX
| platforms, it was mostly the C runtime that ANSI/ISO C didn't
| want as part of the standard than anything else.
|
| Back in 1999 - 2003 I had lots of fun porting code between AIX,
| Solaris, HP-UX and Red-Hat Linux.
|
| A decade earlier the fun was with Xenix, DG/UX, and early
| GNU/Linux flavours.
|
| And it wasn't only the OS specific APIs, each OS SDK had their
| own glitches and deviations from C standard.
| ToddWBurgess wrote:
| Reading this reminds me of the days in the 90s when programs
| could target 10+ different Unix systems and shipped as giant tar
| balls of code. The C code consisted of giant #ifdef <UNIX VENDOR>
| blocks to accommodate how different Unix systems might handle a
| couple system calls.
|
| It wasn't uncommon for the wrong block of code to get included so
| you had to figure out how the compile target's system call
| worked, how your system's system call worked and modify the code
| accordingly to get it to compile (and hope it worked). This was
| the 1990s so getting code to compile on Linux wasn't a high
| priority (since Linux was still in its infancy).
|
| Fun times.
| tyingq wrote:
| The message if you were running Eunice:
|
| https://github.com/Perl/perl5/blob/perl-4.0.36/Configure#L10...
| grymoire1 wrote:
| I remember one C book that described a way to write portable
| code. There were three phases in the header files: (1) The
| vendor, (2) The characteristics, (3) a portable layer on the
| system calls. So the first would define VAX, SUN, etc. The second
| would use the first section to define LITTLE_ENDIAN, BIG_ENDIAN,
| HAS_IOCTL, etc. And the third would implement a portable layer on
| top of the system calls. It was nice to compile because section 1
| would often be defined automatically by the pre-processor, and
| the rest would just be defined properly. No config was needed to
| compile - you just types "make".
|
| Alas - this was extra work on the developer, and at the time
| there were a zillion variations, which a developer had to deal
| with.
___________________________________________________________________
(page generated 2021-09-04 23:02 UTC)