[HN Gopher] Windows NT: Peeking into the Cradle
___________________________________________________________________
Windows NT: Peeking into the Cradle
Author : jmmv
Score : 85 points
Date : 2023-11-27 06:05 UTC (1 days ago)
(HTM) web link (blogsystem5.substack.com)
(TXT) w3m dump (blogsystem5.substack.com)
| jmmv wrote:
| Original author here.
|
| Back in September, I got the recommendation to read the
| "Showstopper!: The Breakneck Race to Create Windows NT and the
| Next Generation at Microsoft" book from some other comment thread
| here in HN. I don't exactly remember which thread that was
| though... but learning about the history of Windows NT sounded
| interesting, so I bought the book right away and have been slowly
| reading through it over the last three months.
|
| The story in the book seemed so exciting (and messed up) that I
| ended up writing a "book review" with commentary on various
| topics that caught my attention. The details might not be super-
| precise, but they are hopefully enough to paint the right picture
| in its context. I thought you might enjoy it due to where I got
| the suggestion from!
| HankB99 wrote:
| Is the book labelled as fiction?
|
| I understand that early versions of NT emitted error messages
| identifying itself as OS/2. Did the book explain when NT became
| "brand new OS from scratch".
|
| I think there's been a bit of revisionist history WRT the roots
| of NT. MS wants to make it something "we did without their
| help."
|
| I didn't read your entire blog so apologies if you addressed
| this.
| jmmv wrote:
| My understanding is that the original collaboration between
| Microsoft and IBM was around certain parts of OS/2, not the
| whole system, so it is plausible that Microsoft worked
| simultaneously on OS/2 and NT as the book describes. Note
| that NT is heavily based on the work previously done at
| Digital on VMS (by people from the same team), so there are
| some pre-existing foundations there, at least in terms of
| design principles and ideas.
|
| https://en.wikipedia.org/wiki/OS/2 provides a recap of the
| history but I cannot find details on exactly who did what in
| that page. And my memory from what I read in the book is
| fuzzy at this point...
| dboreham wrote:
| It's been a few decades since I read the book, but note it
| was published just after NT was released, when everyone knew
| that it had been called OS/2-NT for a while. So probably no
| wool pulling going on there. Note also that this was a naming
| move. NT was never _based_ on OS /2. otoh in my experience
| (I've worked for two companies that had books written about
| them), book authors tend to a) focus on drama and b) get
| their information from a small number of senior participants.
| trealira wrote:
| The book doesn't shy away from describing the collaboration
| between Microsoft and IBM. Also, the book depicts Microsoft
| knowingly screwing over IBM in their collaboration.
|
| _So Gates sided with Maritz. NT would assume the Windows
| "personality," by assuming those of its attributes that were
| visible to the customer. NT also would be redesigned to run
| applications written specifically for Windows. But Gates
| added a wrinkle to his position. Given the depth of IBM's
| attachment to OS /2, Gates reasoned, abandoning the program
| altogether would only trigger a nasty split with IBM. Why not
| maintain a pretense of cooperation by convincing IBM of the
| merits of an operating system that hosted both Windows and
| OS/2? NT could be designed to do this, Gates reasoned._
|
| _If IBM accepted this, Microsoft could continue to describe
| Cutler's program as "NT OS /2." Even though it contradicted
| his rhetoric about the need for a single software standard,
| Gates was prepared to tell IBM: "There's no reason to think
| two [program personalities] can't be of equal importance" to
| customers._
|
| _This was a bold stroke. By denying reality, Gates's tactic
| would freeze IBM long enough for Microsoft to install Windows
| as the standard for computing. Gates would peddle this
| apparent compromise to IBM and rival software makers, while
| Maritz and Cutler's team pursued the real objective
| internally._
|
| ...
|
| _Cutler asked about Ballmer's promise that NT would still
| sport an OS /2 personality. Not to worry, Ballmer assured
| him, someone else could take care of that. Indeed, OS/2 was
| now so unimportant to Microsoft that Ballmer had assigned a
| tiny team in Israel to create an OS/2 variant of NT. The
| Microsofties hated OS/2 so much they couldn't even bear to
| have someone working on it in the same continent._
| voidfunc wrote:
| Dave Cutler has elaborated on a lot of this stuff in some
| recent YT interviews that are excellent:
| https://www.youtube.com/watch?v=9K3eMzF6x28
|
| Part of it is Cutler really really hated working with IBM...
| it starts around 3:00 minute mark.
| canucker2016 wrote:
| Several MSFT employees I talked to mentioned that they
| didn't like working with IBM - too much of a culture clash
| from what I could gather.
| EvanAnderson wrote:
| An earlier comment mentioned Gordon Letwin[0] (lead
| architect of OS/2 at MSFT), and that led me down the
| rabbit hole to an "infamous" USENET post he made re:
| OS/2[1].
|
| It sounds like there was acrimony all around:
|
| _Another problem that came up here was that IBM didn 't
| want us to use the windows API for the graphical
| environment under OS/2. Many key folks inside IBM had
| always hated Windows. IBM had this crazy thing called
| TopView, it was a character oriented windowing scheme and
| not very good. Bill Gates, myself, and some other folks
| made several trips to Boca Raton to try to explain to
| those guys why a character oriented windowing scheme was
| obsolete before it was even written, but to no avail. One
| of IBM's most major problems is that although their top
| guys may be smart, they aren't techically savvy._
|
| [0] https://en.wikipedia.org/wiki/Gordon_Letwin
|
| [1]
| http://gunkies.org/wiki/Gordon_Letwin_OS/2_usenet_post
| canucker2016 wrote:
| You cut off the best bits of Letwin's rant:
|
| "...although their top guys may be smart, they aren't
| techically savvy. And their low level guys are often
| neither. IBM doesn't promote on the basis of your skills
| and ability; they promote on the basis of seniority and
| other secondary factors. So the guy who makes these
| decisions often doesn't know what he's doing. And he
| doesn't know that he doesn't know, because his peers are
| equally butt-ignorant too. So these guys can never figure
| out how other folks, including but not limited to
| Microsoft, keep doing better! Must be dumb luck, they
| think. I always agreed that it _was_ dumb luck. If you
| catch my drift... :-) "
| chasil wrote:
| The NT kernel, a rewrite in C of the VMS kernel (which itself
| was written in assembler), presented three APIs to userland:
| Win32, POSIX, and OS/2. It seems reasonable that the OS/2
| emulator/API/userland inside of early Windows would identify
| itself as truly OS/2, which it was likely seen to be
| internally.
|
| The VMS kernel itself would also emulate programs written for
| the earlier RSX-11 operating system that ran on the PDP-11,
| so there is a history of emulation between these operating
| systems.
|
| "Broad software compatibility was initially achieved with
| support for several API 'personalities', including Windows
| API, POSIX, and OS/2 APIs - the latter two were phased out
| starting with Windows XP...
|
| "Mark Russinovich wrote in 1998 that while creating Windows
| NT, Microsoft developers rewrote VMS in C. Although they
| added the Win32 API, NTFS file system, GUI, and backwards
| compatibility with DOS, OS/2, and Win16, DEC engineers almost
| immediately noticed the two operating systems' internal
| similarities; parts of VAX/VMS Internals and Data Structures,
| published by Digital Press, accurately describe Windows NT
| internals using VMS terms. Instead of a lawsuit, Microsoft
| agreed to pay DEC $65-100 million, help market VMS, train
| Digital personnel on Windows NT, and continue Windows NT
| support for the DEC Alpha."
|
| https://en.wikipedia.org/wiki/Windows_NT
|
| Information on the Windows POSIX subsystem:
|
| https://en.wikipedia.org/wiki/Microsoft_POSIX_subsystem
| mschaef wrote:
| > Is the book labelled as fiction?
|
| > I understand that early versions of NT emitted error
| messages identifying itself as OS/2. Did the book explain
| when NT became "brand new OS from scratch".
|
| Windows NT started out as Microsoft's effort to do OS/2 NT.
| There was a 32-bit Microsoft OS/2 2.0 (distinct from IBM OS/2
| 2.0), and OS/2 NT was to be OS/2 3.0.
|
| OS/2 NT wasn't rebranded Windows NT until Windows 3.0
| eliminated OS/2 from relevance, and with it, the
| Microsoft/IBM partnership. That's when IBM introduced their
| own (very nice) OS/2 2.0 and started a years long effort to
| eliminate Microsoft code from the OS/2 codebase (and the
| Microsoft copyright message from all over the UI)
| borissk wrote:
| The part about NTFS is slightly incorrect. It wasn't really a
| new file system. It was a copy of High Performance File System
| (HPFS) for OS/2, that was originally implemented in assembler
| by Microsoft's best programmer Gordon Letwin. NTFS and HPFS
| even used the same partition id.
| jmmv wrote:
| Oh thanks. I must have forgotten about that; if you have read
| the book, do you know if this was explained? Will need to
| read more and try to update the text!
| jvert wrote:
| NTFS was definitely a new filesystem and supported many
| capabilities not in HPFS. Windows NT also shipped with HPFS
| support which was a complete reimplementation of the HPFS
| filesystem which shipped with OS/2.
| chasil wrote:
| The DEC people added some stuff.
|
| "The MFT structure supports algorithms which minimize disk
| fragmentation... While this strongly resembles the W_FID of
| Files-11, other NTFS structures radically differ."
|
| https://en.wikipedia.org/wiki/NTFS
|
| "ODS-2 is the standard VMS file system, and remains the most
| common file system for system disks (the disk on which the
| operating system is installed)... NTFS - Has many structural
| and metadata similarities with Files-11 and is almost
| certainly conceptually derived from it."
|
| https://en.wikipedia.org/wiki/Files-11
| AshamedCaptain wrote:
| They use the same partition id because that was the partition
| id used for IFSes in OS/2 (and therefore all non-DOS
| filesystems). This should not be taken as a sign of their
| similarities (or differences).
| kimixa wrote:
| Also some of the features listed (Compression, ACLs, volume
| sizes) were incrementally added over years and not available
| at first release, which somewhat invalidates the comparisons
| with other filesystems "Of the era".
|
| The NTFS of today is rather different to the NTFS of 1993.
| chungy wrote:
| NT 3.1 absolutely did have ACLs in NTFS.
| chasil wrote:
| I have just been dealing with problems of VMS this week, and I
| have Dave Cutler to thank.
|
| We have a temporary emulated VAX running VMS 7 and Rdb [a SQL
| database now owned by Oracle]. After several tries to move the
| Rdb backup files through our QA firewall with FTP, we finally
| had to use Multinet SFTP2 to push the backup instead (upping
| the transfer time from 3 to 40 minutes), which autodetected
| "VMS mode."
|
| There are some secret record blocking/delimiter capabilities in
| the ODS-2 filesystem used by VMS that an FTP binary transfer
| will not preserve, and which the Rdb backup/restore process
| absolutely needs.
|
| Thank heavens that UNIX thinks a file is a single stream of
| bytes.
|
| Windows continues the feature experimentation with NTFS
| "alternate data streams." My, how clever that is [not].
|
| EDIT: from the article, the claim "The open-source BSDs and
| Linux had just seen the light of day." is not quite correct.
| Bill Joy was involved in the earliest BSD repacking of v6
| Research UNIX, shipped in 1978.
|
| "In 1975, Ken Thompson took a sabbatical from Bell Labs and
| came to Berkeley as a visiting professor. He helped to install
| Version 6 Unix and started working on a Pascal implementation
| for the system. Graduate students Chuck Haley and Bill Joy
| improved Thompson's Pascal and implemented an improved text
| editor, ex. Other universities became interested in the
| software at Berkeley, and so in 1977 Joy started compiling the
| first Berkeley Software Distribution (1BSD), which was released
| on March 9, 1978."
|
| https://en.wikipedia.org/wiki/Berkeley_Software_Distribution
|
| ...I have also read elsewhere that a conscious decision was
| made in Windows to never, ever support big-endian architectures
| (but I am not certain that this is correct). NetBSD and Linux
| can run big-endian.
|
| "Now, I know you will complain that NTFS is slow... but you
| know what? It almost-certainly is not... NTFS on its own is
| fine..."
|
| XFS appears to trounce NTFS, and it consistently leads the
| performance scores on tpc.org; until very recently, SQL Server
| on XFS/Linux was well ahead of Windows.
| Gravityloss wrote:
| Oh my, that prompted me to read Wikipedia - and we have MUMPS
| also from VMS. A quite different programming paradigm. The
| last common ancestor with current mainstream programming is
| really far back and it looks very odd.
|
| New large scale multi billion dollar healthcare systems are
| still being written in it.
| chasil wrote:
| VMS was just recently ported to AMD64, so you might be able
| to keep using it if your vendor ports their code.
|
| https://vmssoftware.com/about/v921/
| canucker2016 wrote:
| You're blaming tools that don't support the full featureset
| of one of your platforms.
|
| VMS's ODS2 filesystem evidently has concept of files that are
| grouped in records. The ftp program that you use on VMS
| wasn't designed to support sending that info to a destination
| filesystem. Probably because the original Unix filesystem
| only supported the idea of one stream of bytes for a file.
| You'd need an FTP server on the destination side to support
| records seamlessly if you wanted full fidelity.
|
| That's why NTFS supports alternate data streams - the MacOS
| HFS filesystem supported data and resource forks for a file.
| An NTFS alternate datastream was where the resource fork was
| placed so the NTFS user would only see one file from the CLI
| and the GUI and the builtin copy tools would copy all
| streams. deleting such a file would remove both streams on
| NTFS.
| chasil wrote:
| Actually, Multinet FTP does have a couple of switches for
| "VMS mode," but SFTP2 got a working solution in front of us
| with less reading of the manuals.
|
| POSIX was wise in having absolutely none of this. Would you
| rather Apple had won over this standard and we had resource
| forks everywhere? I understand that the largest use of
| alternate data streams at times in the past was malware, to
| evade virus scanners, which seems to be an anti-feature to
| me.
|
| A file should be a stream of bytes that can at least
| survive transfer over a socket (that is 8-bit clean),
| unlike this ODS-2 detritus.
| tedunangst wrote:
| You know that e.g. linux supports extended attributes and
| they don't copy so easily over ftp either?
| chasil wrote:
| Yes, I know that ACLs, security contexts, and zfs/btrfs
| parameters aren't going to move. This would be know as
| "filesystem metadata," and the send/receive operations on
| zfs/btrfs would be the best bet to preserve them.
|
| However, my ODS-2 problem was not metadata, it was _data_
| that lacked esoteric formatting. I certainly don 't need
| that for Oracle RMAN - an FTP binary transfer will always
| works.
| tambourine_man wrote:
| At some point you start wondering what's meta and what's
| data, after all.
| canucker2016 wrote:
| I don't care who wins - I don't have a horse in this race
| at all.
|
| I'm sure the FTP devs were just trying to get something
| working - they weren't worried about supporting all the
| possibilities for the definition of "file".
|
| I'd guess the only people who really care are those
| affected (mainly VMS/Apple people who have to
| send/receive files via FTP).
|
| Well, I guess if I was trying to FTP a file that had data
| in an alternate data stream (ADS) from an NTFS drive, I'd
| care. But then I know that the ADS wouldn't be sent by
| FTP and package the file accordingly.
|
| I don't think POSIX designed anything - they just
| codified/standardized what was running in the wild.
|
| Wouldn't something like the VMS-equivalent of a tar
| archive of your files work better for round-tripping to a
| non-VMS computer?
|
| The largest use of alternate data streams is actually
| when a user downloads a file from the Internet via
| Internet Explorer/Chrome/Firefox on WinXP+. The browser
| creates an alternate data stream, Zone.Identifier, for
| the downloaded file so Windows will know that the file
| may be tainted.
|
| see https://guyrleech.wordpress.com/2018/08/06/where-did-
| that-do...
| MikeTheGreat wrote:
| Ok, now I'm curious :)
|
| Can I ask why a 'single binary stream' view of a file would
| cause problems when dealing with something from a 'file is
| actually a bunch of records' system?
|
| Like, if the FTP program sends an identical sequence of
| bits to the destination machine, how could the destination
| have problems deciphering the (identical) file?
|
| I don't have any experience with VMS, but I'm assuming that
| there's information in the file (somehow, somewhere) that
| tells you where the records start and stop (whether it's a
| terminating character or a set size of each record, etc)
| because otherwise how would VMS know where to find each
| record. Is this not true? Is there information about the
| file that's stored outside the file?
| ale42 wrote:
| As far as I remember when I used VMS while at the high
| school, if you write a text file in VMS and then read it
| byte-by-byte as binary, well, there are no line
| separators at all. Each line is seemingly a record in the
| file. I don't know more about the details, but the
| information is not in the byte stream of the file, it's
| probably some metadata elsewhere in the filesystem. Other
| OSs don't work that way. An FTP transfer in ASCII mode
| will insert line breaks where appropriate while
| downloading from VMS, but other file types relying on
| records can't be reproduced outside VMS as simply.
| anthk wrote:
| XFS was for media and big files. If you were on A/V
| production with huge recordings, XFS was the right choice
| over a network.
| chasil wrote:
| Oracle is putting a LOT of development effort into XFS, and
| I am assuming that they are doing so because it makes their
| database faster.
|
| https://blogs.oracle.com/linux/post/xfs-2019-development-
| ret...
|
| https://blogs.oracle.com/authors/darrick-wong
| HeckFeck wrote:
| I have a habit of selecting XFS for all my Linux
| installations. It is a truly excellent filesystem.
|
| Though I must confess the reason I initially chose it was
| because of the name.
| wannacboatmovie wrote:
| > Thank heavens that UNIX thinks a file is a single stream of
| bytes. Windows continues the feature experimentation with
| NTFS "alternate data streams." My, how clever that is [not].
|
| Macs use Resource Forks, a type of alternate data stream, yet
| I am frequently reminded (here) that macOS is Certified (r)
| UNIX (r)(c)(tm).
| aap_ wrote:
| Recently a build from april 1991 showed up:
| https://betawiki.net/wiki/Windows_NT_3.1_April_1991_build
| causality0 wrote:
| There's an almost mystical air about the era of Windows when
| Microsoft was focused on world domination through offering a
| genuinely better product. I still remember sitting on my bed
| totally transfixed during the install when I moved from 95 to 98.
| dale_glass wrote:
| That's interesting, I didn't like 95 much, and 98 didn't
| particularly impress me either. Both would blue screen a lot.
|
| Back then I was more impressed by NT and OS/2. Windows 2000 is
| the one I remember most fondly.
| golergka wrote:
| Depends on what you had to compare it with. For me, the base
| of comparison was Win 3.11 and DOS, and Windows 95 blew my
| mind.
| stg22 wrote:
| OS/2 would have beaten NT in the consumer and SMB markets.
|
| 95 dealt with every weakness of NT in those markets and
| provided developers, consumers and SMBs with a smooth
| migration path towards it. It had very good backwards (DOS
| and 3.11) and forwards (XP) compatibility, reasonable
| hardware requirements and the ability to boot into DOS
| (important for gaming and some other applications at the
| time). It also offered massive advances over 3.11 (proper
| multi-tasking, the best looking GUI of the time and greatly
| improved stability) that when combined with the above, pushed
| user adoption.
|
| Given its design requirements, 95 could never have been as
| stable as NT, but it was a spectacularly successful interim
| solution for customers not yet ready for the trade-offs that
| would have involved.
| selimnairb wrote:
| And I believe Windows NT 4 (which had the Windows 95 UI)
| was the most stable version of NT for a while, even after
| Windows 2000 came out. Windows 2000 put the video driver in
| kernel space, which was faster than NT 4, but the early
| drivers were not great.
|
| ---EDIT:
|
| I misremembered. It was NT 3.51 that still had the video
| driver in userspace, NT 4 moved it into the kernel:
| https://en.m.wikipedia.org/wiki/Windows_NT_4.0
| mschaef wrote:
| > I still remember sitting on my bed totally transfixed during
| the install when I moved from 95 to 98.
|
| For me, the key Windows 95 moment happened while I was sitting
| in an IBM lab working as a tester for OS/2 Lan Server. We had a
| copy of Windows 95 we were testing for compatibility.
|
| Playing around one afternoon, we decided to try to break it. We
| started Drivespace compression, let it run for a while, and
| pulled the plug on the machine. Plugged it back in and it
| picked up right where it had left off...
| Unfrozen0688 wrote:
| And Windows 95 is DOS too, not NT. So less stable than NT
| mschaef wrote:
| Agreed, although the relationship of DOS and Windows 95 is
| significantly stranger than that wording implies. DOS boots
| Windows 95, and legacy I/O can go through DOS, but DOS is
| also hosted by V86 machines to run DOS apps.
|
| I'm pretty sure it's possible for a DOS process running in
| a window to issue a file system request (a DOS API call)
| that gets trapped and processed by 32-bit VMM code and then
| immediately forwarded back to (a different instance of) DOS
| for handling by a legacy device driver.
|
| There are reasons for this.
|
| It's amazing it worked.
|
| I'm thankful it's gone.
| HeckFeck wrote:
| Not quite. DOS was just the bootloader for Windows 9x.
|
| While Windows 95's kernel didn't have the full feature set
| of NT, it still was more sophisticated than DOS.
|
| Source: https://devblogs.microsoft.com/oldnewthing/?p=24063
| evntdrvn wrote:
| icymi, Dave Cutler's got a great YouTube channel!
|
| https://www.youtube.com/@DavesGarage
| drewg123 wrote:
| That's not his channel, its a different Dave. However, there is
| an _excellent_ 3 hour long interview with Dave Cutler on that
| channel.
|
| https://youtu.be/xi1Lq79mLeE
| eddythompson80 wrote:
| That's not Dave Cutler. That is David Plummer. He interviewed
| Cutler. Plummer is retired, Cutler is still full time employee
| at Microsoft.
| mise_en_place wrote:
| > From reading the book, I learned that Cutler had the same
| mentality for his OS and, in fact, the system wasn't ported to
| x86 until late in its development.
|
| The reason for this is that the RISC architectures at the time
| (MIPS, Alpha AXP, PPC) had weird ISAs. There are things like
| branch delay slots and MIPS in particular required aligning to a
| 32-bit boundary (x86 does not). Dave Plummer has a YT channel and
| I highly recommend his interview with both Dave Cutler and
| Raymond Chen. It was a lot of work to port to these other
| architectures, and they didn't have source control at the time,
| so they were hand merging these changes.
| runjake wrote:
| Link:
| https://www.youtube.com/watch?v=vxmZPMg7vIs&list=PLF2KJ6Gy3c...
|
| And if you're interested in that, you'll likely be interested
| in his Raymond Chen interview as well:
|
| https://www.youtube.com/watch?v=UTPTPJEVYlY&list=PLF2KJ6Gy3c...
| Unfrozen0688 wrote:
| NT really has stood the test of time. Still runs on Windows 11. I
| am weary about the future though.
| tambourine_man wrote:
| >These days we take x86-based personal computers as the one and
| only possibility
|
| The author must not like Macs :)
___________________________________________________________________
(page generated 2023-11-28 23:01 UTC)