[HN Gopher] Building the DirectX shader compiler better than Mic...
___________________________________________________________________
Building the DirectX shader compiler better than Microsoft?
Author : slimsag
Score : 261 points
Date : 2024-02-10 09:22 UTC (13 hours ago)
(HTM) web link (devlog.hexops.com)
(TXT) w3m dump (devlog.hexops.com)
| mcraiha wrote:
| This is also related to Godot "The reason to make it optional is
| that Direct3D 12 support currently relies on the proprietary
| dxil.dll library from the DirectX Shader Compiler being shipped
| together with Godot, and shipping proprietary software goes
| against the mission of the Godot project."
| https://godotengine.org/article/dev-snapshot-godot-4-3-dev-3...
| moffkalast wrote:
| One of those things that the end users won't ever give half a
| shit about. Same thing when installing some linux distros: "dO
| yoU wANT To doWNloAd AND iNStAll 3Rd paRty driVers"
|
| Yes. Yes I want my GPU, wifi and bluetooth to work. Get over
| yourselves and leave that checkbox checked by default.
| dvngnt_ wrote:
| some people do
| AeroNotix wrote:
| Speak for yourself.
| gkbrk wrote:
| GPU, Wifi and Bluetooth commonly work with open source
| drivers
| comex wrote:
| More often the issue is firmware rather than drivers.
| TillE wrote:
| It's not an arbitrary ideological decision, it's a legal one.
| Blame Microsoft for their dumb license on dxil.dll which, as
| I read it, would require Godot to add some kind of click-
| through agreement.
| malkia wrote:
| dxil.dll is managed ("dotnet") code, right?
| yazzku wrote:
| No, it's C++, based on LLVM.
| https://github.com/microsoft/DirectXShaderCompiler
|
| Edit: apparently dxil.dll is not part of DXC (the classic
| move to make "open source" software dependent on external
| proprietary garbage, apparently.) But I'd still doubt it's a
| managed DLL.
| yazzku wrote:
| Which part of dxil/dxc is proprietary exactly? Trying to make
| sense of the license barf at
| https://github.com/microsoft/DirectXShaderCompiler
| charcircuit wrote:
| The source code for dxil.dll is not part of that repo.
| yazzku wrote:
| Looks like this is another case of "Microsoft loves Open
| Source" then.
| charcircuit wrote:
| Are any libraries from the Windows SDK open source?
| Windows application code calling into libraries that are
| not open source is nothing new.
| zerocrates wrote:
| The license (and the code) for dxil.dll/libdxil.so isn't in
| that repo, they just include the blob in releases. If you
| look at a release you'll see an additional LICENSE-MS.txt
| that just covers that dxil signing library.
| yazzku wrote:
| How is that compatible with the GPL licence from autoconf?
| flohofwoe wrote:
| A great overview of the terrible mess that underlies cross-3D-API
| shader compilation (and even though it focuses on D3D and
| Microsoft, it's not much better on other 3D APIs - for instance
| you simply can't cross-compile Metal shaders from a Linux host -
| only from macOS and - somewhat recently - Windows).
|
| If the Mach team can pull off this whole "Zig as a cross-3D-API
| shader compiler" and make it work as smoothly as "Zig as a cross-
| compilation toolchain", then this would be pretty much the
| biggest thing in computer graphics since 1995 or so :)
| lr1970 wrote:
| > If the Mach team can pull off this whole "Zig as a
| cross-3D-API shader compiler" and make it work as smoothly as
| "Zig as a cross-compilation toolchain", then this would be
| pretty much the biggest thing in computer graphics since 1995
| or so :)
|
| And a major boost for Zig -- both as a language and a
| toolchain.
| lastmartyr wrote:
| awesome it is the time to improve microsoft's code is ugly, great
| code is sweet, beautiful is better
| Voultapher wrote:
| What a joy to read. It's this sort of infrastructure work that
| unlocks so many doors. Thank you Stephen!
| mouse_ wrote:
| Microsoft has no incentive to make good software. Most people
| will use it no matter what.
| lukan wrote:
| That is not true. People use it as long as the pain of using
| it, is lower than the pain of switching. Assuming they have
| even something to switch for.
| npteljes wrote:
| >Assuming they have even something to switch for.
|
| This one hits the nail on the head, and the reason why not
| just Microsoft, but a lot of large software players are not
| incentivized to create better software.
|
| At the end of the day, people like power, to make money, and
| the people at Microsoft are no exception. And businesses are
| businesses, enterprises to make money, not altruistic
| benefactors of humanity, or optimizers of a specific domain,
| like software. So what business will do are their original
| thing AND business tactics, and the larger the business, the
| more tactics they have to employ, otherwise they won't be as
| large, or even simply won't be, at all. So, on the top, it's
| all ruthless business tactics. As Microsoft is a large player
| for a long time, they have quite the rep sheet[0], but they
| are not unique in doing this. It's the name of the game.
|
| [0] https://en.wikipedia.org/wiki/Criticism_of_Microsoft
| pjmlp wrote:
| Thankfully Valve is doing the good work to keep game
| developers targeting Windows and DirectX without caring about
| alternatives on the PC space.
| timlatim wrote:
| Apologies if I'm misreading your intention, but are you
| suggesting that Valve's work on Wine is somehow worse than
| asking game developers to target Linux/other OSes natively?
| As a Linux desktop enthusiast, I much prefer the Valve's
| approach: the library of existing Windows-only games that
| are unlikely to be ever ported is too vast, and the
| benefits of targeting a disjointed[1] platform with <2%
| market share[2] for new games are not at all clear. It's
| only thanks to Valve that I (and hopefully many other Linux
| users) do not need to maintain a second Windows system for
| fun, as the majority of games run perfectly fine on Linux
| and require nothing more than clicking Install then Play in
| the Steam client.
|
| [1] Case in point: glibc's compatibility guarantees are
| weaker than what you get on Windows. (For instance, your
| system's glibc cannot be older than what a game is built
| against, which may present problems for devs using
| Fedora/Arch and players on Debian/LTS Ubuntu, something
| I've experienced first-hand for my apps.) The X11 to
| Wayland migration is also still underway. (Though things
| are getting better, the attitudes of some Wayland
| maintainers are a bit concerning: "I don't [care] what you
| think is normal behavior for games. You get certain
| guarantees with wayland. Deal with it. If clients decide to
| do exactly what they do on windows or X11 they won't work
| correctly." [3] I'm not sure game developers would enjoy
| such reception.)
|
| [2] https://store.steampowered.com/hwsurvey
|
| [3] https://gitlab.freedesktop.org/mesa/mesa/-/merge_reques
| ts/18...
| blibble wrote:
| indeed
|
| Windows is going to end up as the 2030s version of dosbox
|
| edit: as in Windows ends up with the same fate as MS-DOS:
| replaced with a piece of software analogous to dosbox
| pjmlp wrote:
| Which runs MS-DOS games, not GNU/Linux.
| blibble wrote:
| yes, that was indeed my point
| pjmlp wrote:
| Yes I am, everyone worshiping Valve for that, has skipped
| classes on OS/2 history lesson.
|
| Porting games from Android/NDK into GNU/Linux is
| relatively a child's play.
|
| Playstation OS is also POSIX friendly.
|
| Finally every serious middleware engine supports
| GNU/Linux.
|
| Still the amount of studios that care about GNU/Linux is
| almost zero.
|
| With Valve, there are no reasons to bother at all as a
| studio, target Windows/DirectX, let Valve do the work,
| collect the money with zero additional effort.
|
| Now with Windows based handhelds, Valve will learn what
| happened to netbooks.
| pjc50 wrote:
| "GNU/Linux" doesn't have a stable binary graphics API.
| Someone rather controversially pointed out that if you
| want to make a binary that runs for the longest possible
| time across the widest set of linux distributions, you
| should make a Win32/DirectX binary and tell people to
| install WINE/Proton.
| kbolino wrote:
| Wine/Proton works because it can be recompiled for every
| version of every Linux distro, whereas the average video
| game can't.
|
| Honestly, somebody could have done something like this
| for "native" Linux but the incentives never really fell
| into place. It could even have been Valve, but I would
| guess they weighed the options and found this to be the
| one that gave them the widest compatibility without
| expecting too much of the game developers, many of whom
| were reluctant to make their games available on Steam at
| all.
| Ygg2 wrote:
| Sure Linux has a stable graphics ABI. It's called
| Proton./j
|
| It's a riff on only stable OS API on Linux is Win32.
| kbolino wrote:
| GNU/Linux means Linux plus bash, coreutils, glibc, etc.
|
| What do I need to make a video game? Input, sound,
| graphics. What use is a shell, a userland, or even a C
| library (if I'm not writing in C) for making a video
| game?
|
| Android is _not_ GNU/Linux. It doesn't need to provide a
| shell or userland as part of the platform, and its C
| library is bionic not glibc. It also provides a lot of
| things GNU doesn't*, like input, sound, and graphics.
| It's also not designed to run on a desktop, and the
| differences between mobile and desktop are non-trivial
| and slowly growing.
|
| Windows and macOS are full graphical operating systems.
| Linux is just a kernel. GNU/Linux is just a command-line
| operating system. Great for servers, terrible for video
| games.
|
| Moreover, commercial video games are distributed in
| binary format, not source. Even if input, sound, and
| graphics were solved on Linux in a consistent way, the
| developers** can't help but break ABI compatibility every
| couple of years. Many apps from Windows 3.1 days can
| still run on Windows 11. What binary from Linux's early
| days can still run today, never mind with graphics or
| sound?
|
| * = You could include GLib/GTK+/GNOME but then you're
| targeting a specific desktop environment and not simply
| "Linux" or even "GNU/Linux" anymore
|
| ** = Except for the kernel developers, upon whom Linus
| forces "never break userspace" as a hard rule
| delta_p_delta_x wrote:
| I think you're talking past each other, and actually
| agree with each other.
|
| I believe pjmlp's point (although it requires a fair bit
| of reading between the lines) is that Windows already has
| fantastic backwards compatibility (as you elaborated on),
| and Valve's work has created a situation such that all
| developers need to do is target and build for only
| Windows, release Windows-only binaries; then, Valve/WINE
| will do the hard work in ensuring they run seamlessly on
| Linux. This means developers don't need to care about
| building _natively_ for Linux (a la Factorio and a tiny
| handful of other games). In other words as another
| commenter said, the real stable ABI on Linux is Win32 +
| WINE.
|
| Furthermore, Valve's work also negates the work of open-
| source engine and game developers who have ported their
| engines and games to native Linux. This is because
| developing for Windows is a known quantity, and there is
| an overwhelming volume of resources, effort, and
| experience in writing games for Windows.
|
| pjmlp concludes with 'Now with Windows based handhelds,
| Valve will learn what happened to netbooks', which I
| gather to mean that the Steam Deck will lose popularity
| to Asus and MSI's (and soon, other manufacturers too)
| handheld systems, since running most games directly on
| Windows is _still_ easier than the occasional faff that
| someone has to do when running games on WINE /Proton.
| WhyNotHugo wrote:
| Honestly, I don't think this is a bad idea. Games are
| usually distributed in binary form, and Linux is an awful
| target for distributing binaries. Especially if you want
| them to work 10 years later.
|
| Imagine games that dynamically link some library (or
| glibc). They wouldn't run on other Linux distributions with
| mismatching dependencies (or a different libc).
|
| Sure, 100% statically linked binaries might work. But wine
| also runs on non-Linux.
|
| Windows is a sufficiently homogeneous target, and wine
| works well enough. Heck, some games run better in
| Linux+Wine than they do in windows.
| chollida1 wrote:
| I mean, this is an absurd statement.
|
| Microsoft uses this software to make their own video games.
| They have a very vested interest in making any software they
| use better.
| yazzku wrote:
| It is not absurd at all. Just look and how every PC ships
| with Windows (and lately doesn't even allow you to boot an
| alternative OS unless you fiddle with Secure Boot in the
| BIOS). There is consequently little incentive to make Windows
| better, and we all know what a complete piece of utter
| garbage it is. Their next milestone is shoving ads in Outlook
| and your start menu.
|
| And then the same can be said about a lot of Microsoft
| products. DirectX is no different; it's the canonical
| Microsoft piece of shit, and that goes all the way back to
| the OpenGL days [1].
|
| [1] https://www.gamedeveloper.com/programming/why-you-should-
| use...
| Emigre_ wrote:
| This is excellent. Bravo, mister! Really interesting stuff.
| msk-lywenn wrote:
| So the signing[1] DXIL.dll does is just a modified MD5?
|
| 1:
| https://github.com/hexops/DirectXShaderCompiler/blob/4190bb0...
| atq2119 wrote:
| That "signing" has always been bullshit security theater.
| You'll note that every other graphics API manages just fine
| without it. I'm glad somebody posted the "signing" algorithm
| out in the open.
| TillE wrote:
| Putting a pseudo-MD5 hash in your file header sounds like
| someone got confused and wound up caught between over-
| elaborate integrity check and half-hearted security measure.
|
| It sort of works if your signing tool is part of a private
| console SDK, but the DirectX SDK was always freely available.
| Culonavirus wrote:
| This is the part of HN I really like. People who can look at
| random C code working with memory and be like "yep this looks
| like modified X". Pretty amazing (to someone like me who mostly
| works in high level languages).
| msk-lywenn wrote:
| With the phrasing of the article, I immediately thought it
| must have been something simple/dumb. I quickly thought MD5
| so I looked at the magic numbers in the C code and looked
| them up on wikipedia. Then I noticed the four F G H I
| functions. Still I'm not sure it is (maybe it's something
| common in hashes?) and I still don't know what the change is
| EMIRELADERO wrote:
| From the commit message[1]:
|
| "dxil.dll is closed-source, so we cannot simply patch it in
| the same way. To fix this, we outright disable runtime
| loading of dxil.dll and silence warnings related to it not
| being present / the final binary not being code signed.
| Instead, once the compiler would emit a compiled shader
| blob, we perform our own code signing algorithm (Mach
| Siegbert Vogt DXCSA) which results in a bitwise identical
| compiled shader, and thus dxil.dll is no longer needed to
| perform code signing of shaders."
|
| [1] https://github.com/hexops/DirectXShaderCompiler/commit/
| 7a013...
| Quarrel wrote:
| I find it frustrating that the Mach guys have this great write
| up, but don't just say what the "signing" is.
|
| Anyone paying attention in the article to that point knew it
| had to be some basic hash or similar, where even in your worst
| case someone has to reverse engineer some assembly.
|
| After all that effort, just out Microsoft, particularly when
| we're talking about opensource code that anyone that cares to
| can just dig through and find (thanks for that, msk!).
| charcircuit wrote:
| Renderdoc, who has had code to do this since 2021 has a good
| comment that explains it.
|
| https://github.com/baldurk/renderdoc/blob/4a620bb5a16b4de4e2...
| amluto wrote:
| Not quite:
|
| https://github.com/baldurk/renderdoc/blob/81b4d3d804c1974b0b.
| ..
| nraynaud wrote:
| Somebody went to take a big one for the team Open Source, thank
| you.
| sylware wrote:
| I guess the less troublesome would be something like
| HLSL/GLSL->SPIR-V<->DXIL, or shaders could be authored directly
| in SPIR-V.
|
| In vkd3d (from wine), I think you have a DXIL->SPIR-V translater
| (much more robust than any high level shading language converter
| since it is a much more simple intermediate language).
|
| That said, apart from this abomination of llvm, is there a
| HLSL->DXIL compiler written in plain and simple C99 (namely which
| require NOT gcc or clang to compile)?
| noahwhygodwhy wrote:
| "Shaders could be authored directly in SPIRV" oh god please no
| lmao
|
| Also to answer your question, no. The hlsl to dxil translation
| is basically owned by microsoft. There's been little effort to
| move away from that
| NekkoDroid wrote:
| Well... they are rewriting & upstreaming DXC to LLVM main (ht
| tps://github.com/microsoft/DirectXShaderCompiler/wiki/Cont...
| ) and want to kinda deprecate DXC. IIRC currently only
| compute shaders are supported but I may be very wrong.
| slimsag wrote:
| Yes, SPIRV->DXIL does seem tempting and worth exploring. Very
| recently I learned Mesa has such a tool, spirv2dxil, though
| unsure how robust/featureful it is compared to HLSL->DXC->DXIL.
| detay wrote:
| Very promising and deserves support.
| a1o wrote:
| SDL people are working in creating some shader language in the
| form of SDL_gpu (not the old one), to launch in SDL3, so it could
| have some cross platform way to work with 3D graphics there for
| games. I have been looking at it closely for a little while.
| jsheard wrote:
| Is SDL_gpu meaningfully different from WebGPU? Looking at their
| goals they seem to be more or less the same, building an
| accessible lowest-common-denominator abstraction over
| DX12/Vulkan/Metal, except SDL_gpu is an early work in progress
| and WebGPU already has two good open implementations.
| stephc_int13 wrote:
| Not sure about WebGPU. Is it really good? Have you checked
| with some benchmarks?
|
| From my experience, it's been disappointing. Big, complex to
| setup, not convinced about many tradeoffs.
| jsheard wrote:
| WebGPU running _on the web_ necessarily has overhead
| compared to native due to all the extra validation it has
| to do for security, but there 's nothing stopping native
| implementations from offering unsafe escape hatches as
| extensions for those who want them.
| cactusplant7374 wrote:
| Playing Halo CE on Mac M1 with wine is an abject failure. I am
| assuming it is because Direct3D support is implemented poorly?
| steeve wrote:
| I highly advise people to look into the Mach ecosytem, especially
| mach-sysgpu, which is a complete reimplementation of WebGPU,
| written (mostly) by Ali Chraghi, who is 17
| fifteen1506 wrote:
| Microsoft <3 Open Source
| delta_p_delta_x wrote:
| > However, it doesn't require any additional .dlls to be shipped
| with the application.
|
| Many video games already do this with all the proprietary
| middleware they use (Bink, SpeedTree, PhysX, etc). Most launchers
| (Steam, GOG, Epic, etc) also require their respective .DLLs. Many
| games also use D3D11On12. Many _shipping_ games in my list have
| dxil.dll amongst their installed files.
|
| Therefore, an honest question: what's the problem with shipping
| an additional DLL? The work done here to reverse-engineer and re-
| implement the code-signing is fantastic--especially the fact that
| it is bitwise equal to dxil.dll's output. But I am ridiculously
| lazy and prefer to take the easier way out, and would've just
| shipped the DLL.
| msk-lywenn wrote:
| As you pointed out, it's really not a problem. However, the
| article points out at the end the real interesting bit: cross
| building from any OS/arch.
| chris37879 wrote:
| This opensource implementation can be included in existing game
| engines without requiring the programmer to know or understand
| what dxil.dll even is. The Mach engine, for instance, can use
| this to create a compiler that compiles zig code into shaders
| across all its targeted platforms in a way that doesn't require
| any additional setup or dependencies for the end user, it's
| just built into Mach's core.
| nightowl_games wrote:
| Using zig itself as a shading language is awesome. Zig is the one
| true language. It's a build system! It's a shading language! Wow!
| unnouinceput wrote:
| Quote: "All that was left was that pesky dxil.dll - what sort of
| magic might Microsoft be employing in that library to "sign
| shaders"? How can they prevent unsigned shaders from running on
| Windows machines that aren't in developer mode? How are they able
| to distribute that binary on Linux, too? I won't comment on any
| of those questions, but will say that you'll find dxil.dll is NOT
| a dependency of mach-dxcompiler in any form. You can compile an
| HLSL shader on a macOS machine using mach-dxcompiler, without the
| proprietary dxil.dll blob - and end up with a DXIL bytecode file
| that is byte-for-byte equal to one which runs it on a standard
| Windows box. Enjoy!"
|
| That above is the real magic. Since he won't comment on the how,
| I guess he took a swing at poping the hood underneath, and did
| exactly what Wine developers did 2 decades ago. Any old timers
| here remembering that scandal? Smart kid to not comment on the
| how, this way Microsoft won't have any legal leg and since times
| have changed with all that "Microsoft loves Linux" shit they yell
| at all corners (not that I believe that for a single second),
| then it will be swept all under the rug and FOSS wins. For now.
| rofrol wrote:
| > So the signing[1] DXIL.dll does is just a modified MD5?
|
| [1]
| https://github.com/hexops/DirectXShaderCompiler/blob/4190bb0...
|
| https://news.ycombinator.com/item?id=39325654
| charcircuit wrote:
| >Since he won't comment on the how, I guess he took a swing at
| poping the hood underneath
|
| The way signing is done is already public knowledge that can be
| found using google as other open source projects have
| implemented it.
| trevortheblack wrote:
| DXIL is pronounced "dixel", like "pixel" but replace the p with a
| d
| chrsig wrote:
| > DXC's fork of LLVM removed and/or damaged much of the code
| generation layer and infrastructure [of LLVM]. Given that,
| supporting DXBC generation in DXC would be a massive task to fix
| and restore broken LLVM functionality. Due to the large scale of
| this issue and resource constraints on our team we're not going
| to address this issue in [the new] DXC [compiler] ever. > > We
| may support DXBC generation in Clang in the future (we mentioned
| that in the original proposal to LLVM). That work is unlikely to
| begin for a few years as our focus will be on supporting DXIL and
| SPIR-V generation first.
|
| I appreciate this quote[0] from the microsoft camp. Setting clear
| expectations that something will not be done is a nice bit of
| fresh air.
|
| [0]
| https://github.com/microsoft/DirectXShaderCompiler/issues/57...
___________________________________________________________________
(page generated 2024-02-10 23:00 UTC)