[HN Gopher] Zig: Drop MinGW-w64
___________________________________________________________________
Zig: Drop MinGW-w64
Author : pcr910303
Score : 99 points
Date : 2021-12-19 00:50 UTC (22 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| olvy0 wrote:
| As someone who has to deal from time to time with code that is a
| mix of old COM components and newer mingw dependencies I found
| this discussion fascinating.
|
| I learned to avoid most issues by wrapping everything at the
| boundaries with a pure C api/abi, and not letting exceptions
| propagate at all, instead catching them and returning an error
| code.
|
| I wonder why zig can't do the same but I guess they want calls to
| directx etc to match what's documented instead of going through c
| like contortions.
| chasil wrote:
| I know that Clang runs on Windows, as PuTTY's "about" dialog
| credits it.
|
| Does this offer a reasonable replacement for MinGW?
|
| https://wetmelon.github.io/clang-on-windows.html
| malkia wrote:
| There is CLANG environment support for MSYS2 -
| https://www.msys2.org/docs/environments/
| jmull wrote:
| It's interesting to see the process at work.
| cyber_kinetist wrote:
| My last experience with MinGW-w64 was when I was trying to
| compile my C++ simulation code in Windows and finding out that
| AVX instructions were not working because the compiler had
| misalignment-related bugs. (The issue is still open in
| https://github.com/msys2/MSYS2-packages/issues/1209)
|
| MinGW/MSYS certainly had appeal to former Linux devs who didn't
| want to touch the horrors of MSVC, but Zig (with its included
| Clang compiler/runtime) might end up being a better solution for
| people trying to compile C/C++ code on Windows in a stable
| manner.
| colejohnson66 wrote:
| Wow. "mingwandroid" is coming off as pretty rude:
|
| > Is it really so difficult to determine the correct place to
| report issues to?
| anonymfus wrote:
| Hmm...
|
| https://docs.microsoft.com/en-us/cpp/build/x64-calling-conve...
|
| _> Return values_
|
| _> A scalar return value that can fit into 64 bits, including
| the __m64 type, is returned through RAX. Non-scalar types
| including floats, doubles, and vector types such as __m128,
| __m128i, __m128d are returned in XMM0. The state of unused bits
| in the value returned in RAX or XMM0 is undefined._
|
| _> User-defined types can be returned by value from global
| functions and static member functions. To return a user-defined
| type by value in RAX, it must have a length of 1, 2, 4, 8, 16,
| 32, or 64 bits. It must also have no user-defined constructor,
| destructor, or copy assignment operator. It can have no private
| or protected non-static data members, and no non-static data
| members of reference type. It can 't have base classes or virtual
| functions. And, it can only have data members that also meet
| these requirements. (This definition is essentially the same as a
| C++03 POD type. Because the definition has changed in the C++11
| standard, we don't recommend using std::is_pod for this test.)
| Otherwise, the caller must allocate memory for the return value
| and pass a pointer to it as the first argument. The remaining
| arguments are then shifted one argument to the right. The same
| pointer must be returned by the callee in RAX._
|
| Isn't this what
| https://github.com/ziglang/zig/issues/9998#issuecomment-9970...
| claims to be undocumented?
| anonymfus wrote:
| Following comments on Github also point this out.
| Andoryuuta wrote:
| > Otherwise, the caller must allocate memory for the return
| value and pass a pointer to it as the first argument. The
| remaining arguments are then shifted one argument to the right.
|
| I think the issue is that it isnt _fully_ documented anywhere.
| The MS docs do partially describe it, but it omits details
| about how this should be handled in the case of C++ member
| functions (which are the COM methods in question). C++ member
| functions always receive the hidden 'this' pointer as the first
| argument in Microsoft's x64 calling convention, which directly
| conflicts with this document stating the returned struct would
| be a "pointer to it as the first argument".
|
| You can see this in some of the godbolt links in the linked
| issue. Ultimately, some COM functions that take no arguments
| actually end up taking two hidden parameters in rcx and rdx
| (this pointer, and the return struct pointer respectively)
| bla3 wrote:
| Windows uses its own calling convention. I'm sure COM win32
| headers have no shortage or atrocities, but this post here is
| just saying that GCC doesn't implement the Windows calling
| convention for struct returns, while clang does.
| jbk wrote:
| That's the opposite of our experience on VLC and Firefox was
| sharing our opinion: mingw-w64 people are skilled, nice and very
| clever and think about all use cases.
|
| This is how we are able to support configurations that even MS
| does not support...
|
| Also, a contrario from what this threads seems to say, the
| Windows SDK headers are faaaar from being open source compatible
| badsectoracula wrote:
| Yeah, i've been using MinGW (and now MinGW-w64) for practically
| decades now, originally via MSYS and later via MSYS2 and i
| never remember having and real issues with it - i even did some
| DirectX programming with it.
|
| The only thing i remember being an issue at some point (and
| perhaps still is) is that it relies on msvcrt.dll that
| theoretically isn't part of the Windows platform but in
| practice that DLL has been there since Windows 98 and the
| chances of this ever being a real problem are zero - so it is
| really an "issue" about discussions with hairsplitting pedants
| than a practical one :-P.
| slimsag wrote:
| If you squint closely, you'll see my issue is not
| hairsplitting but rather a practical one about trouble with
| MinGW headers not being super up-to-date (missing DirectX 12
| APIs, etc.)
|
| I'm a huge fan of MinGW-w64, though. I would bet most
| involved in the issue are. We all win by improving the status
| quo - I'm just a buffoon figuring out the best way to do that
| :)
| jbk wrote:
| DirectX 12 API is a pretty small case to throw all the
| project to trash.
|
| Notably since DirectX headers are always a different case
| compared to the rest of the Windows SDK
| slimsag wrote:
| I'm a massive fan of your work on VLC, but I feel that's
| a bit of a mischaracterization..
|
| The author of the issue spoke about his troubles with
| MinGW in four bullet points at the top that are not
| related to DirectX at all, whether you agree or disagree
| with those the fact is most people (including Andrew
| Kelley) rebutted many of his arguments, and Andrew even
| reached out to MinGW devs on IRC to chat about the topic
| of contributing upstream. That seems a reasonable
| approach to me.
|
| I chimed in on the issue because DirectX 12 was the
| problem _I_ faced, explained what I learned, complained I
| still don't have a super official way to get up-to-date
| DirectX headers that are compatible with MinGW, and
| ultimately said I gave up on working on the issue out of
| frustration.
|
| Throughout the issue there are people suggesting throwing
| out all of MinGW would be reckless, heck even I concluded
| that would not be possible for multiple reasons in my
| write-up.
|
| I think people are reading too much into the title and
| trying to make this out to be some sort of attack on
| MinGW, I really do not think that was anyone's intent -
| we're just exploring if we can improve the status quo..
| AndyKelley wrote:
| Hi JB! Funny to cross paths with you in this context. I don't
| know if you remember me but I was a rookie programmer who got
| the pleasure of joining the VideoLan Conference in Dublin back
| in 2014, and then Paris the next year, and you were very kind
| to me.
|
| The GitHub issue title here is unfortunately misleading. I have
| renamed it to "ideas to improve windows header files and libc".
| Also, I hope it is clear that I rebutted the points made by the
| OP, because I completely agree with your summary that the
| mingw-w64 people are skilled, nice and very clever and think
| about all use cases.
|
| If any drive-by HN readers work at Microsoft, please help us
| with this issue:
| https://github.com/microsoft/win32metadata/issues/766
| rightbyte wrote:
| Ye I watch in horror as the poor Zig devs are rushing into the
| abyss. Like, 'how hard can it be maintaining a minimal posix
| compatibility layer for Windows?'. Or something.
|
| "Despite your apparent emotional defeat, my take on this is
| that you successfully pioneered the way forward, and I think
| the Zig project can carry the torch from here. "
| AndyKelley wrote:
| I have no fear of abysses:
|
| https://github.com/ziglang/glibc-abi-tool/
|
| https://github.com/ziglang/zig/blob/master/tools/gen_stubs.z.
| ..
|
| bring it on
| jmull wrote:
| Reading through that thread, it really doesn't sound like the
| main Zig developers are going to pursue this.
|
| E.g., I believe the comment you quote is appreciatively
| acknowledging the hard work a contributor put it, but isn't
| accepting the general plan to drop mingw. Generally, the core
| Zig dev seems to keep gently shifting the conversation back
| to current Zig goals, not taking on new ones.
| slimsag wrote:
| (I'm one of the people from the thread)
|
| Although this issue has degraded into a conversation at
| this point, which is a bit unfortunate and honestly my
| fault, I feel it's important to mention that the main
| contributors here (myself, other Zig devs, Andrew Kelley,
| and others) all know each-other, have either met in person
| or online, etc. and I believe are all just trying to
| improve the status quo.
|
| There isn't any decision on "should we drop mingw or not"
| because we don't actually know what that would look like at
| all, and that's not really what the issue is about. Better
| phrased the title would be "Issues with libc headers Zig
| ships on Windows" Most importantly, our goal is just to
| improve the status quo of C/C++ dev not just drop mingw out
| of some philosophical reasoning. If Zig takes on that
| burden, there needs to be super solid reasoning behind it
| and it would require some serious investment. If you read
| Andrew's messages closely, you'll see that is what he is
| really reiterating constantly.
|
| I don't think anybody in that thread is really opposed to
| closing the issue and saying we just need to
| improve/contribute back to MinGW, Wine, etc. In fact,
| that's the current status quo and thus far, I would say,
| the most reasonable approach.
|
| It's cool that these conversations are happening with some
| momentum, though. I'm not aware of many other places where
| people are actively discussing improving the status quo of
| Windows C/C++ development.
| jmull wrote:
| Good to know, thanks.
| jchw wrote:
| I think the MinGW project is very useful, but I simply do not
| recommend using it.
|
| My experience with it was trying to debug why a friend's build
| was working on Windows and not Wine. I figured it out: it's
| because MinGW implements something called pseudo relocations,
| which implement symbol-relative relocations inside the entrypoint
| of your executable. It's needed because when compiling, you don't
| know if an unresolved symbol will be external or not, so it gets
| compiled as if it's not. But when linking, and you finally know
| that the symbol is external, it's too late to change the code to
| refer to an import table indirection.
|
| On ELF platforms, relocations can refer to symbols in other
| libraries during runtime linking. So external symbols can be
| treated very similarly to internal ones, just with linking done
| at runtime instead of compile-time.
|
| On Windows, you have to use `__declspec(dllimport)` or some such
| to inform the compiler that the symbol _will_ be external. It can
| then generate code for referring to the IAT.
|
| MinGW wanted to support compiling UNIXy software that didn't use
| this attribute, so it does not require it. Instead... the ELF-
| style pseudo-relocations are used. So at the entrypoint, your
| MinGW program or DLL loops through a table of pointers inside
| machine code, looks up their real address in the IAT, and
| performs fixups. It will even temporarily change the protection
| of the relevant pages to be writable. Check to see if your MinGW
| program has unexpected imports to kernel32.VirtualProtect! (IIRC
| this can also happen if you trigger executable stack by using GCC
| trampolines, another reason to not use GCC...)
|
| (edit: Please read the replies below, but according to MinGW
| developers, this only happens in some edge cases; ordinarily,
| thunks will be used, which do not have this problem. In addition,
| MinGW binaries will always link to VritualProtect, because the
| pseudo-reloc code exists whether or not you use them. I never
| noticed it before, so that's interesting.)
|
| There's only one problem: this is extremely failure-prone. For
| example, if you are on AMD64, the typical CALL instruction
| contains a 32-bit RIP-relative offset. But in a 64-bit process
| space, two modules can happily be more than 2^32 bytes away in
| memory. I assume ELF platforms have a linker that's aware of
| this, since the linker knows about these special relocations, and
| thus has to ensure the binaries are in range... but I don't know
| much about ELF linking, so don't take that as fact.
|
| So why does this break on Wine and not Windows? Well, as far as I
| can tell, it doesn't break on Windows because for some reason
| ASLR never puts any of the modules too far away. I haven't
| figured out why. Wine doesn't implement ASLR and loads modules at
| their preferred addresses as long as it's available, and those
| addresses wind up being too far away. The pseudo reloc code does
| not output any errors when the address overflows; would probably
| be worth a pull request, but I am still puzzling with why Windows
| doesn't ever seem to cause the problem.
|
| Unless I am missing something crucial, it's possible that a large
| amount of Windows x64 binaries compiled with MinGW work almost
| entirely by accident, and maybe not even every time.
|
| Most symbols like this are function calls, so a significantly
| safer approach would be for MinGW to generate a thunk for those,
| and then just fail at linking when encountering a data import
| that isn't decorated. Unfortunately, I don't make the calls. :P
|
| I don't mean to disparage any specific persons. I just generally
| recommend against MinGW even in the face of the fact that it is
| virtually the only free software option to compile Windows
| binaries.
| mstorsjo wrote:
| A couple more details:
|
| > The pseudo reloc code does not output any errors when the
| address overflows; would probably be worth a pull request
|
| I actually did this, to make it error out cleanly instead of
| having to debug this very non-obvious issue, quite recently:
| https://github.com/mingw-w64/mingw-w64/commit/ca35236d9799af...
|
| (It's not in any stable release yet, but MSYS2 might be
| packaging recent git snapshots that could contain it.)
|
| > Check to see if your MinGW program has unexpected imports to
| kernel32.VirtualProtect!
|
| The runtime pseudo relocation fixing code ends up linked into
| your executables even if the executable doesn't use any runtime
| pseudo relocations - so essentially all MinGW programs will end
| up importing this function. That doesn't mean it does get
| called though.
|
| I actually did look into fixing it, to make it only be pulled
| in when something needs it. It would need changes both to the
| linkers (binutils ld.bfd and lld) and mingw-w64-crt. I do have
| WIP patches for this for lld and mingw-w64-crt, but I'm not
| familiar with with the ld.bfd internals to take on fixing that.
| Plus that changing it would break older versions of the linker,
| so it would require quite a bit of a transition phase.
| jchw wrote:
| > I actually did this, to make it error out cleanly instead
| of having to debug this very non-obvious issue, quite
| recently: https://github.com/mingw-w64/mingw-w64/commit/ca352
| 36d9799af...
|
| Thanks! I believe this will save many people a lot of time in
| their debugger.
|
| > The runtime pseudo relocation fixing code ends up linked
| into your executables even if the executable doesn't use any
| runtime pseudo relocations - so essentially all MinGW
| programs will end up importing this function. That doesn't
| mean it does get called though.
|
| Oops. Not a big deal, though; I assume it won't get called if
| the table is empty.
| mstorsjo wrote:
| In general, this doesn't work by accident. In practice,
| whenever calling a function from a different DLL, normally a
| thunk is inserted (if it isn't marked dllimport), and when
| referencing a variable that might reside in a different DLL
| (and be autoimported), its address is read indirectly via a
| full 64 bit pointer.
|
| When linking to a function "func" in another DLL, the import
| library normally provides two symbols, "func" which is a thunk
| which allows it to be called from anywhere, and "__imp_func"
| which is the IAT entry (containing a full 64 bit address). If
| the caller called it via a dllimport marked declaration, it's
| using "__imp_func", otherwise it uses "func" and ends up
| jumping via a thunk.
|
| In some cases, when mingw-w64 wants to override importing
| certain functions (by providing a statically linked version of
| it instead), the functions are marked "DATA" in the def file
| (which is used for generating the import libraries). Then the
| thunk is left out and only the "__imp_func" symbol is left. If
| you hit a situation where you didn't end up linking the
| intended replacement function, then it's possible to hit this
| situation where you autoimport something that only works if
| it's loaded close enough. (In hindsight, it might have been
| even better to comment out those functions entirely from the
| import libraries.)
|
| If you do hit this issue again (or if you remember exactly how
| to reproduce it), then plese do file a bug somewhere so we can
| look into it.
|
| But in general, mingw-w64 linked binaries, even including ones
| using the autoimport facility and runtime pseudo relocations,
| should work even if they are loaded far apart from each other.
| ChrisSD wrote:
| While you're here, do you happen to know if mingw will gain
| support for native thread local storage? Emulated TLS is
| still the only option, right?
| mstorsjo wrote:
| Clang (and lld) do support native TLS, and mingw-w64 does
| have the things that are needed. I think binutils also
| might have what's needed too, but AFAIK the thing that's
| missing is support for it in GCC.
|
| Actually, (upstream) Clang defaults to native TLS instead
| of emulated TLS. In MSYS2, Clang is overridden to use
| emulated TLS by deafult to interoperate better with GCC
| built code and libstdc++ though.
|
| The toolchain I maintain, https://github.com/mstorsjo/llvm-
| mingw, defaults to native TLS throughout.
| ChrisSD wrote:
| Thanks! That's super helpful.
| mstorsjo wrote:
| Oh, I forgot to add; these days, msys2 have got a number
| of extra package repos. They have one, "CLANG64", which
| has got Clang/lld as the default compiler (just like in
| llvm-mingw) and with all packages built that way. And
| there, it defaults to native TLS too. Depending on your
| use case, this might be easier to take for a spin than
| llvm-mingw.
| jchw wrote:
| This is useful information, and I think it explains exactly
| how we wound up with pseudo relocs instead of thunks, but
| honestly, the thing that makes this so bothersome is that
| there's really no documentation that explains this, there's
| seemingly no warning when it occurs, and there's no error
| when it fails.
|
| From my perspective, I saw WINE crashing with a CALL into
| seemingly nowhere. Then I realized the CALL on disk pointed
| directly to the IAT (which obviously makes no sense.) Finally
| I realized that the library probably shouldn't be linking to
| VirtualProtect, traced it to the entrypoint, and found an
| error message that made me aware of the existence of pseudo
| relocs.
|
| There was a 0% chance the person who created this bug was
| ever going to figure it out, and I'm not even sure I could've
| figured out the build issue if I had access to the project
| files. There's plenty of minutiae when it comes to MinGW and
| MSys and a lot of people are not adequately prepared for what
| they are getting themselves into. I think (hope) the mess
| with exception handling is cleared up and everyone just uses
| SEH on Windows, but I bet there's still issues with licensing
| and winpthreads, and MSys and PTYs.
|
| I'd be more than happy to try to submit PRs to help with this
| particular problem, though I'm not going to lie, I find the
| pseudo-reloc thing horrifying and would be afraid of somehow
| breaking some ridiculous edge case.
| mstorsjo wrote:
| Regarding error reporting - yeah, it's super non obvious
| when that happens. See the sibling post I made; I've added
| error reporting for that case now (2 months ago, so not in
| any released version yet).
___________________________________________________________________
(page generated 2021-12-19 23:02 UTC)