[HN Gopher] Bug in reader/writer locks in Windows API
___________________________________________________________________
Bug in reader/writer locks in Windows API
Author : Georgelemental
Score : 200 points
Date : 2024-03-03 15:59 UTC (7 hours ago)
(HTM) web link (old.reddit.com)
(TXT) w3m dump (old.reddit.com)
| mastax wrote:
| I understand why this is the case but it's also extremely
| frustrating:
|
| > It is extremely difficult for programmer-users to report bugs
| against the Windows API (we're supposed to direct you to Feedback
| Hub, but you may as well transmit your message into deep space).
| I've filed OS-49268777 "SRWLOCK can deadlock after an exclusive
| owner has released ownership and several reader threads are
| attempting to acquire shared ownership together" with a slightly
| reduced repro.
|
| > Thanks for doing your homework and creating a self-contained
| repro, plus pre-emptively exonerating the STL. I've filed this OS
| bug as a special favor - bug reports are usually off-topic for
| r/cpp. The microsoft/STL GitHub repo is the proper channel for
| reporting STL misbehavior; it would have been acceptable here
| even though the root cause is in the Windows API because this
| situation is so rare. If you see STL misbehavior but it's clearly
| due to a compiler bug, reporting compiler bugs directly to VS
| Developer Community is the proper thing to do.
| adgjlsfhk1 wrote:
| The best way I know of to report bugs to windows is report them
| as documentation bugs. see for example
| https://github.com/MicrosoftDocs/cpp-docs/pull/3526.
| IshKebab wrote:
| Clever!
| ddlsmurf wrote:
| It looks like it was declared in Nov 21, and in May 23 they
| merged in the "fix" by adding "it's approximate" to the docs
| ( https://github.com/MicrosoftDocs/cpp-
| docs/commit/447b5d8a781... ), so not sure this is the best
| approach for actual bugs.
| 1over137 wrote:
| Nice to know that it's not just Apple to whom reporting bugs in
| hopeless. :)
| masklinn wrote:
| I generally assume that's the case for any large company.
|
| Sometimes I get pleasantly surprised, but generally speaking
| the internal incentives are skewed against, the primary focus
| is whatever the roadmap is followed by tickets from paying
| clients, public bugs generally have a very low hit ratio so
| they're unrewarding, unless you manage to snipe one of the
| company's employees (either nerd-snipe or interest / shock
| them enough to raise the issue internally) it's like playing
| the lottery.
| jansan wrote:
| > I generally assume that's the case for any large company.
|
| Despite Google partly losing its marbles recently,
| reporting bugs to Chromium still works very well.
| bee_rider wrote:
| I wonder if the open source element of the project keeps
| them "honest" to some extent?
| generic92034 wrote:
| > I generally assume that's the case for any large company.
|
| What works to some degree are dedicated maintenance teams
| providing development support. If their main task is fixing
| bugs and they are evaluated on this basis, support tickets
| reporting real bugs have a good chance to receive the
| required attention.
|
| However, there is always the temptation for management to
| redirect resources from those teams. But at least in the
| B2B area costly customer escalations can remind management
| of the importance of good maintenance.
| tgv wrote:
| Depends on the team, I think. I've had no luck at all with
| the OS/framework and "core" apps bugs, except getting the
| report that it has been fixed, and I should install the new
| version of the OS. Only to find out the bug wasn't solved.
|
| The Logic team, however, has been helpful, and in one or two
| cases (that were discussed in musician's forums) went out of
| their way.
| Dalewyn wrote:
| The amount of attention and care a bug report or a piece of
| feedback receives is inversely proportional to how easy it is
| to file it.
| jcalvinowens wrote:
| > It is extremely difficult for programmer-users to report bugs
| against the Windows API
|
| I can't imagine living in this hell. When I find bugs in Linux,
| I E-mail the actual engineers directly and get responses in
| under 24 hours:
| https://lore.kernel.org/lkml/Zcb3_fdyJWUlZQci@gmail.com/
| iknowstuff wrote:
| Thats pretty cool. No comments left in the code after the
| revert though. Are they relying on their minds and commit
| history as documentation of invariants? How do they prevent
| the same mistake from being made again in the future?
| charcircuit wrote:
| Microsoft engineers have emails too.
| dieortin wrote:
| And how would you know what email to contact in a case like
| this one? It's not about "having email"
| charcircuit wrote:
| You can either go down the route of finding the team that
| owns the code and then contacting someone on that team,
| or by contacting someone you know from the company to
| look it up for you or report the bug on your behalf.
| TillE wrote:
| Like the comment says, filing an issue on GitHub would've also
| eventually led to a proper internal report, so the system isn't
| wholly reliant on one guy trawling Reddit.
|
| I have many complaints about Win32 (having half a dozen
| different error code types, for one), but outright _bugs_ are
| really rare.
| bingo3131 wrote:
| Misleading title?
|
| This is a Windows API bug with the slim reader/writer (SRW)
| locks. It's just that the bug was discovered via
| std::shared_mutex as that is implemented using SRW locks.
|
| SRW locks: https://learn.microsoft.com/en-
| us/windows/win32/sync/slim-re...
|
| Confirmation from a Microsoft employee that the bug has been
| raised internally with the Windows API team:
| https://old.reddit.com/r/cpp/comments/1b55686/maybe_possible...
| dang wrote:
| Ok, I've changed the title to say that. Thanks!
| tialaramex wrote:
| Yes, for example if you make a Rust std::sync:RWLock on
| Windows, it will literally be SRWLock because Microsoft
| advertises this API as having exactly the behaviour Rust wants
| so why would you build something worse instead ?
|
| Rust's Mutex on Windows is also an SRWLock but it can't hit
| this bug because it deliberately only uses the exclusive
| locking.
| iknowstuff wrote:
| So it doesn't use a shared lock when .read() is used?
| zozbot234 wrote:
| Rust has both Mutex and RWLock. The Mutex only uses
| exclusive locks, there's no distinction between "read" and
| "write".
| iknowstuff wrote:
| Oh sorry my morning brain thought the comment was all
| about RWLock
| amluto wrote:
| Once upon a time, you could buy various things from MS that came
| with support incidents. I had an MSDN subscription that came with
| two per year. Using an incident got you an actual support
| engineer who would be helpful and escalate issues if necessary.
| And, if your issue turned out to be a real bug of any
| significance in an MS product, your support incident would be
| credited back.
|
| This was great for developers (real support was available, and
| spam was discouraged), and it was great for MS (they found real
| issues that affected paying customers and they got feedback to
| improve their documentation).
|
| I wonder whether this program still exists. I have the impression
| the overall quality of MS documentation has declined.
| twisteriffic wrote:
| That's Premier support, though now with a bunch of extra layers
| of contractors who can't do anything other than follow a
| script, capture traces and wait on the next level. In my
| experience it's been a time consuming waste of effort. I've had
| a lot better luck harvesting names and business cards from the
| teams at conferences and demo days, then going direct to the
| source.
| sebazzz wrote:
| Generally as far as first line support goes, everything is
| outsourced to India. Cheaper, on paper.
|
| Anectodically, I tried to get support for something unrelated:
| I'm trying to use IMAP sync support in Outlook.com, but it
| refuses to work properly with iCloud-IMAP but doesn't give any
| error message either. As a paying M365 subscriber I expect
| proper support. I tried over five times to get support, and
| every time it ended in frustration. Every time I got someone
| who either doesn't understand to product, says IMAP support is
| not available and deprecated (it is not! I was able to set-up a
| different IMAP provider just fine), or I got redirected to the
| Windows or Office support team, who then couldn't help me and
| closed my ticket.
| Someone1234 wrote:
| Often times it just feels like they try to extend the ticket
| out as long as possible asking irrelevant questions, then as
| you said redirect you or close the ticket after wasting an
| appropriate amount of time.
|
| I cannot prove it, but I bet there is an internal number of
| replies before they can close without penalty.
| rwmj wrote:
| If it's any consolation, our company pays for 20,000 GMail
| licenses and we seem unable to escalate any issue at all to
| Google, even major issues such as their clearly not working
| spam filtering, emails being lost, incorrect deduplication of
| emails, or their random IMAP throttling.
| amluto wrote:
| Is there any vendor of actually good email-as-a-service?
| O365/Exchange/Outlook/Hotmail is a mess. Google has a
| support problem. Fastmail has an offline email problem.
| iCloud is not obviously suitable for professional use.
|
| What's left?
| Avamander wrote:
| There are multiple providers out there that have their
| own stacks (Google, Microsoft, Tutanota, Protonmail to
| some extent and others) and there are those that use more
| common combinations (basically just managed mail-in-a-
| box). Nothing perfect though, so pick your poison.
|
| Some issues are also due to the ecosystem itself.
| Avoiding POP3 goes a long way for example.
| marcosdumay wrote:
| Well, maybe last century.
|
| At any point of this century, the response to those support
| incidents from MS was always to deny a problem, and if it's a
| known one, to try to gaslight the customer in a direction
| contrary to solving it.
|
| I have seen organizations lose way too many people-hours trying
| to satisfy the MS support and apply what it recommended. That
| when the real solution was often reachable in a hour or two of
| research on 3rd party knowledge bases.
| amluto wrote:
| ISTR a got an actual useful outcome from a support incident
| related to a bug in the runtime libraries shipped with Visual
| Studio 2005. Support gave me a hotfix.
|
| Sadly the hotfix had its own little bug: installing it took
| about 20 hours. Office productivity was rather low for the
| rest of the day. I don't know everything that goes on under
| the hood with Microsoft's installers, but wow they're slow.
|
| I, blissfully, don't use Visual Studio on Windows any more :)
| kkert wrote:
| I actually remember reporting a crypto bug way back through a
| similar program in Win98, where certain keys would not work if
| the highest bit was set in the key material. It got fixed
| userbinator wrote:
| You can still pay MS for support. Whether you'll get the answer
| you want, as the others have noted in sibling comments, is
| highly variable.
| AndrewStephens wrote:
| Subtle bugs in Reader/Writer locks do not surprise me. I worked
| on an in-house implementation based on Win32 (before C++11 and
| std::shared_mutex) and my recollection is that although the
| implementation sounds simple it is exceedingly easy to make
| subtle mistakes.
|
| The experience left me with such a bad feeling for shared locks
| that I tend to avoid them unless absolutely required. When I last
| tested std::shared_mutex, performance was so poor compared to
| std::mutex that double-buffering the data protected by a simple
| mutex was much faster.
|
| This was a great post by the original Redditor.
| convivialdingo wrote:
| I did some work with testing a cross platform in-house library
| for read-write locking.
|
| We tested cmpxchg16b and found the performance was terrible
| with more than 4 cores.
|
| Ended up using spin-locks similar to Linux kernel RCU.
| tialaramex wrote:
| SRWLock seems like a situation where it's small enough for you
| to construct proofs that what you did is correct, and important
| enough that the enormous expense of such proof is justified.
|
| The C++ std::mutex provided in Microsoft's STL is ludicrously
| big, whereas SRWLocks are the same size as a pointer. In one
| sense being ludicrously big is great - less risk the std::mutex
| will mistakenly share a cache line with your unrelated data,
| but for most people this makes std::mutex annoyingly expensive.
| SunlitCat wrote:
| Sad thing is, Reader / Writer locks are pretty tempting to use
| as they appear to be lightweight.
| userbinator wrote:
| It has been my experience that anything related to concurrency
| can be full of subtle edge cases. I tend to avoid it completely
| unless absolutely necessary.
| chrisjj wrote:
| > Bug in reader/writer locks in Windows API
|
| Correction: Bug in reader/writer locks in Windows
|
| Not in API.
| valleyer wrote:
| Thank you. Blaming the API is like saying there's a problem
| with your gas pedal after you blew a head gasket.
| loeg wrote:
| Some reddit comments mention it reproduces back to Vista (2008).
| I am kind of shocked no one has noticed this bug in that time. I
| guess under typical rwlock usage you just get random instances of
| shared lockers unable to acquire the lock and no deadlock, but
| still.
| Diggsey wrote:
| I think part of the reason is that there's a very similar code
| pattern which _is_ user error, and avoiding that pattern tends
| to avoid this pattern as well.
|
| The similar case occurs when you have: - 1+ threads holding a
| shared lock (Readers) - 1+ threads waiting to acquire an
| exclusive lock (Pending Writers) - 1+ threads trying to acquire
| the shared lock (Pending Readers) - 1+ Reader is waiting on a
| Pending Reader
|
| In this case the Pending Readers will be unable to acquire the
| shared lock even though it is still in "read mode" because in a
| fair RW lock Pending Writers are prioritised above Pending
| Readers so as not to starve the writer side of the lock.
| TillE wrote:
| Vista is when that API was first implemented.
|
| It's doing a pretty weird thing with the locks, I wonder what
| the actual use case was. Readers should almost never care about
| other readers. Typically you just grab the lock, read the
| thing, and release it. You always have to be super careful
| about deadlocks if you're holding a lock and also waiting
| around for something else to happen.
| davekilian wrote:
| I was wondering how something so basic could go unnoticed for so
| long. Halfway down the page on OP's link, a user u/rbmm provides
| a compelling answer: that there are (possibly expected?) cases
| where a thread trying to acquire the lock in shared mode can
| accidentally get it in exclusive mode instead. This is due to
| interleaving of atomic bit test-and-[re]set operations between
| the (shared mode acquire) thread and the (exclusive mode release)
| thread running simultaneously.
|
| The repro code holds the shared lock while waiting for all other
| threads to acquire the shared lock, and thus deadlocks if any of
| the worker threads accidentally gets an exclusive lock. In
| "normal" use cases where the lock is used to protect some shared
| resource, threads holding the lock don't wait for each other, so
| there is no deadlock.
|
| Interesting stuff!
| zozbot234 wrote:
| Why does it have to use bit test and set and interleave with
| other threads, though. AIUI you can use a CAS loop to implement
| any RMW atomically over word-sized (or double word sized, on
| many platforms) data. That seems like a no-brainer.
|
| For comparison, the Rust implementation for lightweight RWLocks
| on futex-capable *nix platforms is here: https://doc.rust-
| lang.org/stable/src/std/sys/unix/locks/fute... It sets the
| "reader counter" in the underlying atomic to a special value to
| signal that the lock is set for exclusive access. So a reader
| thread acquiring the lock as shared can never result in this
| kind of bug. Bits are used to signal whether readers or writers
| are currently waiting on a lock, but this just cannot turn a
| lock that's acquired for shared access into exclusive, or vice
| versa.
| userbinator wrote:
| If that "rbmm" is the same person as I've seen on other sites,
| and the characteristic non-native English is a clue that it is,
| he certainly knows his stuff.
|
| _threads holding the lock don 't wait for each other_
|
| Unless you're doing nested locking.
| Nition wrote:
| > It is extremely difficult for programmer-users to report bugs
| against the Windows API (we're supposed to direct you to Feedback
| Hub, but you may as well transmit your message into deep space).
|
| :'-(
| Avamander wrote:
| This is an issue with a lot of MS products. I honestly have no
| clue how much money one would have to throw at MS to get a bug
| fixed.
|
| For example Microsoft's *own* Pluton-enabled platforms fail
| Windows' Device Health Attestation checks due to an incomplete
| chain (https://call4cloud.nl/2023/04/are-you-there-intune-its-
| me-ha...).
| Nition wrote:
| I've noticed a lot of big products have a user feedback cycle
| that goes something like this:
|
| - Create new feedback tracker
|
| - Direct feedback to tracker
|
| - Stop paying any attention to tracker
|
| - Tracker is hundreds of pages of users shouting into the
| void, and much of it out-of-date
|
| - Delete everything
|
| - Create new feedback tracker...
| Nition wrote:
| There's nothing worse than having a problem with a product,
| finding 250 other people on the feedback tracker that have
| had the same problem over the last three years, and the
| only official response is some support person on the first
| page who's saying your feedback is very important to us,
| and have you tried [troubleshooting that won't work]?
| Avamander wrote:
| At that scale they have 250 000 people that didn't try
| the obvious troubleshooting first though. But yes, it's
| annoying.
| AnthonyMouse wrote:
| The fundamental problem here is that they have a billion
| users and most of them don't know what they're talking
| about. If you create a simple way to contact the company it
| will soon be full of messages from end users who can't even
| articulate what their problem _is_ but it 's usually some
| kind of malware or user error and is definitely not a
| problem with whatever component they're reporting the issue
| against.
|
| What you really need is a way to report problems which is
| _high friction_. You have to submit a git pull request of
| your ssh public key so you can transmit your bug report via
| sftp. Now they only get bug reports from people who can
| figure out how to do that and can actually pay attention to
| them because it filters out all the spam from people asking
| Microsoft how to connect their Android to a Mac.
| mschuster91 wrote:
| Microsoft makes ~16 billion $ a quarter in profit, of
| which they return ~10 billion $ to their investors.
|
| They could go and spend 200 million a quarter on decent
| customer support without making too much of a dent in
| their financial line.
|
| [1] https://www.microsoft.com/en-
| us/investor/earnings/fy-2023-q2...
| AnthonyMouse wrote:
| $200M is approximately $0.15/user. How much support do
| you expect to get for that?
| mschuster91 wrote:
| How many of these 1 billion users actually need support?
| Only a tiny fraction.
| AnthonyMouse wrote:
| Uh... what? If there was a free number you could call to
| get competent technical support, people would spend their
| entire day on the phone with it instead of reading
| documentation or hiring IT staff.
| TexasMick wrote:
| I've ran into this too, the feedback hub seems to be populated
| by first line support types.
|
| I also found a bug in a win32 API and the feedback hub told me
| to reboot my PC
| mook wrote:
| Unfortunately, the opposite of that is everybody trying to file
| random things that aren't actionable. See the collection of
| mail sent to the curl maintainer...
|
| https://github.com/bagder/emails/blob/main/2015/2015-06-08.m...
| userbinator wrote:
| I'm curious if this also occurs in WINE's implementation.
|
| I also want to test this on my highly customised XP install which
| has been patched to add the SRW API among other extensions, and
| where I had also patched the kernel to fix a race condition
| causing a deadlock in the keyed event API that the SRW
| implementation is based on (maybe it's this same one, although in
| Vista+ they changed it significantly; but the same edge case
| could occur.)
| nightowl_games wrote:
| How did you patch the kernel? Like how is that possible?
| userbinator wrote:
| With a hex editor, debugger, and skills that most developers
| these days seem to lack.
|
| I patched the kernel in memory first, using a kernel
| debugger, to verify my fix worked before editing the file on
| disk.
| gorlilla wrote:
| Windows xp and server 2003 sources (though incomplete AFAIR)
| were leaked back in 2020.
| zozbot234 wrote:
| The code is here
| https://source.winehq.org/source/dlls/ntdll/sync.c#0474 and it
| uses compare exchange operations throughout, so it should be
| unaffected.
|
| The ReactOS implementation is more involved
| https://doxygen.reactos.org/d1/db8/srw_8c_source.html but
| still, it uses mostly CAS operations both for the shared and
| the exclusive case. So it should be largely free from issues.
| ww520 wrote:
| The program has a bug. It's mixing atomic and non-atomic
| variables in the yield() checking loop. Non-atomic variables have
| no guarantee on cache consistency for different threads. This can
| cause the loop to run forever. struct
| ThreadTestData { int32_t numThreads = 0;
| std::shared_mutex sharedMutex = {};
| std::atomic<int32_t> readCounter = 0; }; //
| child thread DoStuff() {
| data->readCounter.fetch_add(1); while
| (data->readCounter.load() != data->numThreads) {
| std::this_thread::yield(); } }
|
| The numThreads field is not an atomic variable. It's initialized
| to 0 and set to 5 in the main thread. Its memory address is then
| passed to the child threads to be checked in the yielding loop.
| Since it's non-atomic, there's no memory barrier instruction to
| force its new value (5) to propagate to all CPU's running the
| threads. A child thread might get the old value 0. The logic of
| the yield checking loop using it would never exit.
|
| Since the main thread runs the code in an endless loop, the same
| numThreads memory allocated on the stack is being set to 0 and 5
| repeatedly. Some of the child threads can get the old value in
| one pass of the loop. Thus the hanging.
| userbinator wrote:
| You seem to have missed the part where an actual MS employee
| confirmed it was a bug in their API.
| ww520 wrote:
| He just read OP's code, the C++/STL Standard, and the
| Microsoft Learn document and made that conclusion. That's a
| rather haste determination. Unless he read the actual Windows
| lock implementation and found a bug there, I don't think his
| conclusion is correct.
| forrestthewoods wrote:
| > That's a rather haste determination.
|
| I hope you realize how deeply ironic this statement is. If
| you read the comments you'll find he even produced a
| slightly reduced repro. And in other comments he tried
| minor tweaks like the one you suggested.
|
| You have a thesis that the program has a bug. (It doesn't.)
| Go ahead and test your thesis and report back.
| nemothekid wrote:
| > _Its memory address is then passed to the child threads to be
| checked in the yielding loop. Since it 's non-atomic, there's
| no memory barrier instruction to force its new value (5) to
| propagate to all CPU's running the threads._
|
| Each core would have to fetch the value from main memory, where
| it will be undoubtedly 5. There is no valid reordering (at
| least under x86) that would cause the thread to read 0.
| ww520 wrote:
| The main thread is running the child thread creation in an
| endless loop, repeatedly setting numThreads to 0 and to 5,
| back to 0 and to 5 again. Can the caches of the CPU's
| consistently keep up with the changes?
| zozbot234 wrote:
| > repeatedly setting numThreads to 0 and to 5, back to 0
| and to 5 again.
|
| The reset to 0 and to 5 happens at the start of the loop.
| There's a happens-before relationship between it and the
| threads being created, and then again between the threads
| being joined and the loop cycling back. So there _shouldn
| 't_ be any data race here.
| ww520 wrote:
| CPU's not running the main thread don't care about the
| execution order of the instructions of the main thread.
| They only see their local caches of the same memory
| location got changed from 0 to 5, 5 to 0, and back to 5
| repeatedly. When a new thread lands on a CPU with the old
| 0 cache value, it will hang.
| zozbot234 wrote:
| The code runs in what's effectively a single-threaded
| context. CPUs running other threads will be involved when
| joining and launching threads, this should suffice as
| synchronization.
| cesarb wrote:
| > CPU's not running the main thread don't care about the
| execution order of the instructions of the main thread.
|
| On x86, they do (the x86 family is unusual in having
| strong memory ordering), but that's not the issue here.
|
| > They only see their local caches of the same memory
| location got changed from 0 to 5, 5 to 0, and back to 5
| repeatedly.
|
| Their local caches of that memory see only a 5, since at
| the moment they read that cache line, the value in memory
| is 5; the operating system ensures that the write of the
| 5 value by the main thread is flushed to memory[ _]
| before the main thread starts the child thread, and also
| that the cache of the child thread does not have stale
| data from before that moment. That memory location is
| only set back to 0 after all the child threads have
| exited, so there 's no instant where the child thread
| could read a 0 on that location from main memory into its
| cache.
|
| > When a new thread lands on a CPU with the old 0 cache
| value, it will hang.
|
| When a new thread lands on a CPU core with an old 0 cache
| value for that memory location (which could happen if
| that CPU core had been running the main thread, and the
| main thread was migrated to another CPU core before it
| could set it back to 5), it will still see a 5 at that
| memory location, because the operating system invalidates
| the cache of a CPU core when necessary before starting a
| new thread on it.
|
| [_] Actually, it only has to be flushed as far as the
| last level cache, or the "point of unification" in ARM
| terminology; I simplified a lot in this explanation.
| ot wrote:
| `numThreads` is written before the child threads that read it
| are started, so there is an explicit happens-before
| relationship and no data race. Before `numThreads` is reset,
| the child thread are joined.
|
| There is no bug in the program, it is legal to use non-atomic
| variables across threads as long as they're correctly
| sequenced.
| ww520 wrote:
| See my last paragraph above.
| forrestthewoods wrote:
| Your assessment is completely, totally, and verifiably wrong.
| It's obnoxious, but not surprising, how many commenters have
| chimed in with "program has bug BLAH" when they provably did
| not try to run the program themselves.
|
| Because if you did attempt to run the program you'd find that
| changing numThreads to a constexpr makes no difference.
| cesarb wrote:
| > there's no memory barrier instruction to force its new value
| (5) to propagate to all CPU's running the threads.
|
| The equivalent of the memory barrier instructions is there, but
| it's hidden within the operating system code which creates and
| initializes a new thread. That is, the operating system ensures
| that the value in the current CPU (in this case, 5) is
| propagated to the CPU running the newly started thread, before
| the thread start routine (in this case, DoStuff) is called. The
| value is not modified while the child threads are running (it
| waits for the child threads to exit before clearing the value),
| so there's no chance of the child threads seeing the value
| being set back to zero.
| ww520 wrote:
| Based on the Windows CreateThread API [1], it doesn't say
| anything about memory synchronization guarantee. Does it do
| internally?
|
| [1] https://learn.microsoft.com/en-
| us/windows/win32/api/processt...
___________________________________________________________________
(page generated 2024-03-03 23:00 UTC)