[HN Gopher] Finding Windows HANDLE leaks, in Chromium and others
___________________________________________________________________
Finding Windows HANDLE leaks, in Chromium and others
Author : ingve
Score : 171 points
Date : 2021-07-26 07:36 UTC (15 hours ago)
(HTM) web link (randomascii.wordpress.com)
(TXT) w3m dump (randomascii.wordpress.com)
| tyingq wrote:
| Was not surprised to see MS Teams with 12k+ open handles after
| only running for less than an hour.
|
| Click the details tab in task manager, right click on any column,
| select "add columns".
| pixl97 wrote:
| Heh, if Teams was a linux application the first line of the
| documentation would be about editing sysctl.config to allow far
| more open files by default.
| dafelst wrote:
| If I recall, Teams is an electron app, which means it uses
| Chromium under the hood - possibly the same bug?
|
| I'm not sure that Electron apps have the same local storage
| characteristics as the full browser, but it seems feasible.
| fassssst wrote:
| We use this internally to help prevent leaks:
|
| https://github.com/Microsoft/wil/wiki/RAII-resource-wrappers
| mad_vill wrote:
| ah I didn't know we opensourced wil. sweet.
| minedwiz wrote:
| Think it was around the same time as terminal
| ddulaney wrote:
| That's an awesome set of tools! I've written much jankier
| versions of these myself, and I'll definitely look into using
| the official ones.
|
| That said, I don't know that they would have helped in the
| article's case. It looks like there was a higher-level resource
| getting leaked, which would also most likely leak these.
| aneutron wrote:
| It's always a great day where I'm guaranteed to learn something
| new when randomascii posts something.
|
| Such non trivial bugs and debug methods.
| Arnavion wrote:
| I wonder if these problems would happen less on Linux.
|
| On the one hand, most distros default to 1024 for the (soft)
| limit on open fds, which would be easier to hit if you were
| leaking fds and thus easier to detect.
|
| On the other hand, a lot of things that HANDLEs get used for
| might end up being userspace pointers from userspace libraries
| instead, so the leak would be of memory instead of fds and hard
| to detect again.
| jve wrote:
| I appreciate how humble he is: This I understand, this I don't,
| here I have experience, but not here. Example: "It took months to
| understand the bug, but then a fix was created fairly promptly
| (not by me - I still don't understand that part of the code)"
|
| Also nice how the issues are handled by team members, that know
| more about stuff in their field. Like, maybe other people
| wouldn't be skilled enough to find the issue, but he wasn't
| skilled enough to fix it.
|
| Learning along the way. When tackling issues, there are always
| some knowns and unknowns and the experience is gained. Thanks to
| the way randomascii is written, I'v had some success using WPA.
| bigbizisverywyz wrote:
| Oh I used to love leaky HANDLE hunts. These things could bring
| windows 3.n (and maybe NT 4.v too?) to its knees pretty quickly
| so I was always really careful about diligently freeing them.
|
| And, like many commenters below constantly surprised that so many
| apps still ship with these problems, since it's so easy to spot
| them. Windows perfmon also gives you a nice graph, so you can
| correlate e.g. GUI behaviour with a jump in handles being created
| that then never get freed.
| tialaramex wrote:
| Whilst it's easy to _spot_ the leak, the fact that you can leak
| handles by entirely forgetting they exist can make it a little
| harder for the programmers to find _where_ their code leaked
| the handle and so fix it.
|
| If you leak 6GB of RAM, there's 6GB of evidence about what was
| leaked exactly. If it's full of terrible love poetry you can
| rule out "FootBallScores" and focus on the "TeenagePoems" data
| structure and related code. But if you leak 50 000 HANDLEs then
| er... oops?
|
| The post shows you can narrow it down to say, Event HANDLES but
| after that it gets increasingly sticky. Hopefully somewhere
| there's a C++ object that owns the handle and _that_ has leaked
| which you can trace, but as I understand it, the handles
| themselves might be all that leaked, leaving you to instrument
| software so that you can find out which code made all the
| handles and then trace back those that seem leaked.
| sumtechguy wrote:
| One good rule I used to use and make everyone in my team use
| was when you get a handle write the 'anti' call at the same
| time. Good portions of the older windows API is allocate
| functions and destroy functions. If you write both at the
| same time your mental overhead is less.
|
| many start using 'auto managed' style languages (c++, java,
| etc) where the life cycle is not as clear. The life cycle is
| the same though but you own it in an indirect way. Who makes
| it. Who uses it. Who destroys it. In some languages that is
| easier to do, others you own it front to back. When doing
| this I try to start with create/destroy then usage. It is a
| style that helps remove leaks before they happen. They can
| still slip in there...
|
| I have used the common string thing a few times to help
| narrow leaks (does not work in all cases :( ). You also can
| use tools like valgrind, boundchecker, purify, etc. Think MS
| has a couple that I can not remember off the top of my head.
| pitterpatter wrote:
| Application Verifier can help detect handle leaks afaik.
| Honestly, an amazing tool when you have a situation that
| calls for it.
| asveikau wrote:
| The managed languages create additional hurdles for handle
| leaks. My prior experience with the .net GC is that it
| responds to memory pressure, and not necessarily to open
| handles. So when people write code that relies on GC for
| cleanup, it has a bit of a blind spot for the handles --
| they don't take much memory in your process so it won't
| know to invoke GC.
|
| There are some patterns to help, like using().
| tialaramex wrote:
| Historically there were two kinds of leak. Your program
| might grow more than intended, but eventually give
| everything back when it finished - or it might seize some
| resources permanently by mistake, so that you need to
| restart the computer to "fix" it.
|
| Modern operating systems mostly rule out the latter type of
| leak, you could leak _files_ I guess, and of course Cloud
| users could leak things like S3 objects, even whole
| instances, but many resources are now automatically cleaned
| up when you exit.
|
| As a result of that though, for long-lived processes, such
| as Chrome but also most background tasks and server
| software, just "I definitely clean up the mess eventually"
| doesn't get the job done, the OS was going to do that too.
| The user doesn't care whether the resources would have been
| returned half a second after they closed your program when
| "clean_up_everything()" is called by the main thread, or, a
| second after that when the OS cleans everything left
| behind.
|
| So this is a real problem, about the actual meaning of our
| programs, and (though they are still a good idea) can't be
| helped by good programming techniques, garbage collection,
| Rust's Drop trait, the C++ RAII way of thinking, deferred
| clean-up in languages like Zig or Python, or anything else
| I'm aware of.
|
| We need to actually express in our programs the intent to
| hang on to only what's actually needed and clean everything
| else up as we go. And it can be sorely tempting to consider
| that "It gets cleaned up eventually" is good enough, that's
| where leaks get in.
| sumtechguy wrote:
| Spot on. But as for 'right now' I have to be a bit more
| practical and work with what I got.
|
| Mostly these days most machines have a decent amount
| memory so leaks are not as noticeable, unless you look.
| If in the early days if I leaked 50MB of memory and my
| machine had 16MB. I had a real issue and the machine
| would be borked. If I do the same today you would not
| notice it.
|
| It is why I stressed watching life cycle of an object.
| You made this thing, who is cleaning this mess up, and
| when. 'When' could be anywhere from never 'I need this
| all the time' to bunch it up when idle/reuse (garbage
| collection style), or 'right now' I need this memory back
| right now. There are trade offs and you need to watch for
| that too. The write it down while you are thinking of it
| has served me very well over the years. I personally got
| bit by not following my own rule a few weeks ago. I
| allocated something and I had not cleaned up correctly. I
| got 'lucky' and that was actually the right thing to do.
| But in the code review I rightfully got dinged on it.
|
| One thing I wish many more docs would do is 'this makes
| object xyz use remove_xyz to clean it up'. Or 'this looks
| like it is creating an object it is not, this is
| returning some global'. Right there in the doc. It would
| help so much.
|
| That temptation of 'eventually' is one that some
| languages push hard. But I find many leaks that I have
| chased over the years were just a poor understanding of
| the calls being used (bad docs, not reading them, or a
| combo). You may have had a different experience.
| a1369209993 wrote:
| > can't be helped by good programming techniques, garbage
| collection, Rust's Drop trait, the C++ RAII way of
| thinking, deferred clean-up in languages like Zig or
| Python, or anything else I'm aware of.
|
| Arena allocation. Every allocation must be attributed to
| some arena (eg current tab, current network request,
| current frame being rendered, etc); when the arena goes
| away, so do all its allocations.
| tgtweak wrote:
| First thing I do on a new Windows setup is turn on "handles",
| "threads", "Commit size", "NP pool" columns in task manager
| details...
|
| If you want to see some real offenders... have a quick look at
| Asus's "LightingService.exe" (the daemon that controls their rgb
| LED coloring suite). Gets up to 2m+ handles after a day or two of
| running on my system.
|
| The next biggest offender on my system? Their ez updater
| "EzUpdt.exe" at ~60,000 handles.
|
| Haven't bothered looking at it with process explorer/procmon yet,
| but I'm sure it's a quick fix.
| ziml77 wrote:
| I didn't even check the handles on LightingService. That shit
| got uninstalled fast after I saw it eating an entire CPU core
| and then some.
|
| I honestly didn't even want any lighting control software but
| all the RAM that had the speed and latency that I wanted were
| decorated with them so I needed software to set them to a
| static color. Thankfully I learned about OpenRGB.
| josteink wrote:
| > Thankfully I learned about OpenRGB.
|
| Especially for Linux-users where official support may not be
| available for your RGB HW (mobo, RAM, whatever), OpenRGB is a
| godsend.
|
| And scriptable too, so I can disable all that crap at boot-
| time.
| Scene_Cast2 wrote:
| In my experience, removing that software is tricky as well.
|
| There is OpenRGB (among others) that should be able to control
| lighting on multiple peripherals. And FanControl (among others)
| for fan curves.
| tgtweak wrote:
| It's a shit-show also, with every vendor having their own
| solution and 0 interop. RGB addressing isn't rocket science
| I'm not sure why every app needs its own proprietary daemon
| to handle this - mercy on your soul if you have corsair +
| asus + gigabyte in your system as I do.
| ziml77 wrote:
| I think it's because no one standardized on protocols for
| addressing RGB lighting on RAM, GPUs, and motherboards. We
| at least have those ARGB headers fairly standardized now,
| but wiring up even things that are slotted into the
| motherboard would suck (having just a few fans with those
| is already a mess).
|
| Also the manufacturers probably love it because it pushes
| people towards not mixing products from different
| companies.
| Someone1234 wrote:
| Just enabling the Handles column made me realize that the "Asus
| Update Service" SysTray Icon/application had a handle leak: as it
| had 64K handles open.
|
| There's also an Asus Lighting Service that still has 10K handles
| open.
| bluedino wrote:
| It seemed like in the old days you had to read the Petzold book
| to learn Windows programming. It actually explained what was
| going on underneath the hood. And if you wanted more, you'd
| read something by Jeffrey Richter, David Solomon, or Mark
| Russinovich.
|
| Now? You watch a couple YouTube videos, or perhaps a terrible
| Apress book. You get something to appear on the page, and
| viola!, you're a programmer.
| ripley12 wrote:
| There's a real problem here: those are _still_ the books to
| read if you really want to understand Windows. They're great
| books but they're old, often out of print, and the onus is on
| the reader to distinguish outdated info from still-relevant
| info. Yet as far as I can tell, nobody's publishing better
| books about Windows programming today.
|
| It's remarkably hard in 2021 for a newbie to start learning
| Windows programming "properly".
| pixl97 wrote:
| This doesn't surprise me at all. Vendor tools, like the ASUS
| utilities, or Dells update application, and many others are
| commonly outsourced from different companies and seem to import
| a string of security flaws. Providing the program, not
| providing a secure program tends to be the metric for delivery
| on these applications.
| po1nter wrote:
| I just did this and I'm surprised to see DropBox with 19K
| handles. Even more than Visual Studio at 4.5K handles.
| Handle type summary: <Unknown type> : 4
| <Unknown type> : 2 <Unknown type> : 332
| ALPC Port : 19 Desktop : 2
| Directory : 4 Event : 395
| File : 131 IoCompletion : 55
| IRTimer : 11 Job : 3
| Key : 104 Mutant : 21
| Process : 66 Section : 45
| Semaphore : 18109 Thread : 154
| TpWorkerFactory : 5 WaitCompletionPacket: 70
| WindowStation : 3 Total handles: 19535
| goldenkey wrote:
| Just for others who may not see where the handles column is:
| "Open task manager, go to details tab, right click on one of
| the already listed columns, click select columns, then scroll
| down to check handles."
|
| After sorting by handles, I see similar results. Logitech Hub
| Agent has 13,000 handles. It seems these gaming companies hire
| the worst coders without care for actual performance.
| Yet...they advertise their products as cutting edge
| performance. Quite the irony.
| fullstop wrote:
| I get the feeling that gaming PCs are restarted often.
| Arrath wrote:
| Not remotely true, in my personal case.
|
| I might change that stance when I get home tonight and
| check for leaks like this, though.
| taneq wrote:
| > It seems these gaming companies hire the worst coders
| without care for actual performance. Yet...they advertise
| their products as cutting edge performance. Quite the irony.
|
| Unfortunately gaming companies routinely hire inexperienced
| coders because they're cheap. By the time those coders have
| skilled up, they've been burned by the gaming industry in one
| of the many ways that the gaming industry burns coders, and
| so they no longer work in gamedev.
| sumtechguy wrote:
| Another two good utilities to use is performance monitor, and
| process explorer. Perf mon is good for showing usage over
| time with its built in graphing and recording.
| mook wrote:
| "Process Hacker" can sometimes display more information as
| well; for example, I see that igfxEM.exe on my machine (Intel
| graphics driver) is leaking around 5000 handles to the registry
| key HLKM\system\ControlSet001\Control\Class\\{5c4c3332-344d-483
| c-8739-259e934c9cc8}\0001 over a week of uptime...
| nyanpasu64 wrote:
| On the topic of leaking HANDLEs, Synergy/Barrier's service leaks
| zombie processes and causes systems to malfunction over time, and
| this hasn't been fixed in years.
|
| https://github.com/debauchee/barrier/issues/567
| brucedawson wrote:
| Ouch. Zombie handles are so much worse. Now we're talking
| serious memory, and since it isn't attributed to the process
| that leaks the handles it is easy to not realize what is
| happening.
|
| More details: as I explain in the first link in the blog post,
| if a process leaks process handles then roughly 64 KB of memory
| will be used for each leaked handle (each zombie process) but
| that memory will not be attributed to any process. If you have
| the Handles column open you will see a large count in one
| process, but that process will not have a large memory
| footprint. If you kill that process you will reclaim the memory
| from all of the zombies.
|
| Good times.
| herf wrote:
| You can track handles within your process using
| GetProcessHandleCount. We have discovered some leaks in Windows
| APIs this way (which were fixed pretty fast).
___________________________________________________________________
(page generated 2021-07-26 23:02 UTC)