[HN Gopher] Why bugs might feel "impossible"
___________________________________________________________________
Why bugs might feel "impossible"
Author : atg_abhishek
Score : 150 points
Date : 2021-06-15 14:24 UTC (8 hours ago)
(HTM) web link (jvns.ca)
(TXT) w3m dump (jvns.ca)
| southerntofu wrote:
| My two "favorite" hardware bugs: dead RAM stick partially
| corrupts memory pseudo-randomly (no ECC) so everything sort of
| works but is really weird. CPU that for some reason misbehaves so
| much without a newer non-free microcode/firmware version, so that
| everything sort of works but breaks in random way (just like
| broken RAM).
|
| The former really plays with my nerves because i tend to question
| the whole universe before i question the hardware. The latter is
| even more evil because memtest/smartctl won't complain about
| anything.. i'm not sure if there's an equivalent utility to test
| CPU health? Anyway it took me a while to even think of trying
| non-free intel-microcode, as of the CPU/kernel wouldn't produce a
| helpful error message like "You bought hardware which is pure
| shit. Please install some more binary shit in order to use it at
| all without losing your sanity."
| toast0 wrote:
| On CPU issues, at my 4-year, we did a bunch of projects with
| 68HC11s, which stop branching when voltage is low. After a few
| episodes of this, I eventually figured out when it started
| running straight through everything, it was time to recharge
| the batteries (and probably take a break anyway)
| Hackbraten wrote:
| I love this. Reminds me of a sleep-deprived driver who keeps
| missing highway exits.
| toast0 wrote:
| Fun extra fact. The debugger would work, and you could
| single step through it not taking branches it should
| clearly take. My professor told us it was designed to be a
| low voltage cpu, but they had some issues, so 5v only...
| I'm guessing this was the issue.
| jakeinspace wrote:
| We've solved the halting problem! Just lower the voltage,
| ignore branch/jump instructions, and run straight through.
| Hackbraten wrote:
| I smell a new esolang: Alan. Has a dozen branch
| instructions but they're all ignored.
| CalChris wrote:
| the bug is hard to reproduce locally
|
| One of the things I like about LLVM is that it is written as
| passes. It's possible to stop after a pass and dump everything to
| IR/GMIR text. It's also possible to start a pass with this text.
| This makes unit testing of passes possible.
|
| GlobalISel is a rewrite of the instruction selection mechanism.
| I'm not sure what GlobalISel offers above monolithic SelectionDAG
| (well, it's faster) but it is much easier to test because it's
| broken into irtranslator, legalizer, regbankselect and
| instruction-select passes each of which can be unit tested
| independently.
| ThrustVectoring wrote:
| Had a fun one relatively recently that was a mix of "hard to
| reproduce" and "hard to get internal state information". Flaky
| test in a rails app that would fail one in every ten to hundred
| runs of the full test suite with "this random number is too big
| to be a primary key" kind of message. Root cause was an edge case
| passing through multiple swiss-cheese holes in various
| assumptions:
|
| 1. ActiveRecord makes an assumption that primary keys are
| integers, and does its own check whether or not they are big
| enough to be persisted (rather than catching a database error).
| 2. Furthermore, it does this by coercing the key to an integer if
| it isn't already one. This is done with to_i, which for a string
| takes any leading 0-9 characters and discards the rest. 3. We had
| a table with a string primary key (UUID of some sort) 4. One of
| our test factories was generating a hexadecimal string for that
| primary key 5. And the test factory was not deterministic and did
| not respect the --seed flag in the test suite.
|
| So the end result was a _very_ innocuous-looking line of code
| occasionally generating a hexadecimal string with enough leading
| numeric digits to be larger than ActiveRecord thinks you can
| stuff into a table, causing an extremely cryptic error message.
| It does not reproduce with the same test seed. And the stack
| trace was about three frames and discarded all the context - all
| I could see was that ActiveRecord was throwing a fit over somehow
| mysteriously receiving a large number somehow.
|
| Figuring that out was honestly like, 80% pure luck. Chased down a
| hunch that it was test object generation somehow and did that in
| the REPL, then narrowed it down via looking through child object
| generation until the haystack was small enough that I couldn't
| _not_ find the needle.
| meowface wrote:
| >And the stack trace was about three frames and discarded all
| the context - all I could see was that ActiveRecord was
| throwing a fit over somehow mysteriously receiving a large
| number somehow.
|
| This is Python rather than Ruby, but my average debugging time
| and frequency of "impossible to figure out" bugs drastically
| decreased once I started using a traceback library that
| provides a lot of context to each stack frame.
|
| It makes logs containing any raised exceptions much larger and
| more tedious to scroll through, but the benefits are more than
| worth it; especially for production services.
|
| I use better-exceptions (https://github.com/Qix-/better-
| exceptions), but there are a bunch of other good libraries as
| well.
| zwieback wrote:
| For me the hard bugs are the multi-threading ones where I made a
| bad assumption about order of execution. One from 30 years ago
| that I still remember is when a "packet received" interrupt for a
| response came in before the "packet sent" interrupt of the
| request, due to queuing in the lower levels of the network device
| driver. It totally crashed our system because the pointer
| pointing to a transaction data structure wasn't initialized yet.
| It caused a triple fault so it was an instantaneous reboot.
|
| Since then I've become more defensive with classic
| multithreading.
| hypnoscripto wrote:
| The best/worst is when you've run out of ideas that make sense,
| so you start trying ideas that don't make sense and one of them
| works.
| ChrisMarshallNY wrote:
| I've had all 4 issues, in my time.
|
| I am grateful that my initial training was as an RF tech, and my
| first job was a tech at a microwave receiver factory. It taught
| me how to find _really difficult_ problems.
|
| After that, most software issues are a cakewalk.
|
| The worst ones are occasional threading issues, buried inside a
| dependency. Sort of an "all of the above" bug.
|
| That's one big reason that I avoid dependencies like the plague.
| You only have to have one or two of those, to learn religion.
| ansible wrote:
| Obligatory link to "Debugging Rules":
|
| https://debuggingrules.com/
|
| -----------------
|
| I.... _hate_ debugging. Just hate it. If I 'm developing some
| code, and it doesn't do the right thing, that's fine, I'll find
| and fix the problem (usually, unless something wacky is happening
| with a 3rd party lib).
|
| But ask me to figure out a problem with a large and complex
| system, and I just find that so discouraging. I know how to do
| it, (in part thanks to the above), but I just don't like the
| process. You never know how long it will take. You never know if
| you'll end up digging down further and further, and it is a
| problem with the hardware or something else that is hard to fix
| (I do a lot of embedded development).
| tclancy wrote:
| First off, everything the author writes is worth a read and I
| want to thank her for that. Re:
|
| >the error message has 0 results when you Google it
|
| While this used to (and still does, I suppose) cause a bit of
| throat-tightening for me, I've learned this usually means it's a
| case similar to "your assumptions are wrong". It tends to be
| something I have misconfigured or a dead-bang obvious typo that
| my eyes look right past. Often it's something like having the
| wrong virtual environment in one shell tab which is causing a
| process to half-work but then fail in a misleading way.
| cwkoss wrote:
| Even better: the only result is a stackoverflow post that your
| coworker made, asking about the exact issue you're
| investigating, with zero answers
| QuercusMax wrote:
| Even worse - when it's a stackoverflow post from yourself 2
| years before, and there's still no good solution. This has
| happened to me.
| hmsimha wrote:
| Even more worse: it's a stackoverflow from yourself 2 years
| before, which you then closed because you "figured it out"
| (without expanding on how)
| QuercusMax wrote:
| Oof, that really is even worse, because you only have
| yourself to blame
| hmsimha wrote:
| Relevant XKCD https://xkcd.com/979/
| nexuist wrote:
| Another hole I used to put myself in is I would often copy and
| paste stack traces verbatim, and of course most of a stack
| trace is just filepaths unique to your user account and of
| course nobody has posted similar stack traces like that because
| they don't have the same username and folder as you do. I've
| learned to copy and paste smaller snippets that are more likely
| to be generic across many different machines and found much
| more success with that strategy.
| koksik202 wrote:
| Bugs are fine race conditions that leave no proper log that is
| something really special to troubleshoot especially if only
| happens under certain conditions
| extrapickles wrote:
| Bugs involving printers are among the worst, as everything
| software with them is a few notches down on the quality scale.
| Also, troubleshooting involves reams of paper or giant rolls of
| labels.
|
| A good test for a printer/print spooler is to set the printer
| offline, but still accepting jobs (eg: open paper tray), then
| send 20 print jobs and count how many the jobs get printed.
|
| I've spent countless hours and a 3' diameter roll of labels
| trying to figure out why a printer would occasionally not
| print&apply a label, causing all packages to be labeled with the
| wrong label. The printer could print&apply 1 label/second, so it
| made a lot of mistakes when it failed. We eventually had to
| dismantle the printer and test each major circuit board in
| isolation to find out that its internal network adapter had bad
| firmware that the manufacturer did not want to fix. It turns out
| that most IP->LPT adapters also have this same flaw too, so we
| had to basically buy a huge pile of them to find 1 that worked.
| sergiomattei wrote:
| Printers smell our fear and desperation.
| munificent wrote:
| _> the bug is actually 3 bugs_
|
| I hate these ones in particular. My debugging strategy is
| generally like a surgeon's: do no harm. When investigating a bug
| I try to have as absolutely few moving parts as possible.
| Otherwise it's too easy to create knock-on bugs or interfere with
| the repro. Many times I have "fixed" a bug by changing something
| only to later realize that all I did was cause the repro to no
| longer manifest it.
|
| My process is usually something like:
|
| 1. Come up with hypothesis for cause.
|
| 2. Fix the code according to that hypothesis.
|
| 3. Did it work? If so, done. If not _undo all changes from step
| 2_ and try again with a new hypothesis.
|
| But when a bug is the confluence of several issues, that step 3
| can make it impossible to find a fix. I hate having to make
| multiple speculative changes (especially when the right fix could
| be any of the exponential number of combinations of them). Often
| I end up going in circles because I realize there must be
| multiple different problems interacting to cause the issue.
| thangalin wrote:
| A similar question was asked and answered on SO:
|
| https://stackoverflow.com/a/1268464/59087
| jotux wrote:
| >it's very slow to reproduce
|
| I had an embedded system that hosted USB endpoints that normally
| booted in ~3 seconds and could connect to the host in 5-6
| seconds. We found an issue in environment testing that the unit
| took longer to connect the colder it got: ~10 seconds at 0C, 20
| seconds at -10C, 120 seconds at -20C, and after -30C it could
| take 15 minutes or just never connect.
|
| We had to instrument the whole device and stick it inside of a
| chamber to debug. Every change meant waiting another 20 minutes
| for the chamber to cool down. Eventually found a sense line left
| floating that would eventually float high once it got warm
| enough. Probably 3-4 weeks of troubleshooting and it ended up
| being a single device tree edit to configure the pin with a pull-
| up.
| ultimape wrote:
| I routed out a couple thousand heisenbugs in an old system I was
| working on. Turned out the web based platform had no cache
| invalidation / cache-busting feature enabled, so any time an
| update would be published, the browsers didn't always get the
| changes.
|
| The icing on the cake was that it was our app that was bad, and
| the 'fix' they implemented was a completely broken work-around.
|
| It was SOP to instruct the clients to turn off browser caching.
| So the app was slow as well. Inevitably the Client's on-site IT
| would install a new desktop and forget to turn off browser
| caching. So you'd get these weird states were errors would occur
| randomly and depend entirely on if someone had changed or hadn't
| set that setting.
|
| Years of cruft and chaotic deployment workflow meant there were
| easily 10,000 different places where the cache busting would need
| to be implemented. I figured out a way to fix this using nginx as
| a caching layer, and using features of a brand name Web Toolkit
| which we had already partly implemented without caching. Even had
| a test harness set up and way to catch bugs during a
| transitionary stage.
|
| Shame there wasn't any specific tickets on this exact fix because
| otherwise I would still be working there. Sure, I was assigned a
| bunch of the random error ones and found out what was causing
| them, but apparently being assigned a ticket and fixing the
| problem isn't part of my job?!
|
| The head of the web dev team (and my manager) quit shortly after
| I was hired. I now understand why.
|
| Fixing bugs is impossible sometimes.
| mrguyorama wrote:
| I have an "impossible" debugging situation right now, though it's
| a little outside the scope I think.
|
| My Windows C drive, an SSD, died unexpectedly after less than a
| year of operation (I literally checked the SMART stats the day
| before, for what little info they offer). I bought a new Samsung
| SSD, installed it, restored from a backup (which was harder than
| it should have been _backblaze_ ) and now I get random
| bluescreens. No issue, I've had BSODs before, I know how to use
| WinDbg to explore them.
|
| Problem 1; BSOD claims it is an "unrecoverable hardware error"
|
| Problem 2; No dump file is generated, and I have no idea why. My
| system is set to generate one, I've turned off autorestart,
| Windows claims the SSD volume is "healthy" after restart, but it
| does not write the file
|
| My only hope is that I'll figure out how to set up remote
| debugging or something
| wizzwizz4 wrote:
| I had an impossible debugging situation. One day, my Windows 10
| machine decided that it would get a NTFS_FILE_SYSTEM error on
| boot. A BSOD on boot means it'll try the recovery partition...
| which caused a FAT_FILE_SYSTEM error just after the first
| conhost.exe window appeared. After "recovering", it'd try to
| boot normally again - looping between two slightly different
| OSs, getting basically the same error over and over.
|
| Eventually, I gave up on debugging, and tried to re-install. I
| put in the recovery disk... and it FAT_FILE_SYSTEM BSOD'd, too.
| When booting off a _completely different device_ , with which
| there were definitely no file system errors.
|
| So I put Debian on the machine. No problems, for some reason.
| TacticalCoder wrote:
| My worst "impossible" bug I encountered in 1991 (and I should
| blog about it one day), while writing a DOS game (386/486 PCs
| era). Game was a mix of C and assembly and for the life of me I
| couldn't reason about it / recreate the conditions to make the
| bug appear. It was seemingly random, only very rarely happening.
|
| After days I gave up and invented the thermonuclear weapon: I
| decided to rewrite the entire game's engine to be deterministic.
| This took me along while but then I could record events
| (joystick/keys direction/firing etc.) and at which frame these
| inputs happened, and could deterministically replay the whole
| game.
|
| Thing is: back then deterministic games engine / replay based on
| inputs didn't exist yet (AFAIK). At least I didn't know of any.
|
| The first time I remember reading about a fully deterministic
| game engine was on Gamasutra, a post-mortem on the first Age of
| Empire.
|
| So basically my first impossible bug made me discover the idea of
| deterministic game engines and tiny "replay" save files.
|
| The actual bug? Well eventually it appeared but now I had save
| files and, sure enough, I could have the whole game replayed
| automatically and the bug would now show up. And so I knew it was
| now just a matter of squashing it. Just some good old C dangling
| pointer IIRC. When the hero had the option to get two shots
| (usually he only had one) and when one of the shot shot was still
| active when the level was cleared, that shot would keep being
| alive in the next level, but invisible and would invariably lead
| to corrupting the memory of the next level.
|
| Fun stuff.
| cbsks wrote:
| > I should blog about it one day
|
| Yes, please!
| lloeki wrote:
| I seem to recall '90 Stunts (also known as 4D Sports Driving)
| was somewhat deterministic as it featured replays, with which I
| recalled abusing the physics engine with carefully laid out
| tracks, and looping over the replays to much hilarity
| dfan wrote:
| For the DOS game Terra Nova (1996) I made our game engine fully
| deterministic. I guess this was a year before Age of Empires
| but well after you. It was amazing how much easier this made
| debugging. The effect was so great that I can't even put a
| multiplier on it, because it moved bugs from the "we'll never
| reproduce this" category to the "just see what happened and fix
| it" category. Some of these replays represented more than half
| an hour of gameplay too.
|
| One thing that surprised me is that it also found a bunch of
| bugs waiting to happen (uninitalized variables / dangling
| pointer sort of stuff) that would trigger an error when
| replaying from a file didn't produce the same results as the
| original play (we had a checksum of game state that we could
| check).
| tomlagier wrote:
| Is there any downside to writing deterministic games? It
| seems like the only sane way to do it, but I guess it adds
| some complexity overhead for the initial write otherwise
| everyone would do it by default?
| learc83 wrote:
| Yeah it's just harder and potentially less performant.
| There are different levels--deterministic across the same
| architecture, different architectures etc...
|
| The simplest example I can think of is being deterministic
| across different frame rates.
|
| Imagine you move a player by adding x to it's position each
| frame. You adjust x based on the frame rate so that you
| don't move faster on a faster computer.
|
| So on a computer tuning at 30 FPS you move 10 pixels each
| update. But on a computer running at 60 FPS you move 5.
|
| You have walls that are 6 pixels wide. On the 60fps
| computer it works fine, but on the 30fps machine you can
| teleport through the walls.
| zimpenfish wrote:
| I guess this is similar to the Fallout 76 "physics is
| tied to framerate" snafu (where looking at the ground
| makes you go faster because the fps goes up...)
|
| https://gamerant.com/fallout-76-speed-hack/
| munificent wrote:
| There aren't many significant downsides I know of, it's
| just really easy to accidentally make your game _not_
| deterministic.
|
| You have to entirely isolate the game state from any
| sources of non-determinism. The latter can include: subtle
| CPU timing issues, GPU timing, other GPU artifacts, the
| system clock, timing from IO operations. If you want the
| state to be deterministic across machines (useful for
| debugging multiplayer stuff) then you also need to include
| floating point operations (some chips behave differently on
| some boundary cases, I think) as well as some graphics
| operations (thinks like texture operations and rounding are
| always bitwise identical across GPUs).
|
| If _any_ bit of non-determinism sneaks in from one of
| these, it will wander through and pollute any other
| operations and data that depend on it. Flushing out non-
| determinism bugs can almost feel as difficult as debugging
| a non-deterministic engine.
|
| I was at EA when the Madden team refactored the engine to
| be deterministic. It took a full cycle of bug hunting, but
| it was marvelous once it got there. QA could just send over
| a replay file and any dev could simply load up the replay
| and repro the bugs. In fact, the user-facing replay system
| in the game ("Let's watch that play again in slow mo!") was
| simply restarting the engine and then replaying the user
| inputs deterministically to resimulate the whole game
| again.
| skykooler wrote:
| There's a bunch of excellent blog posts from the Factorio
| developers about tracking down tiny bits of non-
| determinism (since it uses deterministic multiplayer, as
| there is way too much dynamic world state to update
| constantly over the network).
| TacticalCoder wrote:
| That is really cool! There may have been others, earlier
| ones: just not that I knew of.
|
| You came with the idea yourselves or you knew about other
| game developers doing that?
|
| > Some of these replays represented more than half an hour of
| gameplay too.
|
| Same... That was really my main motivation: sometimes needing
| to play for 20 minutes before the bug would show up.
|
| > (we had a checksum of game state that we could check)
|
| Ooooh I love that: that is plain bad---! So you not only had
| your deterministic engine, but a way to directly identify any
| discrepancy between the original state and the replayed one.
| I didn't think about that!
|
| It's amazing that it let you identify bugs before they even
| stroke.
| dfan wrote:
| I don't remember hearing about other developers doing the
| same thing. I think it was just that I had a lot of
| experience debugging deterministic programs (like command-
| line tools) and it was infinitely more pleasant than trying
| to debug an interactive graphical program, so it was worth
| seeing whether we could make the game itself deterministic.
|
| I've never made a system that ambitious again, but one
| thing I've learned from that experience is to never ever
| call global rand(); always always create your own RNGs that
| you can run explicitly, and if you have multiple systems
| that can be disabled independently (e.g., we were able to
| turn off graphics during our replays), give them each their
| own RNG.
| muzani wrote:
| I dealt with one bug that was broken on 8 different points until
| it looked like another bug.
|
| It was paginated content. It would download a page in one class,
| concatenate it to the existing content. It then sent that page to
| the view class, which adds it to the bottom of the list.
|
| This would make duplicate content, e.g. ABCDABCDEFGABC... instead
| of ABCDEFG. Someone had the brilliant idea of filtering new
| content from the existing ones. So it would be ABCD+(ABCDEFG-
| ABCD).
|
| So for the most part it worked exactly as planned. But then there
| would be a point where the app modified data locally. Say, you
| add a comment on B. B becomes b. Now you have AbCD+(ABCDEFG-
| AbCD)= AbCDBEFG. Oops.
|
| In the real world, this was done over so many classes,
| superclasses and so on that it wasn't clear why it was randomly
| inserting B at certain points, and at which points it was doing
| this. The behavior performed exactly as tested, but it was just
| poorly designed behavior and we ended up spending a few weeks
| ripping out and rewriting the code for this.
| carlmr wrote:
| >In the real world, this was done over so many classes,
| superclasses and so on that it wasn't clear why it was randomly
| inserting B at certain points, and at which points it was doing
| this.
|
| If OOP gave us anything it's the joy of trying to piece
| together huge puzzles.
| wizzwizz4 wrote:
| Monolithic hell: 40 bugs in three files.
|
| Classes hell: three bugs in 40 files.
| nwatson wrote:
| This software phenomenon has led me to feel the "reverse" of this
| in real life, and to start assuming some wild things, e.g., (a) I
| know someone was recently in the room with me; (b) they have not
| left through the doors or the window; (c) there are no good
| hiding places in the room; (d) they've >> temporarily left my
| field of vision and when I turn back around I don't see them <<
| ... I immediately ponder the possibility they have disappeared
| (or been raptured) rather than just quietly walking to remain
| outside my field of vision as I turn around.
|
| So I start thinking that maybe impossible-in-real-life things
| actually have happened.
| runawaybottle wrote:
| I particularly love spending a few hours on a bug and then
| committing a one or two line fix. Makes me look like I do nothing
| at work.
| havkom wrote:
| Solved an "almost impossible" today with "0 hits on Google". Felt
| awesome - this is why I like programming.
| foobarbazetc wrote:
| I'm doing this right now with a bug in the Linux kernel. It takes
| something like 5 hours to repro with a very specific condition
| because it's a SMP race / ordering bug.
|
| I literally haven't slept properly since I hit the bug because
| once I find something like this I have to find a fix. lol.
| kr99x wrote:
| I'm "lucky" enough to deal with buggy hardware on a semi-regular
| basis (I start writing firmware before the hardware is finalized
| and run on prototypes), so I really do get bugs where the the
| input data and the logic are all completely correct and the
| hardware is at fault. You get to an add instruction with
| immediate data/no pointers, and somehow it gives you back bad
| data or hangs.
|
| On the one hand, yay, not my fault! On the other hand, HELL to
| debug. On the worst hand, it dramatically increases my
| willingness to SAY it must be a hardware problem, which is not
| always the case!
| kr99x wrote:
| Two "fun" examples:
|
| 1) System trying to boot would hang at seemingly random points.
| Could never be pinned down to a particular instruction, but
| _could_ be caught doing it when stepping through with attached
| hardware debugger. It just wasn 't consistent and never made
| any sense. Hang on an add. Hang on a call and never reach the
| first line of the thing being called. The hang would always be
| _relatively_ late in the boot, but that 's all that could be
| found.
|
| Eventually I got it. It would hang the first time a timer
| interrupt triggered, which would only happen after that
| interrupt was enabled something like halfway into the boot.
|
| Turns out there were disabled cores and the system was waiting
| trying to park those cores before servicing the interrupt, but
| they'd never respond/ack/say "I parked" and so we'd hang.
|
| Disable the interrupt and there was no problem.
|
| 2) Operating in Cache-As-RAM mode early in boot, no "real"
| memory, just the L2 cache mapped as memory. Two valid/available
| address ranges could not both be written to. Writing to 0xA and
| then 0xB, or 0xB and then 0xA, would hang the system. Data
| being written didn't matter. Writes didn't need to be back to
| back. Just couldn't play nice.
|
| Knowing it's a hardware problem spoils the fun of trying to
| debug that. Bad cache, couldn't properly convert addresses to
| cache lines, wrapped back on itself and panicked. Solution -
| move and resize "usable" cache region to exclude the
| overlapping ranges.
| com2kid wrote:
| Bus timing errors! Fun times!
|
| Forgot a wait state? It'll probably work, on most chips!
|
| Even better when suppliers fix, or add, bugs and don't tell
| you. Or change the firmware they are shipping on a part that's
| hanging off a UART. Or how about discovering that in the 21st
| century, one of your suppliers doesn't use source control for
| their firmware and every time they send you over a firmware
| blob it consists of some patches applied to whatever code
| happened to be laying around on some developer's machine!
| apples_oranges wrote:
| The more "impossible", the more interesting and rewarding in the
| end!
| dgb23 wrote:
| I disagree. When you're debugging a blackbox it can be even
| more frustrating to find out what the issue was. Often some
| unintuitive, inconsistency that was not documented. Those are
| not rewarding, because you learn very little from them except
| to be suspicious.
| mrfusion wrote:
| As I got better at coding, more and more of my bugs were check-
| the-plug situations. And it's just the worst because your brain
| excludes the easy stuff as a possibility.
| staticassertion wrote:
| > There are bugs where you know exactly how to reproduce it, but
| it takes a long time (like 20 minutes or way longer) to reproduce
| the bug. This is hard because it's hard to maintain your focus:
| maybe you can only try 1 experiment per day!
|
| Ugh I hate these. It's almost worse than the "only happens
| sometimes". I'd rather walk down a longer path fast than a
| shorter path miserably slow.
|
| > there's no output at all
|
| Hangs!! I hate hangs. I hate them. Crashes give information.
| Hangs could be so many things, but it is the _absence_ of
| information.
|
| > 4. one of your assumptions is wrong
|
| #1 assumption I always make sure to check now is "is the code
| that's running actually from the source code I'm looking at". So
| may "impossible" bugs I run into are in fact impossible... in the
| codebase I'm looking at. And it turns out what's deployed is some
| other code where it is very much possible.
| AlotOfReading wrote:
| At one point, I encountered a low level bug that would only
| manifest if the computer was physically located in a particular
| room. It ended up being an undocumented cabling revision, but I
| revisited a lot of assumptions about the nature of software and
| reality before getting to that point.
| cortesoft wrote:
| Sounds like the 500 mile email bug
|
| https://web.mit.edu/jemorris/humor/500-miles
| yummypaint wrote:
| I have come to love the feeling that the universe is broken
| when bug hunting. It means the solution is close at hand. If
| there are still lots of plausible explanations left then it
| means there is lots of work left to do. Things always seem
| most insane just before figuring it out.
| pineconewarrior wrote:
| Thank you for this positivity! I'll try to remember this
| next time I am frustrated.
| laurent92 wrote:
| And then you find it and you remember the sentence:
| "Everything that can fail, will fail".
| hutzlibu wrote:
| "but I revisited a lot of assumptions about the nature of
| software and reality before getting to that point."
|
| I encountered some bugs that felt like this (software only
| though), where I did not found the all explaining logical
| solution and ended up rewriting that part.
|
| But I saved the whole state of the project and data in a
| zipped file and keep them locked away - so one day, when I
| feel like madness (and maybe have better tools avaiable) I
| will jump into it, either to proof my own stupidity - or the
| existence of dark magic, poltergeister and demons.
| tyingq wrote:
| The ones I really hate are the ones that disappear when run
| under a debugger, or with higher logging levels. Usually race
| conditions.
| rubatuga wrote:
| Yep, GDB also regularly splits up TCP network packets, so
| beware of changing stream packet boundaries
| robotnikman wrote:
| Ah, the dreaded heisenbug
| amelius wrote:
| The solution is simple, though: ship with the debugger.
| laurent92 wrote:
| ...with the JVM debug port open, so you can unlock your
| customers when they are stuck.
|
| Even works for software embedded on rockets - After all,
| network latency to the moon is only 2s, right?
| lamontcg wrote:
| > Hangs could be so many things, but it is the absence of
| information.
|
| start adding information.
|
| you need to know what the code is getting to and/or what its
| not getting to and progressively narrow it down. spamming debug
| print statements can be useful if you have no better ideas to
| get yourself moving.
|
| or if you can manage to hook up an interrupt handler to be able
| to get it to dump a stack trace that'll immediately tell you.
|
| or attach a debugger while its spinning in order to get a stack
| trace.
|
| or attach something like strace to dump out system calls or
| ltrace to dump out libc calls. that can't catch processing
| spinning in "math" though, but that absence of information is a
| hint that eliminates lots of possibilities.
|
| you may need to add a feature to the software in order to do
| this kind of debugging, so you write your own USR1 interrupt
| handler or something. it will be worth it.
|
| i think the kids these days use stuff like systemtap?
| perryizgr8 wrote:
| > 20 minutes or way longer
|
| One of my coworkers was working on a bug that only manifested
| itself after ~2 weeks of putting the system under continuous
| stress. Bad times.
|
| Stress tests often seem to unearth strange bugs. Once when we
| used to sell boxes running a single-threaded real-time OS, the
| OS vendor gave us a separate machine that would sample the
| registers and memory of the box 10000 times a second (using
| jumper cables on the motherboard), in the hope that we would
| catch the exact state of the system as it crashed after days of
| stress testing. The sampler machine was a crazy overpowered
| beast of a machine with really high clock CPUs and oodles of
| RAM so it could dump as many samples as possible into a ring
| buffer in memory.
| poidos wrote:
| > the code that's running actually from the source code I'm
| looking at
|
| Super common when working with Clojure(script)! Bites me in the
| ass all the time.
| darksaints wrote:
| Speaking of clojurescript and bad assumptions:
|
| Clojure and clojurescript are both dynamically typed, but
| clojure is strongly typed (type errors throw exceptions), and
| clojurescript is weakly typed (type errors are technically
| valid code). Too many clojure enthusiasts act like they're
| both the same language, but that is an absolutely massive
| difference.
|
| Try the following code out in each: (+ "1" 1). In clojure you
| get an exception, in clojurescript you get "11" (maybe a
| warning at compile time which won't show it's face at runtime
| when the data is fed to you from an API call).
|
| That bug silently corrupted analytics data for 3 months for a
| service I worked on. That alone was the reason I stopped
| using dynamically typed languages.
| opportune wrote:
| It took me some time but I have a good workflow for debugging
| hangs/deadlocks/etc. If you know what part of the code produces
| it, you iterate over it indefinitely in a debugger until it
| hangs, then once you notice the iteration has stopped you "step
| in" to the debugger. Then you run another script that dumps the
| current trace back for each existing thread. That should be
| enough to detect the lock normally.
|
| The big problem is if you don't know how to reproduce the issue
| in a debugger for some reason.
| TeMPOraL wrote:
| Last time I had a "silent crash" heisenbug that produced no
| output, I solved it by system-level tracing (ETW, Event
| Tracing for Windows). It turned out not to be a crash at all!
| We had a script in our test harness that cleaned up stray
| processes, and that script sometimes managed to run long
| enough that it caught the start of the next test, and killed
| the tested executables before they managed to output
| anything.
| wooly_bully wrote:
| Your last point on the correct code running, ugh. I haven't
| gotten better at it even thought I'm aware of it more now. Had
| an incredibly persistent linting config issue just last week,
| turns out I had the wrong config open for editing in vim.
| steerablesafe wrote:
| The rise of reversible debugging for C/C++ made debugging much
| more enjoyable for me. For those tedious to reproduce bugs you
| can reproduce and record it once and replay it many times with
| the bonus of reverse-* operations. With rr the execution isn't
| even slowed that much.
| 2ion wrote:
| Indeed. I actually quite like hung programs :) Attach a
| debugger, and now lessee what it is doing...
| mateo411 wrote:
| I also hate hangs. If you are working with the JVM you can send
| it a signal kill -3 [pid] which can give you a thread dump.
| That's a QUIT signal if you are curious. I once found a stuck
| HTTP request when I did this, so I fixed the issue by adding a
| timeout to the HTTP connection object.
|
| When you are designing a system or a program you should have a
| way to get diagnostic information to make it easier to fix
| issues that can arise.
| pixl97 wrote:
| Always design in timeouts. And always make them configurable.
| mateo411 wrote:
| Yes, I agree.
| bennofs wrote:
| JVM also has jstack and other tools like jprofd to introspect
| running vms.
| hughrr wrote:
| If anyone is complaining about a few hours here and there,
| relax. It took 7 years for me to solve an intermittent one
| once.
|
| The issue turned out to be an immeasurably small thread
| synchronisation issue that went away when we bought new CPUs
| which were fast enough for it to not happen.
|
| In the end this task was rewritten in a single threaded process
| as the CPUs are now fast enough to complete the work on one
| thread.
| [deleted]
| vmception wrote:
| This hurts me to read
| hughrr wrote:
| That's less than 1% of the pain I experienced there.
| amalcon wrote:
| _> #1 assumption I always make sure to check now is "is the
| code that's running actually from the source code I'm looking
| at"._
|
| Once I got so far as stepping through some malfunctioning code
| at ASM level on a production machine, only to discover that the
| ASM had a jump if equal instruction while the source code
| prescribed exactly the opposite. I compared to the executable
| binary that was _supposed_ to be there, and it was different
| from what was actually there. There had been a bit flip
| somewhere along the line.
|
| I now spend more time than I probably should comparing hashes
| of any executable binary that's acting strangely enough.
| felipemnoa wrote:
| I've been there a couple of times.
|
| Relevant: https://xkcd.com/1316/
| lloeki wrote:
| >> there's no output at all
|
| > Hangs!! I hate hangs. I hate them. Crashes give information.
| Hangs could be so many things, but it is the absence of
| information.
|
| variants that made me scream include:
|
| - misleading information, the kind you can't dismiss as
| irrelevant, and you go down a thousand rabbit holes that
| seemingly lead nowhere, but it somehow all ties together into
| understanding the issue, only after a thousand turns and while
| keeping the holistic view of it all
|
| - useless information. kind of like the one above but every
| piece of information you get truly leads nowhere
|
| Once I had an actual crash that somehow happens before jumping
| into the program's code (like, before main), but changing your
| code does toggle the trigger... but when it crashes you have no
| core and can't attach a debugger, so you're left with debugging
| on the happy path and constantly think about what could
| possibly go wrong, and you don't even know what you're looking
| for. understanding why crt0/ld trips over is not fun.
| sillysaurusx wrote:
| My most impossible bug was a Direct3D issue. It was impossible in
| the sense that what we were seeing on the screen wasn't a result
| of the code we were writing.
|
| It turned out to be Direct3D's debug mode. Once we flipped it
| back to non-debug, everything started working again.
|
| I became a lot more skeptical of debugging tools after that day.
| bluedino wrote:
| Hah! One of the first things I learned programming was that you
| could never trust 'debug' and 'release' builds to behave the
| same. This was back in the late DOS, early Windows days.
| praptak wrote:
| Ambiguous title. I thought it's about learned helplessness in
| insects.
| parafactual wrote:
| "Feel impossible" is a pretty awkward way to say "experience
| learned helplessness".
| jollybean wrote:
| Given the enormous costs of some of these long tail bugs, it
| should be more evident that we have things in place to try to
| prevent them from happening in the first place.
|
| When I started in C++ I knew there was a lot I didn't know, but I
| assumed it would get sorted out over the life of the program.
|
| Now I'm almost afraid to write C++, or at least to step off the
| yellow-brick-road of tried and true things 'I know for sure'. I'm
| afraid for people on my team doing that and feel weirdly
| paternalistic when requiring my devs. to do the same. I feel like
| an adult explaining why we 'walk on the sidewalk not the road' to
| adults who actually do know the sidewalk is safer, but are
| probably unaware of the statistical liability of walking along
| the road even as an adult.
|
| As a developer - I love solving these problems. As a product
| leader they scare me to death - losing x% of your development
| time to never ending rabbit holes.
| lamontcg wrote:
| There's always a cause, nothing is impossible. Today might not be
| your day though.
| commandlinefan wrote:
| I got past feeling that bugs were "impossible" early in my career
| - if it happens in a computer, it can be fixed. What _is_
| impossible is providing the demanded estimate as to when such a
| bug will be fixed. But we have to anyway.
___________________________________________________________________
(page generated 2021-06-15 23:01 UTC)