[HN Gopher] Cosmic rays causing 30k network malfunctions in Japa...
___________________________________________________________________
Cosmic rays causing 30k network malfunctions in Japan each year
Author : uncleberg
Score : 149 points
Date : 2021-04-07 09:56 UTC (13 hours ago)
(HTM) web link (mainichi.jp)
(TXT) w3m dump (mainichi.jp)
| hyko wrote:
| Is _appeal to cosmic rays_ a candidate for an engineering logical
| fallacy?
| KWxIUElW8Xt0tD9 wrote:
| How many are caused by Windows? :-)
| Serenntop wrote:
| We could reduce the cosmic ray flux by thickening the atmosphere.
| A small increase in normal air density would give us a
| significant reduction in cosmic rays. Talk to the terraformers.
| They've given a lot of thought to how to make air.
| fennecfoxen wrote:
| > Soft errors occur when the data in an electronic device is
| corrupted after neutrons, produced when cosmic rays hit oxygen
| and nitrogen in the earth's atmosphere, collide with the
| semiconductors within the equipment.
|
| Uhhhhhhh. Neutrons? Doubt goes here. I was pretty sure _charged_
| particles and ionizing radiation are to blame for messing up your
| electronics.
| fennecs wrote:
| Supposedly yes neutrons [0 they give a link under causes]. I
| guess neutrons are more penetrating than protons, can pass
| through chips/buildings etc more easily, and even though weak
| interaction, it ends up more significant contribution than
| protons etc that interact more easily.
|
| [0] https://en.m.wikipedia.org/wiki/Soft_error
| yummypaint wrote:
| One of the ways high energy neutrons interact is by scattering
| protons elastically. Their masses are essentially the same so
| the neutron can transfer its momentum very efficiently. These
| scattered protons (perhaps from plastic in the packaging etc)
| are then the charged particles that flip bits. These protons
| are generally lower energy than cosmic ray protons, which means
| they deposit MORE energy per distance traveled, so they can be
| worse than cosmics directly.
| zaarn wrote:
| Neutron radiation is in fact ionizing radiation.
|
| It is much more dangerous than other types, as it readily
| passes most materials, yet dosage is absorbed. Absorbed
| neutrons cause particles to (almost always) become radioactive,
| which causes havoc as those new radioactive particles decay.
|
| The other common types of radiation (alpha, beta and gamma) do
| not cause things to become radioactive (usually), instead they
| are a threat if the actually radioactive material accumulates
| as dust on your skin (for example).
|
| Neutron radiation is much more dangerous because it seeds
| radioactivity deep in the body, so you can't clean it off by
| washing or waiting for the absorbed particles to be flushed
| back out of the body naturally within a few days.
| notum wrote:
| Cosmic rays are, or rather were, the reason why manufactures
| avoid shipping high end cameras by plane, cosmic rays kill pixels
| in imaging sensors: https://youtu.be/98FZ8C6HneE?t=479
|
| It's also the reason why all ISS footage has dead pixels on it.
| andy_ppp wrote:
| Very interesting, so should people who travel with photography
| equipment take night flights if possible?
| notum wrote:
| Rays hit our planet regardless of Sun, if anything it might
| block a few. They are caused by ultra-energetic cosmic
| events, such as star collisions, black hole feeding cycles.
|
| Cosmic rays are just very potent photons, capable of knocking
| out electrons from an atom (meaning they are ionizing)
| causing havoc in precision electronics and, well, our DNA.
| moftz wrote:
| Cosmic rays can also be high energy ions. Most of the
| photons come from the sun and the ions can come from
| outside the universe and the sun.
| andy_ppp wrote:
| What is the tolerance for cosmic rays for a human... I ask
| because on the journey to Mars you'll get quite a few I
| guess and also on the planet's surface after you're there
| with such a thin atmosphere?
| btkramer9 wrote:
| As far as cancer rates go I've heard it's about as bad as
| taking up smoking cigarettes. Which is not great but
| doesn't seem like a show stopper
| tinco wrote:
| This is actually a thing people are criticizing Musk for,
| he is being accused of sending people to Mars with
| insufficient shielding against cosmic rays.
|
| In my opinion it's a bit silly argument, as there's a
| whole bunch of other risks and quality of life sacrifices
| made by the persons who are going to undertake that
| journey. Some raised chance of cancer is probably the
| least of their worries.
| mlindner wrote:
| He can't send people to Mars yet so how are people
| accusing him of sending people to mars with insufficient
| shielding? There are limits to criticism of Musk one
| would think.
| typon wrote:
| Humans going to Mars should expect to never come back
| andi999 wrote:
| But at least they should reasonably expect to arrive.
| bostonsre wrote:
| Anyone know what expectations Columbus or other
| expeditions had about survival/risk?
| [deleted]
| iso1631 wrote:
| For a normal NASA flight
|
| "NASA told Business Insider it estimates there's a
| 1-in-276 chance the flight could be fatal and a 1-in-60
| chance that some problem would cause the mission to fail
| (but not kill the crew)"
|
| That's fairly low - feels like the age of sail would be
| far higher -- Columbus's second voyage alone had about 25
| deaths just from scurvy. His first voyage was only about
| 90 crew.
|
| But remember that life wasn't exactly easy on land
| either.
| tinco wrote:
| A lot less than an astronaut has when going to Mars for
| sure. We at least know how far away Mars is exactly, and
| what obstacles lie in the way, and what inhabitants it
| has. They just got on a boat, and pray for fair wind and
| good weather so they might reach india, and thank god
| there was a whole other continent in between.
| andi999 wrote:
| I remember vaguely that if there would have been a sun
| eruption at the lunar mission the team might have died. I
| tried a while ago to find a good source on that though.
| If this were true, forget Mars.
| Sanzig wrote:
| That's actually a fascinating question! It turns out we
| don't know enough to answer one way or another.
|
| The model commonly used in radiation protection to assess
| cancer risk with respect to radiation dose is called the
| Linear No-Threshold (LNT) model [1]. The model critically
| assumes that (1) total radiation dose is the only
| predictor of cancer risk, and (2) _any_ radiation
| exposure results in an increased cancer risk.
|
| This model works at high absorbed doses, however, its
| applicability is highly controversial when used with low
| abosorbed doses or with relatively high absorbed doses
| that occur over a long period of time (ie: low dose
| _rate_ ).
|
| The thing is, the human body has built-in defense
| mechanisms against cancer such as DNA repair. There is a
| good body of evidence that small doses and low-rate
| exposures do not result in cancer risk (ie: there is a
| threshold absorbed dose and probably also a threshold
| dose rate), but the model does not account for this.
|
| This is particularly problematic when trying to assess
| excess mortality from things such as radiological
| accidents: when you multiply the small LNT-predicted risk
| for a low dose times a very large population, you end up
| with a lot of cancers. This is one of the reasons you'll
| see estimates for deaths from the Chernobyl accident vary
| by orders of magnitude.
|
| It's also problematic when assessing something like a
| Mars mission: yes, the astronauts would get large
| cumulative doses, but the dose _rate_ is pretty low over
| most of the mission (other than during high dose rate
| solar events where they would need radiation shielding).
| How much of an elevated cancer risk is it actually?
| Nobody is quite sure.
|
| [1] https://en.wikipedia.org/wiki/Linear_no-
| threshold_model
| foxyv wrote:
| Cosmic rays are not exclusively associated with the sun.
| There are a lot of Galactic Cosmic Rays still. It's pretty
| much leftovers from supernovae, black hole mergers, neutron
| stars, accretion disks and such. But yeah, at night would be
| better.
|
| Edit: Correction, there is very little difference and fewer
| cosmic rays during the day. Source:
| https://arxiv.org/pdf/physics/0105005.pdf
| woodruffw wrote:
| This is a fascinating claim (and video), but I can't find a ton
| of corroborating evidence for it: searches for "digital camera
| cosmic rays" bring up (1) considerations for cameras in space (
| _much_ higher background radiation levels than a commercial
| flight), and (2) consumer photography forums where people are
| discussing that 2011 talk. Some of the latter seem to think
| that Kodak has /had a vested interest in spreading fear around
| digital photography, since their analog division was imploding
| at the time.
| jmcqk6 wrote:
| Does anyone know how this works with our space-based
| telescopes? Or even our mountain top observatories? Surely if
| it's this crazy, they have to deal with that as well?
| supernova87a wrote:
| Well one thing they also do is that pretty much any desired
| image is taken multiple times/multiple exposures, and
| multiple offset positions in close succession to remove
| exactly such artifacts. The images are then stacked, etc.
|
| So this allows removal of effects that do not correlate with
| what is physically should be in the image, but is an artifact
| of the sensor, image system, etc:
|
| -- sensor artifacts: dead pixels, flat field irregularity,
| pixel response variations, electronic noise
|
| -- imaging system issues: optical problems, lens/mirror
| defects
|
| -- and then exactly what's being discussed here: cosmic rays,
| transient objects (satellite tracks!)
| reportingsjr wrote:
| Yes, the electronics, including imaging sensors are all
| radiation hardened.
|
| This can involve things such as using digital circuits that
| are radiation resistant (e.g. look up radiation resistant
| flip flop). Using multiple computers running the same thing
| that all "vote" for the correct result, so if one computer
| has an error from radiation you don't suffer. Using
| semiconductors that are more resistant to radiation (larger
| band gaps mean more energy required to flip a bit).
|
| Physical shielding is key as well. The infrared imager on the
| Cassini probe had a case made out of tantalum, as tantalum is
| a very dense material which prevents a lot of radiation from
| going through it.
| [deleted]
| jstrieb wrote:
| This reminds me of Bitsquatting, which is like typosquatting but
| uses domains one bit-flip away instead of one mistyped letter
| away.[0] In a small experiment, for some popular domains, the
| author reportedly got a lot of hits.
|
| In the associated DEFCON talk, the author notes that even if you
| use ECC memory, it's unlikely that the DRAM in your hard drives
| does too.[1]
|
| 0: http://dinaburg.org/bitsquatting.html
|
| 1: https://www.youtube.com/watch?v=9WcHsT97suU
| stuartcw wrote:
| I have always wondered whether new ideas are seeded by cosmic
| rays disturbing neurons in the brain..
| gotostatement wrote:
| seems too low-level, no? the temporary excitation of a single
| neuron would most likely just fade away into the equilibrating
| process of the brain
| CameronNemo wrote:
| >In chaos theory, the butterfly effect is the sensitive
| dependence on initial conditions in which a small change in
| one state of a deterministic nonlinear system can result in
| large differences in a later state.
|
| https://en.wikipedia.org/wiki/Butterfly_effect
| gotostatement wrote:
| Is there reason to believe the brain is chaotic on that
| level? The reason I doubt it because any stable biological
| system has as one of its primary tasks - if not its #1
| primary task - to maintain equilibrium. Otherwise it would
| collapse into chaos due to the inherent imprecision of
| molecular phenomena. So I would be very surprised if you
| could stimulate a neuron - not even a neuron, but an
| electron inside a neuron - and cause something as complex
| as an idea to form. More likely it would be equilibrated
| away
| deepsun wrote:
| That'd mean people underground (or under thick layer of
| concrete, like in skyscrapers) are not creative.
| supernova87a wrote:
| But grad students are often sent to work in the basement, and
| yet they need to have the most ideas!
| moistbar wrote:
| I guess you could say they'd be...
|
| Living under a rock.
|
| Yeah, yeah, I'm leaving. Put the weapons away...
| dleslie wrote:
| Sounds like we have a hypothesis and a test environment; now
| we just need a test for creativity and we can start doing
| science!
| [deleted]
| adenozine wrote:
| That's actually mind blowing to think about. Thanks for sharing
| this.
| SketchySeaBeast wrote:
| > That's actually mind blowing to think about.
|
| That would be too many cosmic rays.
|
| And the "flipped the wrong neuron" possibility is much
| scarier in my mind. And probably in everyone else's brain as
| well.
| AlanSE wrote:
| I suspect that modern processors are closer to quantum limits
| than what neurons are.
| sosuke wrote:
| This is a great place to ask. Are our brains susceptible to this
| kind of interference and if so do you think the structure evolved
| to handle some corruption of this kind?
| dole wrote:
| Cosmic bitflip is like the Godwin's Law of BOFH excuses.
| neonological wrote:
| https://www.wnycstudios.org/podcasts/radiolab/articles/bit-f...
|
| A more comprehensive an interesting take on the cosmic ray
| phenomenon. It's actually a global issue that is accounted for in
| certain electronics.
| arbitrage wrote:
| Ahh, the ol' Cisco get-out-of-jail-free card.
|
| Headline would be more accurate if it said "30K unidentified bugs
| blamed on unverifiable phenomenon each year in Japan".
| 1-6 wrote:
| I'm speculating here but it would be more likely that software
| bugs cause more errors than cosmic rays.
| dnautics wrote:
| Found the rust programmer.
| marcosdumay wrote:
| The rust programmer is the one out there ending that phrase
| with "and this a problem that must be solved".
|
| ...
|
| Well, the rust programmer and the nuclear engineer,
| possibly.
| 1-6 wrote:
| As we move to 5nm chips, I wonder if there's a greater likelihood
| of errors from background radiation.
| shiftpgdn wrote:
| Kind of a fun ancedote: In "the old days" when you'd run a server
| room in your office we had a very large HPC cluster + a
| significant amount of storage and other one off servers on the
| top floor of a mid rise office building. I eventually moved it
| all to a former nuclear fallout facility where our systems were
| three floors down and under a gigantic pool of water. Error rates
| and random crashes fell off IMMEDIATELY. I believe Microsoft
| reported similar findings with their submerged data center.
| sandworm101 wrote:
| Wonderful. Yet another reason for the IT office to be relegated
| to a sub-basement.
| yunohn wrote:
| Why the whole IT office? Just the hardware running the
| software, not the humans who develop it.
| sandworm101 wrote:
| Most offices/businesses have nothing to do with software
| development. Most IT shop are in basements, or at least on
| lower levels near the core servers and other infrastructure
| they manage. "The IT Crowd" was a stereotype but was still
| based on the reality of most businesses. I have never seen
| or even heard of an IT shop with a skylight. Even windows
| are rare.
| wmichelin wrote:
| I feel like OP was joking
| lolthishuman wrote:
| And people still think the sun is nuclear. Ha!
| digikata wrote:
| A reduction in floor and building vibration might have
| contributed too?
| oconnor663 wrote:
| Or for example, old cables and improperly installed cooling
| fans might tend to get fixed during a move. Hard to know for
| sure.
| shiftpgdn wrote:
| Well yeah, that's why it's presented as an anecdote. :)
| TTPrograms wrote:
| Can you give OOM of any of the improvements? Eg. crash rates
| halved?
| kevin_nisbet wrote:
| Yea, when looking into these myself I did float in conversation
| that we'd need build a giant pool over our data centers. It was
| more of a joke though and never went anywhere.
|
| Although our error rate was more like 1-2 per week on the
| equipment I was looking at.
| brobinson wrote:
| I have no experience with tracking this kind of thing. How do
| you do it, what kind of analytics/tracking program is used,
| etc.?
| wang_li wrote:
| Sun's UltraSPARC II CPU modules were claimed to be particularly
| susceptible to cosmic rays circa 2000.
| phonon wrote:
| https://www.forbes.com/forbes/2000/1113/6613068a.html?sh=28e.
| ..
|
| https://www.computerworld.com/article/2584471/q-a--
| mcnealy-d...
|
| http://www.cedix.de/VorlesMirror/Band1/UnsafeAtAnySpeed.pdf
| p_l wrote:
| Not so much susceptible as "came with their own radioactive
| source"
| suifbwish wrote:
| Awesome. Next time clients call about their services being down
| I am totally using this one.
| belter wrote:
| They have to do two things :-)
|
| 1) Use ECC memory
|
| 2) Go underground
|
| "One experiment measured the soft error rate at the sea level to
| be 5,950 failures in time (FIT = failures per billion hours) per
| DRAM chip. When the same test setup was moved to an underground
| vault, shielded by over 50 feet (15 m) of rock that effectively
| eliminated all cosmic rays, zero soft errors were recorded.[6] In
| this test, all other causes of soft errors are too small to be
| measured, compared to the error rate caused by cosmic rays."
|
| "Soft Errors"
| https://en.wikipedia.org/wiki/Soft_error#Cosmic_rays_creatin...
| ourcat wrote:
| A 'Faraday Cage' should also help.
|
| edit: No idea why the downvotes. Faraday cages have long been
| thought of as a way to protect electrical devices from the
| electromagnetic waves which can be a result of solar flares.
|
| I even chatted to someone from NASA's Solar Dynamics
| Observatory about it in the past.
| belter wrote:
| Because a Faraday Cage would make it worst not better:
|
| https://www.reddit.com/r/askscience/comments/1fsiv9/how_effe.
| ..
| titzer wrote:
| Cosmic rays are not electromagnetic waves. They are highly
| energetic particles, like protons and naked helium nuclei.
| raisedbyninjas wrote:
| 5950 failures per billion hours is approximately equal to 1
| failure every 19 years.
| nightcracker wrote:
| And if you have a datacenter with 200 machines, that's
| roughly once a month.
| contravariant wrote:
| That's a bit more than just 'go underground'. I've seen people
| dig for cables but I've never seen them dig a hole 15m deep.
| tyingq wrote:
| Polyethylene is supposedly good at blocking cosmic rays. It
| would be funny to me if the fix was just a drop ceiling full of
| old grocery bags.
| robbmorganf wrote:
| Have a source? That seems like an awesome way to recycle
| grocery bags.
| 1-6 wrote:
| https://www.nature.com/articles/s41598-017-01707-2
| supernova87a wrote:
| Water is also a good protector. And the polyethylene is
| actually a proxy for "hydrogen atoms". The reason is that
| dense hydrogen has a lot of protons to interact with the
| incoming radiation.
|
| But unfortunately plastic bags are not dense polyethylene.
| You would kind of need full blocks of solid polyethylene...
| teachtwolearn wrote:
| You can make solid polyethylene out of plastic bags by
| pressing them in a mold heated to ~100-150C.
|
| But it's quite flammable, so you might not want to use it
| as a building material.
| tyingq wrote:
| The peer comment posted a source that has lots of
| references.
|
| Though most of them are tests in space, where I assume the
| thickness requirements would rule out grocery bags. I am
| curious how thick a layer of HDPE you would need on earth
| to make any notable difference.
| ortusdux wrote:
| That linked nature paper seems to indicate 10g/cm^2 for a
| 50% reduction. A standard shopping bag is about 5g, so
| you would need roughly 20k bags/m^2, or ~2k bags/ft^2. At
| 0.9g/cm^3, that would be roughly 11cm of solid
| polyethylene
| tyingq wrote:
| Ah, so bags are out, as is practically the ceiling. But
| cut sheet HDPE is easy to find, so tiles atop your
| machine would be relatively cheap and easy.
| wyldfire wrote:
| Doesn't sound like that solution is plenum-rated.
| 1-6 wrote:
| Nice, I'm going to line the rooftops with Tyvek now.
| kevin_nisbet wrote:
| > 1) Use ECC memory
|
| Not exactly. When I was in telco, where I had this problem was
| in FPGA's, we had all ECC memory and I never linked any
| problems to bit flips in RAM. But as I remember, the FPGA's we
| had were using a type of SRAM cell, but because it's not a
| memory module the FPGA programming could bit flip. So the
| product had a checksum function, that read back the program on
| a cycle and reset itself if the program no longer matched the
| checksum. So we would see 1-2 crashes / restarts per week in
| our FPGAs that we believe were bit flips.
|
| We then ran an anlysis on any of these that higher than
| expected error rates to try and identify actually bad hardware
| and replace them.
|
| I think the vendor eventually came up with a way to reprogram
| the FPGA without just crashing and rebooting the entire board.
| belter wrote:
| Interesting and makes sense. Do you have any additional
| references you would suggest and particularly in the context
| of FPGAs ?
|
| Would you say this quick reference is a good overview ? https
| ://www.intel.com/content/dam/www/programmable/us/en/pdf...
| kevin_nisbet wrote:
| Sorry, I should have mentioned this was quite a few years
| ago, so I'm very out of date. So I don't have any known
| good references that are handy. That link you shared seems
| pretty good on a quick scan through and inline with what I
| remember, I'm pretty sure I dug up similar resources for
| other vendors, including one I think was looking at
| satellite hardware.
| viraptor wrote:
| Compaq/HP handled this with many redundant resources / cores:
| https://en.wikipedia.org/wiki/Tandem_Computers
| roamingryan wrote:
| Many modern FPGAs now include dedicated logic for config SRAM
| "scrubbing." This logic continuously checks config frame
| checksums to identify upsets. These can then be fixed in
| real-time either using the error correction properties of the
| checksum technique, or from the non-volatile config memory
| (typically NOR flash). It's also important to note that only
| a subset of the SRAM config bits are critical for a given
| application. Usually this is a small percentage of the
| overall array.
|
| https://www.xilinx.com/support/documentation/application_not.
| ..
|
| If even higher levels of reliability are needed, there are
| rad-hard-by-design FPGA families (e.g. Xilinx Virtex 5QV).
| These have a special config SRAM cell that has more charge
| storage sites than a conventional SRAM cell. It is less area
| efficient than a conventional SRAM cell, but geometry of the
| charge storage sites ensures that a single cosmic ray can't
| flip the state of a majority of them. Essentially the cell
| can self-correct, no scrubbing required.
___________________________________________________________________
(page generated 2021-04-07 23:01 UTC)