[HN Gopher] NASA Pulls Off Delicate Thruster Swap, Keeping Voyag...
___________________________________________________________________
NASA Pulls Off Delicate Thruster Swap, Keeping Voyager 1 Mission
Alive
Author : Stratoscope
Score : 251 points
Date : 2024-09-12 03:22 UTC (19 hours ago)
(HTM) web link (gizmodo.com)
(TXT) w3m dump (gizmodo.com)
| qingcharles wrote:
| Great to see they bought her some more time.
|
| I watched this excellent Voyager documentary recently:
|
| https://www.itsquieterfilm.com/
| a_e_k wrote:
| I continue to be fascinated by how they:
|
| 1. Are able to diagnose the problems remotely at such distances
| and on such old hardware. How can they even measure the thruster
| tube apertures here?
|
| 2. Decide what actions to take. It's not like they have a local
| test device to experiment on is it? (Even if they did I can't
| imagine how they'd reproduce the conditions of the real thing.)
| And if they choose poorly, I'd assume the mission's over. There's
| no replacing Voyager 1 if they brick it.
|
| 3. Have such fine control over the hardware. For something built
| in the 70's when RAM was largely measured in kB, they seem to
| have an insane amount of flexibility to remotely reconfigure the
| equipment. Whatever they did, there must have been some real
| foresight.
| big-green-man wrote:
| What blows my mind is the organizational knowledge needed after
| so many years to keep it going. You don't just hand a guy some
| man pages when he comes onto the project. I'm sure people have
| aged out, yet they still understand the complexities in the
| design. That is something we need to understand and prioritize
| in the systems we build today.
| tommiegannert wrote:
| Startup: "Do you remember when we inserted this quirky module
| running in AWS? We can use that to implement this next
| feature. That was a useful decision!"
|
| Voyager: "Do you remember when our parents inserted this
| quirky module that has since left the solar system? We can
| use that to turn it off and on again. That was a mission
| critical decision!"
| exe34 wrote:
| that aws module isn't supported anymore though, we need to
| npm install half the internet for the 2 line library that
| replaces it.
| HeatrayEnjoyer wrote:
| Not parents for most of us. Grand or great-grandparents.
| The senior engineers were highly educated and experienced,
| implying ages 30-50s during development. They are in their
| 80-100s now.
| tverbeure wrote:
| When Voyager failed last year with a CMOS memory error, one
| of the big problems was that a bunch of low level information
| was gone or conflicting. For example, they sometimes had to
| guess assembler instructions because the code printouts were
| low quality photocopied pages. Or because there were
| handwritten comments or comments scratched out with pencil
| without any clue about why it was done.
|
| One saving grace was the fact that the architecture and the
| code space was simple enough so that they could reason
| through the symptoms and actions to take, something that
| would have been much harder with modern spacecraft.
|
| Check out this amazing talk:
| https://youtu.be/dF_9YcehCZo?si=W_b3NJ7vgxaYS1__
| methuselah_in wrote:
| This thing hits me. True.
| tomooot wrote:
| I would imagine if the design/assembly information was broadly
| available (internally) in the past, there's probably one or
| several "digital twin" emulations of the craft, or at the very
| least specific subsystems of it's computing resources. There
| must be some kind of analog/simulation of it's software just
| for proving "bugfixes" before upload, like the coms error and
| subsequent setting of the "solar system record" for "furthest
| distance remote code update" earlier this year.
| onedognight wrote:
| There isn't a simulator or digital twin for voyager. It has a
| bespoke processor made with 74* style logic. One guy will
| puts together a command and they will have a review where the
| other engineers will try and independently verify it. Then
| they copy and paste the command somewhere to "run it". It
| happened, fairly recently, that the command had a typo that
| was caught in review, but the "wrong" pre-review command was
| used and the attitude became off by so much that they lost
| contact. It was only by cranking up the power at Goldstone
| that they got a command through. This fundamentally changed
| their understanding of the largest angle for which they
| _could_ still communicate with the spacecraft. They just
| hadn't wanted to try larger angles before because it was too
| risky.
| wildzzz wrote:
| That's the great thing about such a simple design, you can
| actually sit down with pen and paper and verify operations.
| tverbeure wrote:
| Forget about a digital twin, they don't even have an
| assembler for the CPUs. It's all hex values.
| basementcat wrote:
| In short, the Voyager spacecraft were designed (after
| considerable design and operational experience from Mariner and
| Pioneer spacecraft) to be able to operate for a long time
| largely automatically without too much hand-holding. All major
| systems are multiply redundant and may be remotely turned on
| and off. While there has been personnel turnover on the
| program, it has not been of a magnitude to jeopardize program
| continuation. Finally, program management has been media savvy
| and well politically connected ensuring that operations are
| still funded. (Contrast with other missions such as Magellan to
| Venus which was deorbited while it still had propellant reserve
| leaving some portions of the planet unmapped)
|
| JPL has (or had?) a "retiree badge" program that permitted
| retired staff to continue to access their office. Many programs
| benefited from highly knowledgeable personnel essentially
| continuing to report to the office every day without pay (not
| being paid comes with the luxury of not having to worry about
| being laid off if your charge accounts don't have enough
| funds!) It was an absolute privilege to learn from these
| people.
| SSLy wrote:
| So, culture preservation is important for success of highly
| technical endeavours. Don't tell it to your run of the mill
| MBA.
| basementcat wrote:
| I think it is inevitable that organizational "culture"
| changes. The tricky part is figuring out exactly what parts
| need change and what parts don't.
|
| For example JPL used to have beauty pageants ("Miss Guided
| Missile"). More recently, management appears to be trying
| to adopt policies and procedures from some venture funded
| commercial space companies. It is not clear how helpful
| these efforts will be given that these organizations are in
| fundamentally different businesses.
|
| https://www.jpl.nasa.gov/images/slice-of-history-70th-
| annive...
| ozim wrote:
| Run off the mill MBAs are not allowed anywhere near to
| stuff as important as Voyager.
|
| All the web apps, that run of the mill devs are
| implementing are nowhere near as important as voyager.
|
| I think we are fine ;)
| darby_nine wrote:
| It also helps that JPL is a public good and not a wealth-
| extraction machine
| me_me_me wrote:
| but if someone is paid $0 it must mean they are worthless,
| surely the excel file doesnt lie
| efitz wrote:
| No, you need to think like an MBA.
|
| It means that they are the most valuable employees,
| because their productivity per dollar is infinite.
|
| It also means that when you sort all the (non-executive,
| of course!) employees by total comp in preparation for
| layoffs to make the budget look better, they are at the
| bottom of the list.
| me_me_me wrote:
| oh yes, you are absolutely right :D
| tecleandor wrote:
| Yeah! Being almost 50 years old it's not like people is not
| working there anymore but that probably a bunch of people in
| the original project has already died!
|
| Great forward thinking
| mistermann wrote:
| Any idea how many people took advantage of the retiree badge
| program, or any individual people who continued to put in
| substantial hours?
| wildzzz wrote:
| We do something similar at my job for the greybeards that
| were very influential on our current projects. They
| "retire" but are retained as a very part time employee that
| only get paid if we need to bring them in for a day or two
| to help answer some questions. The managers love it because
| they don't need to find work for this person, pay for any
| benefits, or need to get approvals to pay them like
| contractors. As long as we keep doing relevant work to
| their expertise, they will continue being retained. There
| is a limit for how many hours they can put in because these
| people are incredibly expensive as they usually retire at
| the top engineer level and retain the equivalent hourly
| wage for that position.
| olabyne wrote:
| And quite recently they also recovered from a faulty ram
| module, with a radio link speed that is maybe below the kbit/s
| now !
| lisper wrote:
| And you should see the ping times.
| liamwire wrote:
| I suspect it's far lower than that.
| stordoff wrote:
| > Uplink communications is via S-band (16-bits/sec command
| rate) while an X-band transmitter provides downlink telemetry
| at 160 bits/sec normally and 1.4 kbps for playback of high-
| rate plasma wave data.
|
| https://science.nasa.gov/mission/voyager/spacecraft/
|
| The data playbacks were initially transmitted at 7200 bps,
| but this was dropped to 1400 bps in 2007
| (https://voyager.gsfc.nasa.gov/Library/DeepCommo_Chapter3--
| 14... - page 72).
|
| RTT ~46 hours (one-way light time 22:49:59, per
| https://science.nasa.gov/mission/voyager/where-are-they-now/)
| selimnairb wrote:
| > It's not like they have a local test device to experiment on
| is it?
|
| I would bet they have one or more simulators ("digital twin" in
| the parlance of our times). I'd want one simulator to always
| reflect the current state of the probe (with state data
| assimilated periodically from the real probe). Then other
| simulators can be used to test management changes to see how
| the system might respond.
| onedognight wrote:
| > I would bet they have one or more simulators
|
| Sadly they do not, but they are starting to write one.
| ForHackernews wrote:
| There's a documentary "It's Quieter in the Twilight" about the
| team keeping the Voyager missions alive:
| https://www.youtube.com/watch?v=8vJT8AW0wYw
| ornornor wrote:
| Looks very interesting , thanks for sharing
| whycome wrote:
| Included with Amazon Prime in Canada
| tverbeure wrote:
| You should totally check out the talk from a few weeks ago that
| I link to in my other comment. It answers your 3 questions and
| more.
| japanuspus wrote:
| The Voyager mission is such a wild achievement. Both the sublime
| design and craftsmanship that must have gone into the hardware,
| and the deep institutional knowledge required to keep it running
| is awe-inspiring.
| litoE wrote:
| It's been operating for 47 years and it still has fuel left to
| make attitude corrections. I wonder how they managed that feat.
| turblety wrote:
| I presume solar power?
| Tor3 wrote:
| No, way too little sun out here. The sun is just a dot far
| away. The Voyagers both use RTGs, radioisotope thermoelectric
| generators. Decaying plutonium, essentially. That's for the
| electrically powered systems. The thrusters use liquid
| hydrazine, which is common for those kind of thrusters.
|
| Edit: There's more about that in the NASA link in a sibling
| comment.
| krige wrote:
| Solar has nothing to do with it. Voyager uses hydrazine, of
| which over 80% has been used up over the years. They simply
| use not that much of it as it's not for thrust, but for
| aiming at Earth more or less.
| meindnoch wrote:
| Lol. How would that work exactly?
| JumpCrisscross wrote:
| > _How would that work exactly?_
|
| Light sails. Laser propulsion. Photon rockets. None of
| which apply to Voyager. But none of which are laughably
| dismissible.
| gpderetta wrote:
| Even with something like an ion engine (which I'm not sure
| were available when Voyager was launched), you would need
| leftover reaction mass.
| Dennip wrote:
| IIRC They have some type of nuclear power. Not like a reactor
| but something much simpler that generates heat from radioactive
| decay.
| robbiep wrote:
| the flow of electrons to allow systems to work is not the
| same as the expulsion of gasses to provide thrust
| reacweb wrote:
| https://xkcd.com/2115/ The RTGs generated about 470 W of
| electric power at the time of launch. In 2023, it was 260W.
| magicalhippo wrote:
| Not to downplay how impressive the Voyager probes are, but it
| seems they packed a fair bit of hydrazine. From "Engineering
| the Voyager Uranus mission":
|
| _While it was not a design requirement, the option for an
| extended mission past Saturn was always protected, unless it
| meant compromising a major mission objective at Jupiter or
| Saturn._
|
| _Even though the probability of Voyager 2 lasting another five
| years was calculated to be in the range of 60 to 70 percent --
| well below NASA 's usual acceptable probability-of-success
| threshold -- the decision was made to send Voyager on to
| Uranus._
|
| _After its Uranus encounter, Voyager 2 still carried 48% of
| the hydrazine initially loaded into its tanks, eight-and-a-half
| years before._
|
| [1]: doi:10.1016/0094-5765(87)90096-8 (can be found on the hub
| of science)
| NaOH wrote:
| Probably a better link:
| https://science.nasa.gov/missions/voyager-program/voyager-1/...
| perihelions wrote:
| - _" A fuel tube inside the thrusters has filled up with silicon
| dioxide, a side effect of age within the spacecraft's fuel
| tank."_
|
| Where the heck do you get SiO2 from on a spacecraft? Some kind of
| silicone?
|
| edit: _" clogged with silicon dioxide, a byproduct that appears
| with age from a rubber diaphragm in the spacecraft's fuel
| tank"_[0] --I'm guessing that is a silicone rubber. I didn't know
| that rubber can decompose into sand.
|
| [0] https://science.nasa.gov/missions/voyager-
| program/voyager-1/...
| tecleandor wrote:
| Seems like silicone rubber is extracted from SiO2, so maybe
| it's impurities? Or maybe the Si and O are doing something to
| leave the polymer? (I have no idea if that last thing is even
| possible)
|
| https://en.wikipedia.org/wiki/Silicone_rubber
| numpad0 wrote:
| They expel the Hydrazine(N2H4) fuel out of a spherical Ti tank
| by inflating a rubber balloon that involve Teflon inside the
| tank using helium supply. I guess N2H4 was potent enough to
| degrade even those space age materials.
|
| 1:
| https://ntrs.nasa.gov/api/citations/19810001583/downloads/19...
| onewheeltom wrote:
| Just another day at the office for the Voyager 1 team. Wow.
| Buttons840 wrote:
| I've heard good things about some Voyager documentaries, and I've
| wanted to watch one with my daughter, but NASA keeps making the
| documentaries out-of-date and incomplete. How many amazing
| stories will there be to tell by the time Voyager is truly beyond
| our knowledge?
| Twisol wrote:
| I had the _extreme_ pleasure of seeing Bruce Waggoner of the
| Voyager team give a keynote at !!Con just last month. The
| recording landed on YouTube just a couple days ago, so this is
| great timing: https://www.youtube.com/watch?v=dF_9YcehCZo
| tverbeure wrote:
| 2 weeks ago, Bruce Wagoner from the Voyager program gave a talk
| at !!Con about how they recovered from the CMOS memory issue that
| they had a year ago.
|
| It's basically blind debugging with a latency of 45 hours.
|
| The talk is amazing and goes through the computer architecture of
| the spacecraft as well as the challenges of dealing with
| something that is so old, with so documentation that has
| conflicting information or unreadable etc.
|
| https://youtu.be/dF_9YcehCZo?si=W_b3NJ7vgxaYS1__
| ChrisArchitect wrote:
| Official release: https://science.nasa.gov/missions/voyager-
| program/voyager-1/...
|
| (https://news.ycombinator.com/item?id=41505008)
___________________________________________________________________
(page generated 2024-09-12 23:01 UTC)