▲ 101 points
back
103 comments
I highly recommend watching "Its Quieter in the Twilight" (https://www.dailymotion.com/video/x9rrxr)
a cool documentary about the engineers still running voyager (a lot of them have been on it for decades)
Video isn't there... but I found it on YT.
Unfortunately geoblocked it seems.
ty!
you copypasted link missing "g" in the end
this is correct link: https://www.dailymotion.com/video/x9rrxrg
Content not available. Any chance it wasn't like that 10 minutes ago?
he cropped one character from address, the correct address is this
As the should. Voyagers are still active and this maintenance is needed in case issues occur. In a way due to the +24 hours oneway communication to correct software issues should they occur, this will help speed corrections up.
Now I wonder how the test it ? Is it on a software emulator on modern equipment or do they have a Voyager replica ?
What they have is surprisingly fragmentary. For the recent software issue, they had to be very conservative in what they changed because they aren't 100% sure about the details of the ISA on the actual spacecraft (due to amiguity between different revisions of the wiring of the CPU), and so they don't really have a simulator they can trust.
Maybe I'm used to think of NASA as this infinite budget agency, but considering Voyager's significance...
To me that souns like we should test all revisions of the CPU, not none of them
> Now I wonder how the test it ?
They have a simulator for one of the three computers on Voyager, the Computer Command Subsystem. They don't have a simulator for the other two computers, the Attitude and Articulation Control Subsystem and the Flight Data Subsystem. They used to have a Voyager replica, the Capability Demonstration Lab, but they got rid of it after they moved the Voyager team into a satellite office following the end of the main mission. There's more information here if you're interested: https://arc.aiaa.org/doi/pdf/10.2514/6.2016-2415
That would be an interesting project, I assume a lot of not so young engineers will want to play with it, as a hobby. Also if that project exists, I'm sure that someone will try to port DOOM.
I believe they have both.
Simulators for the spacecraft and replicas for some bits of hardware.
It's mind boggling that they didn't digitize every last scrap of paper around the project years ago, for starters.
Keeping a filing cabinet in the basement is awfully cheap...
Per the article, “much of the original documentation has been lost or fragmented. Voyager paperwork from the 1970s and 1980s was largely paper, and each time the project moved offices, more of it disappeared”.
Tearing down the project is even cheaper. But presumably we're alive for a reason
Maybe not years ago, but scanning documents with the phone in your pocket has become incredibly efficient. That combined with AI transcription and indexing for search makes such a project faster and cheaper in 2026 than at almost any other time in the past.
I scanned an entire book in 2016. It was way faster than scanning with the phone in my pocket. It’s just not ergonomic to use a general purpose device (a phone) instead of a specialized device (dedicated scanner).
I mean, they were never meant to last this long. NASA has a shoestring budget. I understand not taking the time and resources to do that when it could stop working a week later.
NASA has a significantly larger overall budget than SpaceX for info
I'm pretty sure volunteers will happily do this if NASA gave permission.
So it's 60 years old codebase still running. And in the two human-made objects the further away from earth.
Maybe only a few COBOL codebases still active can beat that? Or not?
Why does the picture at the top of the article look more like Cassini–Huygens than Voyager?
How else will we be able to assist the crew of the Enterprise?
Regarding "that almost nobody on Earth fully understands anymore", I claim this is nonsense, and definitely not an obstacle.
I've audited codebases in languages that I haven't programmed in. It is a matter of grasping a few basic concepts, like branch execution, branch destination, where data is stored, how it is communicated. Don Lancaster told us how to do this: https://www.tinaja.com/ebooks/enhance_vI.pdf.
I was puzzled by this claim, too. I think that the article is wrong, and that the code is written in HAL/S, a NASA-only language that sort of started off as a preprocessor to Fortran, though it has some PL/I-like features. If it really was written in Fortran, it probably was in a vendor-extended Fortran IV, which a lot of old guys like us know. But NASA used HAL/S for a number of projects, including the Shuttle. (And, by the way, thanks to perplexity.ai, here's a link to the HAL/S language manual: https://archive.org/details/nasa_techdoc_19750002029.)
The parent comment is apt. Of course, languages have their own quirks. But, as Christopher Strachey is once claimed to have said, “I use the same language no matter what compiler I run.”
Now what is more likely to be true is that the code is strangely structured (both because structured programming was new then, and because of memory and processor limitations), and also that much of the internal documentation has been lost. I wish the article had been clearer on that.
I think that the Voyager software is in assembly languages (there are several distinct computers on board), and that it is the program preparation software is written in a “Fortran V” extension.
> Regarding "that almost nobody on Earth fully understands anymore", I claim this is nonsense, and definitely not an obstacle.
It seems like you aren't very familiar with the history of the Voyager or the computer systems it uses. The Voyager has three onboard computers, each of which has a custom ISA. Two of the three computers were only used on Voyagers. After the main Voyager mission ended in the late 1980s and they were repurposed to collect data on interstellar space, the probes were reprogrammed to only require limited commands on one of the computers, and no intervention whatsoever on the other two. They also got rid of the testbench for testing code on the ground, so the only working Voyager systems are billions of miles away from Earth. Since then, Voyager has been operated by a skeleton crew that has been shrinking over time as people retire. The result of all of this is that they legitimately do not have a full understanding of how the hardware and software operates. Here's one example from a talk about how the crew fixed an issue with the Flight Data Subsystem, one of the computers that hadn't needed any significant software changes since the interstellar mission began (https://www.youtube.com/watch?v=dF_9YcehCZo):
> So our top priority was to figure out what the FDS was and how it worked. Because unfortunately the person who was the real expert had retired decades ago and the person who was their fill-in, their backup, had retired two years ago. So it was the worst place it could have hit us. These are some examples of some of the documentation we dug up on the FDS. So they're all hand written. These are some very dim circuit diagrams and these are hand written timelines for how to change FDS processors. We were lucky to find these. Some things we couldn't find. A lot of it was like this, that had been scanned. And frequently, these sources were contradictory, ambiguous. Why? Because we changed the way the spacecraft worked with every planetary encounter and entering the VIM and when things broke. So there was a lot of opportunity for ambiguity to arise in the documents over the course of 50 years. This is my favorite example. So this is a page out of an important FDS document. The change bar on the far side indicates it's a change, but somebody went in and made a very cryptic circle of the sentence and crossed it out. I have no idea to this day what it means. I have no idea. Maybe it was important. Maybe they crossed it out, because they thought crossing out was important. I'm not sure. So there were other cases like this. So we were even so confused in some cases, we weren't sure if we were sending data to the FDS - should it be least significant bit first or most significant bit first? We had that level of uncertainty.
> We didn't know we had the right instruction set. I showed you the instruction set there. We didn't know that was the one used to build the software, back when there was an assembler. We didn't know the source code version was the same as what was running onboard the spacecraft. We had a listing of the code, but it was a Microsoft Word document, with optical character recognition scans. We didn't know if there were errors in it. We had no assembler. So they're just playing with bits. There's no simulator and there's no test bed. So there were a few challenges. So basically this was bare knuckle binary manipulation. And they had to think of all the problems. I'm sure you guys are far better than I am of thinking of the problems associated with playing with memory like this. You could overwrite something. You can fail to recognize jumps. How do you debug it in flight on a flying spacecraft? And how do you know you're not missing something in those instruction sets and codes? And the list goes on and on.
I was responding to the headline that says:
>NASA still maintains some of the Voyager spacecraft code in a 1970s-era programming language that almost nobody on Earth fully understands anymore
I did not address the issue of the hardware documentation, just the issue raised by the headline. I fully understand the issues that the hardware and the problems of not having a full map of the hardware or a test bed to check out the code.
> Microsoft Word document, with optical character recognition scans
It's incredible that there is someone out there in this world who thought that OCRing documentation into a Word document is a good idea. And implemented it.
> younger engineers often have the capability but not the inclination
Kids these days... Why would someone in their right mind think working on the Voyager project could damage their careers? You can work on new and fancy tools all you want to improve supporting tools, and it's still one of the coolest space missions active. Plus, it has a real end - at some point, support will be further reduced and the person will move on to another space exploration job, with the extra golden star of having been on the Voyager.
I would go further... This project gives a rare opportunity for a young engineer to learn to build truly mission critical, resilient software while requiring complete, top to bottom understanding of the software and hardware stack.
I am no longer a junior, but would have been upset to be tasked with refreshing the old historical obsolete laundry (no matter how sacred or distinguished), expecially when I already had experience delivering safety critical products packing much more modern technologies.
The opportunity they would be offering is not rare at all! The opportunity to research and design something truly new on the other way is very scarce.
Even all that to the side, it lets you say you worked on the Voyager project!
Q: What was the hardest debugging problem you had?
A: I had a very slow network connection to the computer, and it was 23 light-hours away from me.
> Kids these days... Why would someone in their right mind think working on the Voyager project could damage their careers?
It's an isolated legacy-project with no future. Mostly everything you learn for it will be only useful for this specific project, so all time to invest there is time to can't invest into something useful. Sure, there are probably some parts to learn from this too, but It's less than what your competitors will learn on their fancy modern projects.
> You can work on new and fancy tools all you want to improve supporting tools
Voyager is in maintenance, there is no big innovation or big progress to be made there. It's just work to hold the line as long as possible. And I guess nobody want's to be the one killing it because of a poor attempt to innovate something.
Because software development back in the day wasn’t like how it’s now now, the charade so called software development now is a clown show: scrum, daily stand ups, open office style, tickets, tons of ci/cd BS, and of course, the wrangler aka PM and all politics involved, none of this existed like the cult it is now, I only had one experience in such environment and despite the effort I had to ask for some common sense, it was like insulting someone’s religion, “how dare you challenge the sacred methods that the silicone valley companies are using?!!”
Additionally, back in the day there was true ownership for the code you write, the code is owned by you not the company, and I know few old engineers that until now (they are retired) the companies still pay them for using their code they wrote while working there. That sense of ownership encourages you to tackle hard issues rather feeling like a machine spewing code for someone else’s business, I have seen some contracts too where the company will have ownership for anything you do while you are in the contract, including your personal projects on your own free time.
>"Why would someone in their right mind think working on the Voyager project could damage their careers"
Assembly? Understanding how things actually work? No Agile? No K8s? No Rust, No React? - death knell for someone's resume
>"and the person will move on to another space exploration job, with the extra golden star of having been on the Voyager"
this is the best case with the result of being tied to another single project for years and unemployable anywhere else. in more realistic case - warm goodbye in few years and start your life from scratch with no credits for the thins done.
> The succession problem matters most in the next decade. After that, the question becomes academic. There will be no Voyagers left to maintain.
Well, the Voyagers will still be there, there will be just no way to contact them anymore once the power runs out or communication is lost (whichever happens first)...
Many of the issues could potentially be solved by modern LLMs?
Reading, analyzing and assembling documentation could be probably done by LLMs.
And by including old code and snippets into the training set, the LLM could be fairly proficient in writing this code probably too?
Maybe someone knows more about the use/not-use of LLMs in this context?
I don't think you want LLMs touching projects that cost over $800.000.000, even to assemble "documentation" (since the LLM can't really document in as much it's translating what it's reading, because documentation includes much more information than what's stored in the code itself).
It's a cool idea, though, I'd like to see this done as an experiment :)
> Many of the issues could potentially be solved by modern LLMs?
Yesterday I asked an LLM what customizations niri made to the KDL language.
It said niri modified the language to add single line comments with //. However if you visit the official home page of KDL https://kdl.dev/, the very first example shows single line comments as being part of the official spec. There's also a whole page dedicated to comments in the spec that mention this.
The moral of the story is LLMs are honestly really really bad and I'm sincerely concerned at how they manipulate people into thinking what they produce is accurate or trustable. I didn't believe the AI because my spidey sense said that doesn't feel right, so I double checked the real source.
It's gotten to the point where I'm finding a huge majority of the time, it provides incorrect information on really basic things. Pure hallucinations. I'm at the stage now where even if you paid me, I wouldn't use it. There's a 0% chance I'd ever consider paying for it in its current state and it's upsetting because it is killing off the web in real-time. It's becoming harder and harder to find useful and accurate information.
That's insane.
Of all the possible projects, this is the one where it's both highly feasible for humans to learn and historically critical that we have a full understanding and control of its operation.
You don't want your spacecraft that's 15.8 billion miles away pointing backwards because the AI's fairly proficient code misunderstand something.
“You’re absolutely right, I shouldn’t have jettisoned the RTG”