Item — HN Reader

Might be time to remove working on the 787 from my resume. I feel like the poor thing has been one disaster after another in the news.

I can't speak to the quality of the A and B level (most critical) code, but the development process for the C level software I was working on definitely could have used a lot of improvement. Messy code, tests and documentation were an afterthought/checkbox item, etc. The incentives were just wrong.

I think there's a ton of room for process innovation in avionics software development. One thing I wanted to build for a long time was a tool for tracing. In theory, every industry requirement (DO-178B, etc) was supposed to trace to a hight level Software Design Document (SDD) requirement, which was supposed to trace to a Software Requirements Document (SRD) requirement, which would trace to a code function. We maintained all of that BY HAND. It was a huge mess. Perfect example of something that could have been an extremely valuable development tool, but ended up just being a hassle to try and maintain.

Then of course there's language choice. C is king, which isn't necessarily a bad thing, but it's certainly not the safest, even in the restricted forms used in avionics. Sadly, my very first ever project as a programmer was porting an Ada codebase to C for the 787 (off by one errors for days...). It's almost cliche to say nowadays, but I would be really excited to see Rust gain some traction in avionics over the next 20 years or so. Because that's how far behind avionics is. We were using Visual Studio 6 in 2011!

12 comments

I work in the medical device field, and we have a similar process requirement (traceability from Design Input Requirements -> Software Specification -> Software Verification Procedure (and implicitly, the actual code function) -> Software Verification Report). We're currently wrangling with a giant-ass spreadsheet to keep track, and it totally sucks.

Ah yes I completely forgot SRD -> TESTS -> Then Code. Maybe that's because we always did it the other way around...

I'm telling you, there's money to be made building tools for this stuff. I think a big part of the reason things aren't being improved is that the people in a position to recognize bad process and tooling maybe aren't the type of people to see an opportunity to make money solving the problem rather than putting up with it. I wouldn't associate most of the engineers I knew at Honeywell as the type to stay up until 2AM every night for 3 months working on a side project to pitch to their boss.

I think it's really exciting what's happening in healthcare right now though. The innovative culture is exploding. Ultimately I care much more about what happens in medicine than avionics, as long as planes aren't falling out of the sky every 248 days...

There are already tools for this stuff. Problem is that they are all various forms of crappy and the market is so small that there is little incentive to improve them. I work in Medical Devices and one of the tools I'm supposed to use (we find every excuse to avoid it) has a UI like a 2001 Swing app and while it works, it is insanely painful to use due to its absolutely counterintuitive interface.

We're actually integrating more and more of our work in to Visual Studio since its tools are excellent. The problem is that the organization needs to validate any tool before we can use it as a part of Quality Management and that process can take forever.

Visual Studio is awesome. I'm really excited to see how Code turns out on Linux, especially for things like building GUI apps and 3D stuff.

I worked at Honeywell on these kinds of projects. I agree that these people definitely weren't that type. :)

Bizarrely now I am also helping (in a small way) with an innovative healthcare tech project. Maybe many people go through similar trajectories in programming without realizing it.

I'm curious about this process. Do you have contact info or pointers to other resources about it?

It's essentially the v model [1] for software development. It's commonly used in systems engineering and the industries that implement it, so military, avionics, medical, safety engineering etc. I work on industrial safety systems, and would echo the other posts here - it's a requirement of the industry, but it's rarely done well due to the cost to implement, and the difficulty in implementing tools like Doors.

NASA and the U.S. Military have good guides and white papers on it, which you should be able to find online with a bit of digging.

[1] http://en.m.wikipedia.org/wiki/V-Model_(software_development...

Unfortunately for you, I don't. I stay far away from the regulatory stuff - I really only enough to get my job done, mostly in the form of directives from my manager and our quality section.

That said, our software development process is really just a mild adaption of all our other development processes (like ones we use for hardware and reagent development). As I've described it above, it's a pretty standard engineering approach to design control, and there are all sorts of variations on them.

In fact, the FDA makes note of how broad the field of medical devices are, and does not actually enforce any specific design control process, but rather provides a framework for you to develop a process that meets your developmental and regulatory needs. As part of getting FDA approval, your design control process will be audited, as well as how well you followed it. In addition, medical devices are classified into 3 classes, corresponding to patient risk. Our product is currently classified in the lowest risk level, and so we can avoid a lot of additional requirements. As I recall from skimming through the standard (sorry, we only had a hard copy at work.. costs money apparently), higher risk devices with software components must have certain types of tests performed. In our risk class, we have total freedom to define our testing requirements - that said, the rigour of our testing is still under FDA scrutiny, it just means that there are no specific checkboxes to hit (like integration testing for example).

I guess if you want a place to start off, this might be a good place to look: http://en.wikipedia.org/wiki/Design_controls

> Then of course there's language choice. C is king, which isn't necessarily a bad thing, but it's certainly not the safest, even in the restricted forms used in avionics. Sadly, my very first ever project as a programmer was porting an Ada codebase to C for the 787 (off by one errors for days...). It's almost cliche to say nowadays, but I would be really excited to see Rust gain some traction in avionics over the next 20 years or so. Because that's how far behind avionics is. We were using Visual Studio 6 in 2011!

C has the big advantage that having been used for many years, it is very well understood, the compilers that exist are mature, and the formal analysis tools you want to use in a high assurance setting all exist also.

Higher level languages usually have none of these things.

> C has the big advantage that having been used for many years, it is very well understood,

C is probably the least understood language I know of. Vast parts of it are undefined, everyone makes their own decisions and compiler writers define the language.

C is a practical choice because it is performant, legacy code exists, and because a lot of people know the language.

Personally, I would never take a job writing safety-critical code in C. To give a single example of a far superior language for such applications, I would consider SPARK Ada, amongst others. I don't know how people that work on safety critical systems using C can sleep at night.

C is for systems programming where performance is king, not for building software that can kill people.

Ada is a good decision in this field.

Nevertheless, the parent was comparing C to Rust, where the language is immature, has no static analysis tools (to pick up the most important bugs), no well defined semantics and is entirely unproven in the field. This is the barrier for entry.

Rust's compiler is itself a static analysis tool that is more powerful than any static analysis that can be written for C. There is also a blossoming collection of "lint" libraries that extend Rust's type system to provide user-defined static analysis passes. Servo uses lints like these extensively to help manage interaction with SpiderMonkey, whose (C++-based) garbage collector manages the nodes in Servo's (Rust-based) DOM.

It's absolutely true that Rust is yet to be proven. We need to be ever careful not to oversell its benefits. But here's something that's been proven time and time again: use of C is fraught with horrors and pitfalls, and we should all be frantically seeking to improve the situation; if not via Rust, then via something else.

> the poor thing has been one disaster after another in the news

everything this complicated has always had these problems, difference is news jumps on it like a hawk now

In theory, every industry requirement (DO-178B, etc) was supposed to trace to a high level Software Design Document (SDD) requirement, which was supposed to trace to a Software Requirements Document (SRD) requirement, which would trace to a code function. We maintained all of that BY HAND. It was a huge mess. Perfect example of something that could have been an extremely valuable development tool, but ended up just being a hassle to try and maintain.

This is a symptom that group culture there is interfering with rational decision making around tool use.

Then of course there's language choice. C is king, which isn't necessarily a bad thing, but it's certainly not the safest, even in the restricted forms used in avionics.

This is another symptom. Perhaps it's excusable if there were preexisting libraries/codebase.

It's almost cliche to say nowadays, but I would be really excited to see Rust gain some traction in avionics over the next 20 years or so. Because that's how far behind avionics is. We were using Visual Studio 6 in 2011!

Another indication that group culture is blocking rational decision making about tool/language choice.

> This is another symptom. Perhaps it's excusable if there were preexisting libraries/codebase.

Uh, what would you use? As far as I know, the choices are: Assembler, Ada, and C. We're talking about avionics code, here.

C++ is not going to help you in this case. Java will be too slow/unpredictable. Real-Time Java is kind of sketch in my opinion (I could be wrong there). Most everything else is totally out.

Actually I'd probably choose Ada... but there are reasons the industry switched away from Ada.

> Java will be too slow/unpredictable. Real-Time Java is kind of sketch in my opinion (I could be wrong there).

* http://en.wikipedia.org/wiki/Real_time_Java

* http://www.rtsj.org/

I'm already familiar with those (which is why I mentioned it). Maybe you are just linking it for other people, in which case, thanks.

Why did the industry switch away from Ada? I thought they preferred Ada because it was such a 'safe' language?

Because Ada programmers were too expensive (i.e., there weren't that many of them).

At least, that's what I remember hearing---can't really vouch for it and I'd be happy to see someone else weigh in.

Also, my understanding was that Ada was basically forced on the defense industry by the Pentagon, and then there was eventually a backlash, and they dropped it. It may have had more success if it had been adopted voluntarily all along. In fact, it may yet have a resurgence. I know there is at least one serious Ada vendor out there still.

As far as I know, the choices are: Assembler, Ada, and C. We're talking about avionics code, here.

In that case, "group" may refer to the avionics software industry as a whole.

When you say Java I'm assuming you're refering to a JVM? Now what JVM are you refering to?

I'm not sure how C is specifically to blame for something like int overflow - while in many languages this specific problem won't happen, in many popular ones it would happen the same way as in C. E.g. doesn't Rust also have limited integer types?

In Rust, integer overflow is defined behavior, though: https://github.com/rust-lang/rfcs/blob/master/text/0560-inte...

True, but as far as I see from this RFC, you have to explicitly check for overflow or run code in debug mode (which of course doesn't help unless your CI includes the one-year-without-reboot test, which would probably do wonders to the production schedules ;). If you thought about it, when writing that code, then you could add the same check in C too! The whole issue is that most people do not make such checks consistently.

> I'm not sure how C is specifically to blame for something like int overflow

C is not a good choice when safety is paramount.

"The low-level nature of C and C++ means that bit- and byte-level manipulation of objects is commonplace; the line between mathematical and bit-level operations can often be quite blurry. Wraparound behavior using unsigned integers is legal and well-defined, and there are code idioms that deliberately use it. On the other hand, C and C++ have undefined semantics for signed overflow and shift past bitwidth: operations that are perfectly well-defined in other languages such as Java."

http://www.cs.utah.edu/~regehr/papers/overflow12.pdf

We use Doors for that. I think we tried TeamCenter for one project but it was a bag of hurt.

The UI for doors is a little clunky (looks like a win95 app), but it fulfills it's purpose.

Doors equivalent functionality could be built with a web interface and a sql db backend. Maybe any kind of database would work, relational just makes the most sense to me.

That said, assuming DO178B is similar to DO254, it's more about making sure you have a design process and that you are following it, and less about how good that process is.

Ah yes DOORS! I remember using that on one of our projects. Pretty sure it was 787. I had the feeling that if used properly it could be a powerful tool. But I also remember the whole IBM Rational suite being incredibly slow...

Weird, I had to do the same thing (porting Ada to C) for a similar project (the JSF). It's an awful idea, because so much is missing in the translation.

> Then of course there's language choice. C is king...

Is that true for all of Boeing? I recall reading about the Lustre synchronous dataflow language being used by Airbus. It apparently allows all sorts of interesting safety guarantees.

Not sure about all of Boeing. I worked for a subcontractor who got a small piece of the project from Honeywell who contracted with Boeing. But we worked on aircraft from Boeing, Gulfstream, Airbus, etc and other than the 787 Ada stuff (which we ported to C), I don't recall seeing anything that wasn't written in C, C w/ classes (heavily stripped C++), or Javascript. Of course it's possible that my boss just asked for those types of projects since we had the most expertise there.

Curious: what was Javascript used for? (In terms of general usage contexts.)

That was a joke ;)

> Because that's how far behind avionics is. We were using Visual Studio 6 in 2011!

Wait, what about autoPilot.js ?

> Rust

so you want to fly in planes using software written in Rust

A subset of it, yes. Any dynamic memory allocation is out, which would pare it down heavily, but you can still reap the safety advantages with the static stuff.

Rust happens to have already defined a subset of itself that forbids dynamic memory allocation, which makes use of a pared-down standard library called libcore.

Awesome! I figured such a thing would exist eventually but didn't realize it was already in the wild.

I definitely don't see a problem telling people the plane was built with rust

Did you consider using dtrace for tracing?

Edit: Oops, misread the comment.

Different type of tracing. I'm referring to textual tags which are used to associate a design requirement with a piece of code, the idea being that you can show a requirement is fulfilled by showing that it traces to tests and code, and that the existence of any piece of code needs to be justified by tracing back to the requirements.

http://en.wikipedia.org/wiki/Requirements_traceability

Borland's Caliber was once popular for requirements traceability, http://www.borland.com/Products/Requirements-Management/Cali...

Different type of tracing. This is basically tracing line items from a document to another set of line items on lower level documents, potentially all the way down to specific lines of source code.

Jinx!

> Sadly, my very first ever project as a programmer was porting an Ada codebase to C for the 787 (off by one errors for days...).

that's a terrifying statement. to those of us who know what a person's first programming projects are like, quality-wise

just terrifying. mindlessly stupid on your management's part

Keep in mind we're talking about C-level communications software, not an autopilot system. And the code was eventually tested heavily, just in a very inefficient and expensive manner. But I don't completely disagree with your assessment.

Do you think Java or a JVM language have been better suited for this task instead of C?

fair considerations