Skip to content
Coding is important

Boeing acknowledges “gaps” in its Starliner software testing

"They did an abundance of testing."

Eric Berger – | 277
Starliner touches down in December 2019 for the first time. Credit: NASA/Aubrey Gemignani
Starliner touches down in December 2019 for the first time. Credit: NASA/Aubrey Gemignani
Story text

On Friday, during a detailed, 75-minute briefing with reporters, a key Boeing spaceflight official sought to be as clear as possible about the company's troubles with its Starliner spacecraft.

After an uncrewed test flight in December of the spacecraft, Boeing "learned some hard lessons," said John Mulholland, a vice president who manages the company's commercial crew program. The December mission landed safely but suffered two serious software problems. Now, Mulholland said, Boeing will work hard to rebuild trust between the company and the vehicle's customer, NASA. During the last decade, NASA has paid Boeing a total of $4.8 billion to develop a safe capsule to fly US astronauts to and from the International Space Station.

At the outset of the briefing, Mulholland sought to provide information about the vehicle's performance, including its life support systems, heat shield, guidance, and navigation. He noted that there were relatively few issues discovered. However, when he invited questions from reporters, the focus quickly turned to software. In particular, Mulholland was asked several times how the company made decisions on procedures for testing flight software before the mission—which led to the two mistakes.

He struggled to answer those questions, but the Boeing VP said the reason was not financial. "It was definitely not a matter of cost," Mulholland said. "Cost has never been in any way a key factor in how we need to test and verify our systems."

Two software errors

The first software error occurred when the spacecraft captured the wrong "mission elapsed time" from its Atlas V launch vehicle—it was supposed to pick up this time during the terminal phase of the countdown, but instead it grabbed data 11 hours off of the correct time. This led to a delayed push to reach orbit and caused the vehicle's thrusters to expend too much fuel. As a result, Starliner did not dock with the International Space Station.

The second error, caught and fixed just a few hours before the vehicle returned to Earth through the atmosphere, was due to a software mapping error that would have caused thrusters on Starliner's service module to fire in the wrong manner. Specifically, after the service module separated from the capsule, it would not have performed a burn to put the vehicle into a disposal burn. Instead, Starliner's thrusters would have fired such that the service module and crew capsule could have collided.

NASA and Boeing have been conducting a joint assessment of these software problems, and they're expected to report their findings in a week, on March 6. But on Friday, Mulholland was prepared to discuss two issues with Boeing's software verification that the company intends to fix.

First of all, he acknowledged the company did not run integrated, end-to-end tests for the whole mission. For example, instead of running a software test that encompassed the roughly 48-hour period from launch through docking to the station, Boeing broke the test into chunks. The first chunk ran from launch through the point at which Starliner separated from the second stage of the Atlas V booster. Unfortunately for Boeing engineers, the mission elapsed timing error occurred just after this point in time. "If we would have run the integrated test through the first orbital insertion burn time frame, we would have seen that we missed the burn," Mulholland said.

Looking at logic strings

During the validation process, Boeing engineers also did not test every complex "logic string" in the software code, Mulholland said. Essentially, this means they checked the basic code but did not follow through every possibility during complex logic strings such as if/then/or etc. Boeing has performed an audit to discover their gaps in testing, and the next step is to do both end-to-end integrated testing as well as fill in those gaps.

image of starliner spacecraft.
Boeing shared recent images of the Starliner used during December's test, left, and the vehicle being built for a crew test flight.
Credit: Boeing
Boeing shared recent images of the Starliner used during December's test, left, and the vehicle being built for a crew test flight. Credit: Boeing

"I don’t want you to get the impression that this team took shortcuts," he said. "This team didn’t. They did an abundance of testing. In certain areas obviously we have gaps to go fill."

NASA is in the midst of several reviews of Boeing's safety culture and the underlying issues that led to these software problems. Beginning with the briefing next Friday, we are likely to get some answers on how those are progressing. However, the answers to the question of when Starliner flies again—and whether that mission will carry astronauts or NASA will require another test flight—will probably remain months into the future.

Listing image: NASA/Aubrey Gemignani

Photo of Eric Berger
Eric Berger Senior Space Editor
Eric Berger is the senior space editor at Ars Technica, covering everything from astronomy to private space to NASA policy, and author of two books: Liftoff, about the rise of SpaceX; and Reentry, on the development of the Falcon 9 rocket and Dragon. A certified meteorologist, Eric lives in Houston.
277 Comments
Prev story
Next story