ApolloHoax.net

Off Topic => General Discussion => Topic started by: Jeff Raven on August 15, 2021, 11:02:01 AM

Title: Boeing Starliner
Post by: Jeff Raven on August 15, 2021, 11:02:01 AM
Is it time to (finally) consider pulling the plug on this project? It's not just years behind schedule and over budget, but now it's back to the shop for more repairs. I am not an engineer, and therefore my reaction is gut based rather than informed, and I welcome any corrections on that.  That said, the fact that Boeing admits that some of the moisture which led to the issue with the valves could have simply been due to the humid Florida weather, and that this wasn't already accounted for, seems, well, pathetic.  I'm sure this is oversimplified, but to not account for the fact that Florida has humid weather would be like building something in McMurdo Station but not insulating it against cold.  If this is actually what happened, that they messed up something so basic, how can NASA (or anyone) be confident that they got the more complicated things right?
Title: Re: Boeing Starliner
Post by: jfb on August 15, 2021, 11:38:48 PM
Pulling the plug on Starliner means SpaceX is the sole US ride to the ISS until it is decommissioned. Which may wind up being the case anyway if Boeing takes another couple of years to fix this issue. The next earliest opportunity is November, but I suspect they won’t make that.  God knows what other nasties are still lurking. Note that Boeing doesn’t get paid until they meet this milestone - they’re paying for this second test out of their own pocket.  If this test fails, I’m not sure they wouldn’t pull the plug themselves. 

I will say this feels like a more "normal" issue with new vehicle development than pulling the wrong value for MET from the booster and screwing up your thruster programming.  But it ain’t awesome. Thank God they caught it on the ground. 

I’m just hoping this was a process issue, not a design or manufacturing issue. 
Title: Re: Boeing Starliner
Post by: cjameshuff on August 16, 2021, 10:04:01 AM
As far as NASA is concerned, Starliner is still closer to being a working second spacecraft than any other option. And on Boeing's part, giving up now will not save them money or win them future contracts.
Title: Re: Boeing Starliner
Post by: jfb on August 16, 2021, 11:23:56 AM
Yeah, ditching Starliner and going with something like Dreamchaser just pushes everything back several more years, and there aren't that many years left in the ISS.  Boeing will likely make it there by 2022, SNC wouldn't get there before 2024, and that's if they can figure out how to launch a manned variant that doesn't require riding in a fairing. 

Again, this delay and the reason for it is bad, but it's not stupid like the software errors in OFT-1.  I feel like they can get this worked out, if not by November, then by early 2022. 

But they had better go over that thing with a fine-toothed comb before stacking it again.  If nothing else purge those lines properly. 
Title: Re: Boeing Starliner
Post by: JayUtah on August 18, 2021, 10:03:14 AM
In this particular race there are actually lots of points for second place.  However, after the dismal failure of OFT-1, OFT-2 was seen by many as a highly crucial mission in terms of Boeing's credibility.  This might be a nail in the coffin of the old funding model.  The ship will fly, but it's definitely going to change how NASA and Boeing do business moving forward.

I think I said it before, but Boeing isn't the same company I started doing business with back in the 1990s, and not even in the same ballpark as the company that it was back in the 1960s.  They've largely given up on what made them successful in the past.  My brother-in-law just started there as a software engineer, but he's in the commercial airframe side of the business.  We'll see what he reports about ongoing company culture.
Title: Re: Boeing Starliner
Post by: Peter B on August 18, 2021, 10:28:48 AM
In this particular race there are actually lots of points for second place.  However, after the dismal failure of OFT-1, OFT-2 was seen by many as a highly crucial mission in terms of Boeing's credibility.  This might be a nail in the coffin of the old funding model.  The ship will fly, but it's definitely going to change how NASA and Boeing do business moving forward.

I think I said it before, but Boeing isn't the same company I started doing business with back in the 1990s, and not even in the same ballpark as the company that it was back in the 1960s.  They've largely given up on what made them successful in the past.  My brother-in-law just started there as a software engineer, but he's in the commercial airframe side of the business.  We'll see what he reports about ongoing company culture.

Can you elaborate on what has changed? Is it unique to Boeing or are these changes happening in other aerospace/defence companies? And do these changes affect the ability of Boeing (and potentially the other companies) to provide good products to the US military?
Title: Re: Boeing Starliner
Post by: jfb on August 18, 2021, 03:17:34 PM
In this particular race there are actually lots of points for second place.  However, after the dismal failure of OFT-1, OFT-2 was seen by many as a highly crucial mission in terms of Boeing's credibility.  This might be a nail in the coffin of the old funding model.  The ship will fly, but it's definitely going to change how NASA and Boeing do business moving forward.

I think I said it before, but Boeing isn't the same company I started doing business with back in the 1990s, and not even in the same ballpark as the company that it was back in the 1960s.  They've largely given up on what made them successful in the past.  My brother-in-law just started there as a software engineer, but he's in the commercial airframe side of the business.  We'll see what he reports about ongoing company culture.

Can you elaborate on what has changed? Is it unique to Boeing or are these changes happening in other aerospace/defence companies? And do these changes affect the ability of Boeing (and potentially the other companies) to provide good products to the US military?

This article (https://www.theatlantic.com/ideas/archive/2019/11/how-boeing-lost-its-bearings/602188/) explains it pretty well.

Short version - Boeing bought McDonnell Douglas, which was a failing company for all the same reasons Boeing is struggling right now, and somehow the MD executives wound up in charge. 
Title: Re: Boeing Starliner
Post by: Peter B on August 18, 2021, 08:06:59 PM
In this particular race there are actually lots of points for second place.  However, after the dismal failure of OFT-1, OFT-2 was seen by many as a highly crucial mission in terms of Boeing's credibility.  This might be a nail in the coffin of the old funding model.  The ship will fly, but it's definitely going to change how NASA and Boeing do business moving forward.

I think I said it before, but Boeing isn't the same company I started doing business with back in the 1990s, and not even in the same ballpark as the company that it was back in the 1960s.  They've largely given up on what made them successful in the past.  My brother-in-law just started there as a software engineer, but he's in the commercial airframe side of the business.  We'll see what he reports about ongoing company culture.

Can you elaborate on what has changed? Is it unique to Boeing or are these changes happening in other aerospace/defence companies? And do these changes affect the ability of Boeing (and potentially the other companies) to provide good products to the US military?

This article (https://www.theatlantic.com/ideas/archive/2019/11/how-boeing-lost-its-bearings/602188/) explains it pretty well.

Short version - Boeing bought McDonnell Douglas, which was a failing company for all the same reasons Boeing is struggling right now, and somehow the MD executives wound up in charge.

Wow, what an interesting article. Clearly shows how Starliner and the 737-Max belong in the same basket.
Title: Re: Boeing Starliner
Post by: Glom on August 20, 2021, 11:27:36 AM
I was given a paper to read from Boeing on some outfitting they did to one of their derivatives. Kind of reads like a high school report what with the language. Things like, "engineers were mesmerised by the video."

Definitely not something you'd see in a major engineering publication.
Title: Re: Boeing Starliner
Post by: JayUtah on August 28, 2021, 12:36:27 PM
This article (https://www.theatlantic.com/ideas/archive/2019/11/how-boeing-lost-its-bearings/602188/) explains it pretty well.

Short version - Boeing bought McDonnell Douglas, which was a failing company for all the same reasons Boeing is struggling right now, and somehow the MD executives wound up in charge.

Yeah, that article pretty much sums it up.  All I have to add just extends the points made there, and adds some color.  The phrase, "McDonnell Douglas bought Boeing with Boeing's own money," has been in circulation in the aerospace community for years.

The philosophy of doing it cheaply rather than doing it right was McD's downfall, and will almost certainly be Boeing's.  But I think the "color" I would add is that Boeing has fallen even more deeply in to the chasm of short-term shareholder and executive profits over good business.  That's what every major American company seems to have done in the past 20 years.  They seem to have become piggy-banks for the wealthy elite, who are just pillaging their assets and reputations for short-term profits.

And yes, this has begun to affect government contracts.  Ironically that's why Boeing has been such an attractive target for pillaging.  The near-bottomless revenue stream of government contracting makes dollar-signs ring up in the eyes of potential executives and investors.  They seem to believe that Boeing can do no wrong, and that all manner of managerial misconduct or short-sightedness will be tolerated.

Aerospace requires strategic planning at least into the next decade, because that's how long development schedules have typically extended.  None of that seems to be happening now.  The focus seems to be on extending existing product lines for as long as possible with minimal investment.  I fear the Dreamliner (which I was privileged to work on) will be the last new commercial airframe they develop.  And I fear the Starliner will continue to stumble until Boeing finally loses NASA's confidence.
Title: Re: Boeing Starliner
Post by: JayUtah on August 28, 2021, 01:03:07 PM
I was given a paper to read from Boeing on some outfitting they did to one of their derivatives. Kind of reads like a high school report what with the language. Things like, "engineers were mesmerised by the video."

Definitely not something you'd see in a major engineering publication.

No, but the industry is changing.  20 years ago we paid no attention to public relations.  We knew who our customers were, and they knew who we were.  We all spoke the same (very nerdy) language, and there was never any reason to reach out to the public or to cultivate some image outside the industry communities.  No one's business success depended on that, so it was an extraneous expenditure.

Then along came Elon Musk, who put a face on space and made it cool again.  Everyone else is playing catch-up.  You see Tory Bruno now trying to put a face on United Launch Alliance.  You see the other Space Billionaires trying to grab spotlights.  Boeing doesn't have anyone to sing in that chorus line.  Neither does Northrop Grumman, the other major firm I work with.  (And all our best guys came from Lockheed.)  But by the same token, it seems like they're trying to jump on the bandwagon of "Space is cool again!" and do things that the general public might approve of, but they aren't sure how.  Honestly to me it comes off like my grandpa trying to ride a skateboard.  You are Boeing.  Be Boeing.  Or rather, go back to what being Boeing used to mean, if you're going to reinvent yourself.

SpaceX -- and to a lesser extent, other newer companies -- are eating everyone's lunches not just because they are new and innovative and arguably good, but because they are good at telling everyone how good they are.  The aerospace establishment probably can't re-invent themselves in the SpaceX mold, not just because grandpas look ridiculous riding skateboards, but because they've never been good at riding skateboards.  The type of fast-paced, far-reaching, sexy innovation that SpaceX can do (by virtue of private billions) is just not something Boeing can realistically do now.  They are the tortoise to Musk's hare, if this post will bear another metaphor.
Title: Re: Boeing Starliner
Post by: jfb on August 31, 2021, 04:42:16 PM
An anecdote on what it was like to work with Boeing's defense side a little over a decade ago.  At the time I convinced myself that the problems I saw were the result of this project being kind of a prototype/proof of concept thingy, but after the last few years I think I was seeing the overall rot. 

We were subcontracting for Boeing on a situational awareness/common operational picture system.  The concept was that you had services running on various platforms reporting status and combat readiness, objects in the battlespace obtained from human and sensor observations, and a few other things - these services were written by the various subcontractors.  I was responsible for the service that received battlespace object reports (HUMINT and sensor).  These services would send updates to a separate GUI service running on its own platform, which would integrate and present this data for a warfighter to monitor and validate before being forwarded on to the COP at HQ - this GUI service was written by the Boeing team.

Except, here's the problem - the GUI service didn't actually integrate or manage any data.  Instead, they exposed hooks to all the graphics primitives (tables, maps, etc.) and relied on the back-end services to perform the actual screen management - the back-end services were responsible for managing the screen state. 

I didn't just forward BSO reports to the GUI service - I actually sent the commands to add or remove rows from the table that displayed those reports on the GUI - over a network with a quarter-second latency.  And the API Boeing provided only allowed me to add or remove one row at a time, which proved to be a major issue during usability testing.  Same thing was true for all the other services.  The positioning service that was responsible for updating the map had to send commands to draw the actual graphics primitives on the map itself. 

The GUI team (Boeing) successfully deferred all the hard work onto the backend services.  And the GUI service still didn't quite work right. 

Now, this design had two big problems.  The biggest problem was that it could not meet functional requirements.  Anytime the warfighter selected an object on the map, all the relevant tables were supposed to automatically update to show status, combat readiness, etc., for that specific object.  This design specifically prevented that from happening - the backend services didn't communicate with each other, and since the GUI didn't actually keep track of anything it was displaying, it couldn't send the corresponding command to the back-end services to update their information either.

The second problem was performance - all the backend services were doing screen management over a network, which was already slow.  Add to this that the APIs were exceedingly primitive.  Like I said, I could only add or remove one row at a time from the table I was responsible for, and with network latency each command took about a quarter second to execute.  Not a problem when you're monitoring 5 or 10 objects, deadly when you're managing a couple hundred (it took on the order of 5 minutes to update the table if you were managing hundreds of reports).  Under realistic scenarios the whole thing just fell down hard.  We suggested to the Boeing team that they add calls to perform bulk updates (or at least a call to delete all the rows in the table), but those suggestions were dismissed as being "too hard". 

The API code that we called was generated from a spreadsheet that was chock full of spelling errors - I had at least one build break a day because I insisted on spelling it "activity" instead of "avtivity". 

How this got past review and approved (especially since it couldn't satisfy some key functional requirements) is a mystery to me, meaning it was never really reviewed.  Someone at Boeing should have kicked that design back with "are you f___ing kidding me" stamped on every page in red ink, but didn't. 

In the interests of full disclosure, I have to point out that my code was hot garbage.  It was my first adventure in multithreading and I botched it badly enough that I put the schedule at risk all by myself.  But ... I wouldn't have had to do that if I didn't have to do two completely separate jobs in the first place.  My service shouldn't have cared about what rows were actually visible on the screen, it shouldn't have cared about the state of a dropdown menu, it shouldn't have cared about anything but maintaining a connection and sending updates. 

At the time I told myself it was because this wasn't something in a critical path so they weren't putting their best team on it, but it was part of a larger modernization program and proved to be an accurate representation of the overall effort.  The Army finally got a clue and cancelled the whole thing.  And while it ultimately cost me my job, as a taxpayer I was thrilled. 
Title: Re: Boeing Starliner
Post by: Glom on September 01, 2021, 03:20:15 AM
I was given a paper to read from Boeing on some outfitting they did to one of their derivatives. Kind of reads like a high school report what with the language. Things like, "engineers were mesmerised by the video."

Definitely not something you'd see in a major engineering publication.

No, but the industry is changing.  20 years ago we paid no attention to public relations.  We knew who our customers were, and they knew who we were.  We all spoke the same (very nerdy) language, and there was never any reason to reach out to the public or to cultivate some image outside the industry communities.  No one's business success depended on that, so it was an extraneous expenditure.

Then along came Elon Musk, who put a face on space and made it cool again.  Everyone else is playing catch-up.  You see Tory Bruno now trying to put a face on United Launch Alliance.  You see the other Space Billionaires trying to grab spotlights.  Boeing doesn't have anyone to sing in that chorus line.  Neither does Northrop Grumman, the other major firm I work with.  (And all our best guys came from Lockheed.)  But by the same token, it seems like they're trying to jump on the bandwagon of "Space is cool again!" and do things that the general public might approve of, but they aren't sure how.  Honestly to me it comes off like my grandpa trying to ride a skateboard.  You are Boeing.  Be Boeing.  Or rather, go back to what being Boeing used to mean, if you're going to reinvent yourself.

SpaceX -- and to a lesser extent, other newer companies -- are eating everyone's lunches not just because they are new and innovative and arguably good, but because they are good at telling everyone how good they are.  The aerospace establishment probably can't re-invent themselves in the SpaceX mold, not just because grandpas look ridiculous riding skateboards, but because they've never been good at riding skateboards.  The type of fast-paced, far-reaching, sexy innovation that SpaceX can do (by virtue of private billions) is just not something Boeing can realistically do now.  They are the tortoise to Musk's hare, if this post will bear another metaphor.

A curious metaphor since the tortoise won in the end.
Title: Re: Boeing Starliner
Post by: JayUtah on September 02, 2021, 06:04:34 PM
Except, here's the problem - the GUI service didn't actually integrate or manage any data.  Instead, they exposed hooks to all the graphics primitives (tables, maps, etc.) and relied on the back-end services to perform the actual screen management - the back-end services were responsible for managing the screen state.

I showed this to my software engineering department head.  He facepalmed.

Quote
In the interests of full disclosure, I have to point out that my code was hot garbage.  It was my first adventure in multithreading and I botched it badly enough that I put the schedule at risk all by myself.  But ... I wouldn't have had to do that if I didn't have to do two completely separate jobs in the first place.

Indeed.  The fact that you had the typical problems with multithreading doesn't seem to have much to do with an API that doesn't allow for distributed back-end functionality.  But your example rings true with everything I've experienced with Boeing and software.
Title: Re: Boeing Starliner
Post by: JayUtah on September 02, 2021, 06:06:33 PM
A curious metaphor since the tortoise won in the end.

Only because the hare eventually screwed up.  But yeah, I should have left out that last part.
Title: Re: Boeing Starliner
Post by: cjameshuff on September 03, 2021, 06:43:37 PM
Bezos like to describe the company in terms of the tortoise and hare fable, even incorporating tortoises into the "coat of arms", but they don't seem to understand that the tortoise didn't win because it was slow, it won despite it, and only because the hare didn't take the race seriously. It's a poorly-fitting metaphor for this: Blue Origin is actually more like the hare in that they keep stopping, but they never raced ahead in the first place. Boeing's lost all ability to move without someone dangling a government contract in front of them. SpaceX sometimes falls down but always bounces right back, and occasionally veers off on a wild tangent, but they're quick to abandon bad ideas when they find better ones.

What Bezos calls "skipping steps" has allowed SpaceX to gain a wide range of experience, explore a variety of different approaches, and develop multiple generations of successful launch systems while Blue Origin is still working on getting New Glenn off the ground and Boeing's still trying to pretend everything's fine the way it is. Boeing's a fat old hog that won't budge without a carrot waved in front of his nose. SpaceX is more like a pack of hounds frantically scouring the landscape for a trail. And Blue Origin is doing a fair imitation of an unidentifiable piece of roadkill, breeding pests and emitting foul odors.
Title: Re: Boeing Starliner
Post by: Jeff Raven on September 05, 2021, 02:01:03 AM
I was given a paper to read from Boeing on some outfitting they did to one of their derivatives. Kind of reads like a high school report what with the language. Things like, "engineers were mesmerised by the video."

Definitely not something you'd see in a major engineering publication.

As someone who teaches scientific writing, that example is truly horrifying.
Title: Re: Boeing Starliner
Post by: Jeff Raven on September 05, 2021, 02:19:52 AM

The second problem was performance - all the backend services were doing screen management over a network, which was already slow.  Add to this that the APIs were exceedingly primitive.  Like I said, I could only add or remove one row at a time from the table I was responsible for, and with network latency each command took about a quarter second to execute.  Not a problem when you're monitoring 5 or 10 objects, deadly when you're managing a couple hundred (it took on the order of 5 minutes to update the table if you were managing hundreds of reports).  Under realistic scenarios the whole thing just fell down hard.  We suggested to the Boeing team that they add calls to perform bulk updates (or at least a call to delete all the rows in the table), but those suggestions were dismissed as being "too hard". 

How this got past review and approved (especially since it couldn't satisfy some key functional requirements) is a mystery to me, meaning it was never really reviewed.  Someone at Boeing should have kicked that design back with "are you f___ing kidding me" stamped on every page in red ink, but didn't. 

At the time I told myself it was because this wasn't something in a critical path so they weren't putting their best team on it, but it was part of a larger modernization program and proved to be an accurate representation of the overall effort. 

A thousand years ago I started college as a computer science major. (switched out of that after a little over a year) One of the primary things my teachers drilled into us - other than you damn well better proof your work, anticipate every possible human error, no matter how small or unlikely, and provide a fix for it, and put comments in everywhere - was that you should make every bit of coding as fast and efficient as possible. It's not a big deal if your bit takes a quarter-second (back then that would have been blazing fast) until you realize that it might be a module that's part of a larger program, one which calls your little slice of code 100 times a day. And if that code could be executed in a tenth of second instead, that's a 60% time savings, multiplied by 100, and that adds up to real differences. An idealized view? Perhaps. But I mentioned that story to my nephew, who has a doctorate in and teaches computer science, and he said that that's still the standard, and something he also tells his students.

And, as you know, with the complexity of programs, a seemingly minor module or path could somehow find its way into a very critical thread, so them not checking this stuff and putting that red stamp on it (love the imagery) really can't be excused.
Title: Re: Boeing Starliner
Post by: JayUtah on September 05, 2021, 05:48:51 PM
Boeing's lost all ability to move without someone dangling a government contract in front of them.

The majority of Boeing's profits are still from private-sector business.  Without a customer or a potential customer driving specific forms of innovation, it's hard to make the case that Boeing should use its investor funds to do the same kinds of things as younger companies are doing.  It has a fair amount to do with Boeing having become a dysfunctional company.  But it has more to do with Boeing being a publicly-held company.
Title: Re: Boeing Starliner
Post by: jfb on September 06, 2021, 01:12:43 AM

The second problem was performance - all the backend services were doing screen management over a network, which was already slow.  Add to this that the APIs were exceedingly primitive.  Like I said, I could only add or remove one row at a time from the table I was responsible for, and with network latency each command took about a quarter second to execute.  Not a problem when you're monitoring 5 or 10 objects, deadly when you're managing a couple hundred (it took on the order of 5 minutes to update the table if you were managing hundreds of reports).  Under realistic scenarios the whole thing just fell down hard.  We suggested to the Boeing team that they add calls to perform bulk updates (or at least a call to delete all the rows in the table), but those suggestions were dismissed as being "too hard". 

How this got past review and approved (especially since it couldn't satisfy some key functional requirements) is a mystery to me, meaning it was never really reviewed.  Someone at Boeing should have kicked that design back with "are you f___ing kidding me" stamped on every page in red ink, but didn't. 

At the time I told myself it was because this wasn't something in a critical path so they weren't putting their best team on it, but it was part of a larger modernization program and proved to be an accurate representation of the overall effort. 

A thousand years ago I started college as a computer science major. (switched out of that after a little over a year) One of the primary things my teachers drilled into us - other than you damn well better proof your work, anticipate every possible human error, no matter how small or unlikely, and provide a fix for it, and put comments in everywhere - was that you should make every bit of coding as fast and efficient as possible. It's not a big deal if your bit takes a quarter-second (back then that would have been blazing fast) until you realize that it might be a module that's part of a larger program, one which calls your little slice of code 100 times a day. And if that code could be executed in a tenth of second instead, that's a 60% time savings, multiplied by 100, and that adds up to real differences. An idealized view? Perhaps. But I mentioned that story to my nephew, who has a doctorate in and teaches computer science, and he said that that's still the standard, and something he also tells his students.

My mantra has been:

It doesn’t matter how fast your code is if it’s wrong.
It doesn’t matter how fast your code is if you can’t maintain it.
It doesn’t matter how fast your code is if it’s a malware vector.
It doesn’t matter how fast your code is if it falls over when someone sneezes in the next room.

Speed matters, but those things matter more.  Yes, if you can shave a couple of cycles out of each iteration of a tight loop that executes thousands of times, that’s worth doing.   Shaving a couple of cycles off a subroutine that executes exactly once at startup, however, isn’t worth the effort. 

Different war story on the maintainability front.  We were making a sonar simulator for a Navy lab.  We handled modeling the noise spectra and the movement of objects in the simulation.  A second group built a massive DSP to take our noise spectra and turn them into the signals you’d get from a towed array.  A third contractor was responsible for a 3D graphical display.

At one point we were asked to look at the graphics code to see if we could speed it up (this was written in C on a Silicon Graphics  system using (not-yet-open) GL).  A small program, only about 5000 lines or so, but they were all in main.  The original author believed that using actual subroutines was too inefficient, so he used gotos to branch all over the place - something like 15 gotos in all. It took my coworker a couple of weeks to puzzle out the flow of control. 

The code was so tightly coupled with itself we could not make any changes without breaking something.  We tried compiling with level 1 optimization - the compiler ate all the RAM, then ate all the swap space and the system panicked.  This code was literally unmaintainable.

We finally gave the lab two choices - let us rewrite the whole thing from the keel up, or buy faster hardware.

They wound up buying faster hardware.

Quote
And, as you know, with the complexity of programs, a seemingly minor module or path could somehow find its way into a very critical thread, so them not checking this stuff and putting that red stamp on it (love the imagery) really can't be excused.

Yup.  But again, the performance wasn’t the main problem, it was the absolute inability to satisfy a basic functional requirement that was so boggling.  How that got by anyone is an absolute travesty.
Title: Re: Boeing Starliner
Post by: Peter B on September 06, 2021, 03:44:47 AM
I was given a paper to read from Boeing on some outfitting they did to one of their derivatives. Kind of reads like a high school report what with the language. Things like, "engineers were mesmerised by the video."

Definitely not something you'd see in a major engineering publication.

Out of interest, is the paper publicly available?
Title: Re: Boeing Starliner
Post by: grmcdorman on September 06, 2021, 09:59:41 AM
With respect to optimization, premature optimization is also bad. Yes, you can write your code such that it would need to be completely rewritten to be more efficient, but you can also write code where optimization isn't needed - and as jfb says, that can result in unmaintainable code. There is often a balance between clarity (and stability) and maximum speed.

One approach is to try to write things reasonably efficiently, but with a focus on clarity/stability/maintainability. Then run a performance analysis tool to see where you need to focus your optimization efforts.
Title: Re: Boeing Starliner
Post by: Glom on September 06, 2021, 11:03:32 AM
I was given a paper to read from Boeing on some outfitting they did to one of their derivatives. Kind of reads like a high school report what with the language. Things like, "engineers were mesmerised by the video."

Definitely not something you'd see in a major engineering publication.

Out of interest, is the paper publicly available?

Not sure. It is related to a military product so I don't really want to reveal any details to avoid trouble.

With respect to optimization, premature optimization is also bad. Yes, you can write your code such that it would need to be completely rewritten to be more efficient, but you can also write code where optimization isn't needed - and as jfb says, that can result in unmaintainable code. There is often a balance between clarity (and stability) and maximum speed.

One approach is to try to write things reasonably efficiently, but with a focus on clarity/stability/maintainability. Then run a performance analysis tool to see where you need to focus your optimization efforts.

What are some examples of being too optimised?
Title: Re: Boeing Starliner
Post by: Peter B on September 06, 2021, 11:55:00 AM
I was given a paper to read from Boeing on some outfitting they did to one of their derivatives. Kind of reads like a high school report what with the language. Things like, "engineers were mesmerised by the video."

Definitely not something you'd see in a major engineering publication.

Out of interest, is the paper publicly available?

Not sure. It is related to a military product so I don't really want to reveal any details to avoid trouble.

Yeah, no worries.

Quote
With respect to optimization, premature optimization is also bad. Yes, you can write your code such that it would need to be completely rewritten to be more efficient, but you can also write code where optimization isn't needed - and as jfb says, that can result in unmaintainable code. There is often a balance between clarity (and stability) and maximum speed.

One approach is to try to write things reasonably efficiently, but with a focus on clarity/stability/maintainability. Then run a performance analysis tool to see where you need to focus your optimization efforts.

What are some examples of being too optimised?

I assume it would be over-optimising to spend a lot of time refining code which ends up being deleted.

Like that Everyday Astronaut video walking around the SpaceX Starship - the first iteration of the first stage has grid fins that don't fold into the side of the booster during launch. Presumably once they've worked out what the grid fins should look like or even if they're needed, then they can work out how they can be designed to fold down into the side of the booster.
Title: Re: Boeing Starliner
Post by: cjameshuff on September 06, 2021, 01:35:05 PM
Boeing's lost all ability to move without someone dangling a government contract in front of them.

The majority of Boeing's profits are still from private-sector business.  Without a customer or a potential customer driving specific forms of innovation, it's hard to make the case that Boeing should use its investor funds to do the same kinds of things as younger companies are doing.  It has a fair amount to do with Boeing having become a dysfunctional company.  But it has more to do with Boeing being a publicly-held company.

I was mainly speaking that portion of their business that more or less overlaps what SpaceX and Blue Origin are doing. Publicly held companies can still take risks, try to reach new markets or start projects that let them improve their capabilities. Boeing used to do so, even as recently as the Delta IV which they initially had commercial ambitions for, but they now seem content to let ULA handle Vulcan, with costs and risks shared with Lockheed and government contracts paying for much of development, while not supporting anything more than an updated launcher to replace Delta IV and Atlas V. SLS is purely a government project with no commercial applications whatsoever, and they've shown no real interest in doing anything with Starliner beyond the NASA contract. They make commercial satellite busses, but they're content with minor variations on what they've done before. They no longer seem interested in developing new products or capabilities of their own initiative.
Title: Re: Boeing Starliner
Post by: cjameshuff on September 06, 2021, 02:04:23 PM
I was given a paper to read from Boeing on some outfitting they did to one of their derivatives. Kind of reads like a high school report what with the language. Things like, "engineers were mesmerised by the video."

Definitely not something you'd see in a major engineering publication.

Out of interest, is the paper publicly available?

Not sure. It is related to a military product so I don't really want to reveal any details to avoid trouble.

With respect to optimization, premature optimization is also bad. Yes, you can write your code such that it would need to be completely rewritten to be more efficient, but you can also write code where optimization isn't needed - and as jfb says, that can result in unmaintainable code. There is often a balance between clarity (and stability) and maximum speed.

One approach is to try to write things reasonably efficiently, but with a focus on clarity/stability/maintainability. Then run a performance analysis tool to see where you need to focus your optimization efforts.

What are some examples of being too optimised?

Optimizing a spin-delay function?

In realtime code, you have to guarantee that you have sufficient processing time for the worst case. You may have only microseconds before you have to finish one item of work and start the next. There's not really any point in optimizing the other cases, and it's sometimes actually better to write things so they always take the same amount of time. The processing cycles you consume have to be available for your use anyway, and it's easier to verify that your code can actually do the job in the time available if it always takes the same amount of time to run. If you do have different paths that take different amounts of time, any effort spent optimizing for time is wasted unless it's spent on the worst-case path.

Other times, you don't have the space to do things the fastest possible way, and have to make tradeoffs that sacrifice performance to do things in less space, such as by recomputing things as neded. Or you usually need to process data in large batches, and take advantage of GPU acceleration or SIMD instructions that don't perform nearly as well when you just need to do a small number of operations. It might take a thousand times longer for those cases, but since they're small and rare you'll never save more than a fraction of a second by optimizing for them. And init code on a long-running daemon or interactive program isn't really worth optimizing unless it's taking long enough that startup time is actually an issue. It might be months or years before you run that code again.

And in general, you can usually make a much bigger difference by a change of approach at a higher level than you can in micro-optimizing every little piece of code. Implementing a more appropriate data structure or algorithm can make a bigger difference than any amount of time spent optimizing a search of an unsorted list.
Title: Re: Boeing Starliner
Post by: molesworth on September 06, 2021, 04:50:31 PM
Different war story on the maintainability front.  We were making a sonar simulator for a Navy lab.  We handled modeling the noise spectra and the movement of objects in the simulation.  A second group built a massive DSP to take our noise spectra and turn them into the signals you’d get from a towed array.  A third contractor was responsible for a 3D graphical display.

At one point we were asked to look at the graphics code to see if we could speed it up (this was written in C on a Silicon Graphics  system using (not-yet-open) GL).  A small program, only about 5000 lines or so, but they were all in main.  The original author believed that using actual subroutines was too inefficient, so he used gotos to branch all over the place - something like 15 gotos in all. It took my coworker a couple of weeks to puzzle out the flow of control. 

The code was so tightly coupled with itself we could not make any changes without breaking something.  We tried compiling with level 1 optimization - the compiler ate all the RAM, then ate all the swap space and the system panicked.  This code was literally unmaintainable.

We finally gave the lab two choices - let us rewrite the whole thing from the keel up, or buy faster hardware.

They wound up buying faster hardware.

That takes me back to my days developing simulator graphics, also on SGI hardware (no, it wasn't for a US Navy lab  ;) ).  Luckily our team were all pretty good, but we still did have to be aware of the temptation of premature optimisation.  It is, as the saying goes, the root of all evil...

After a few years doing that I moved into games development, which took optimisation to a whole other level!!
Title: Re: Boeing Starliner
Post by: jfb on September 06, 2021, 04:52:15 PM
What are some examples of being too optimised?

Stuff I’ve run across - replicating the behavior of a library function in your code (badly) to avoid the overhead of a function call, using bitwise operations in place of arithmetic operators, using pointer notation instead of array subscript notation, being clever with bit packing, etc. 

Being clever with bit packing caused some heartburn for Mac programmers back in the ‘80s.  The original MacOS ran on a Motorola 68000 CPU, which had a 32-bit word size, but only 24 lines on the address bus.  Pointer types (used to store address values) were 32 bits wide, with the upper 8 bits unused.  Enterprising Mac programmers would stuff data into that uppermost byte to get the most out of that precious 128k of RAM.  Of course, when the 68020 came out, it used a 32-line address bus, so all that code had to be rewritten. 

Title: Re: Boeing Starliner
Post by: molesworth on September 07, 2021, 06:31:12 AM
I guess we've wandered way off topic from the thread's original subject, but I think the most outstanding piece of optimisation I've ever seen, and which still blows my mind, is the Fast Inverse Square Root.  The code looks like complete nonsense but gives a result within a percent or two of the actual value, and back in the days when Pentium processors were top of the range it was huge performance improvement, especially for graphics processing where 1/sqrt(x) is used a lot.  Even today it's still faster than general-purpose processors or libraries can achieve.

A good article on it is - https://medium.com/hard-mode/the-legendary-fast-inverse-square-root-e51fee3b49d9 - see if you can figure out how it works before reading the explanation  :) 
(Warning - code contains sweary word...)
Title: Re: Boeing Starliner
Post by: smartcooky on September 07, 2021, 07:46:02 AM
What are some examples of being too optimised?

Stuff I’ve run across - replicating the behavior of a library function in your code (badly) to avoid the overhead of a function call, using bitwise operations in place of arithmetic operators, using pointer notation instead of array subscript notation, being clever with bit packing, etc. 

Being clever with bit packing caused some heartburn for Mac programmers back in the ‘80s.  The original MacOS ran on a Motorola 68000 CPU, which had a 32-bit word size, but only 24 lines on the address bus.  Pointer types (used to store address values) were 32 bits wide, with the upper 8 bits unused.  Enterprising Mac programmers would stuff data into that uppermost byte to get the most out of that precious 128k of RAM.  Of course, when the 68020 came out, it used a 32-line address bus, so all that code had to be rewritten. 



Try dealing with anything to do with a CDP1802 (aka RCA1802). Talk about heartburn!

It was an arse of a thing to work with and trouble shoot, and even worse in the application we used it for -  the Canadian Marconi AN/APN510 Doppler GS/DA radar. This piece of equipment used the 8 bit CDP1802 in a 16 bit application. The data and address bus lines on the cct boards were all the same 8 physical lines, and the high and low order 8 bit bytes of each were multiplexed. This made a normal logic probe impossible to use because at any given moment, the logic state of  pin you were probing could have been representing any of four conditions.  Say you were probing bit 8 of the bus, you wouldn't know if you were seeing bit 8 of the High Order or Low order byte of the Data Bus or the Address Bus. What we had was a special-to-type Logic Probe (it had a special name that I can't recall) that could be clipped onto a DIL CMOS chip, with a seven segment display and a row of LEDs at the top. One if the pins picked up the multiplex synch pulse, and there were two micro-switch buttons on the side - the four possible conditions (both in, both out, one in one out, or the reverse) gave you the 1 or 0 bits on the LEDS and the hex value on the seven-segment display for whatever part of the bus you had select with the micro switches

It was a true nightmare to fault find.

NOTE: Apparently the CDP1802 was used in a number of space probes such as Magellan, Galileo and the HST.
Title: Re: Boeing Starliner
Post by: Peter B on September 07, 2021, 10:49:18 AM
To pull things back onto topic, can I point out a couple of Scott Manley videos in the last week, covering launch failures by a couple of new companies - Astra and Firefly (I assume you've all seen the sideways lift-off of the Astra). Which just goes to show that it's not just well-established companies that can suffer from rookie errors...
Title: Re: Boeing Starliner
Post by: jfb on September 07, 2021, 04:15:47 PM
To pull things back onto topic, can I point out a couple of Scott Manley videos in the last week, covering launch failures by a couple of new companies - Astra and Firefly (I assume you've all seen the sideways lift-off of the Astra). Which just goes to show that it's not just well-established companies that can suffer from rookie errors...

I'm still amazed at the Astra's ability to recover; that could have been a wrecked pad.  The guidance and control guys deserve a case of beer over that. 
Title: Re: Boeing Starliner
Post by: Peter B on September 07, 2021, 06:23:35 PM
To pull things back onto topic, can I point out a couple of Scott Manley videos in the last week, covering launch failures by a couple of new companies - Astra and Firefly (I assume you've all seen the sideways lift-off of the Astra). Which just goes to show that it's not just well-established companies that can suffer from rookie errors...

I'm still amazed at the Astra's ability to recover; that could have been a wrecked pad.  The guidance and control guys deserve a case of beer over that.

Agreed! I was so struck by what I saw that I showed the video to my kids. I explained to them that there are videos of spectacular rocket failures from the 1950s and 1960s where engine shutdowns straight after launch resulted in pretty much complete loss of control and big explosions close to the ground. Even they (after their fashion) were impressed by the Astra recovery - and laughed at Manley's "flamey-end down, pointy-end up" comment.

Having said that, as one commenter pointed out: just as well it wasn't the opposite-side engine that failed, otherwise the rocket would have slid sideways into the launch tower...

And regarding the Firefly launch, I was surprised the rocket survived its tumbling motion as long as it did. I thought rockets travelling other than straight ahead disintegrated quickly.
Title: Re: Boeing Starliner
Post by: cjameshuff on September 07, 2021, 10:17:11 PM
I guess we've wandered way off topic from the thread's original subject, but I think the most outstanding piece of optimisation I've ever seen, and which still blows my mind, is the Fast Inverse Square Root.  The code looks like complete nonsense but gives a result within a percent or two of the actual value, and back in the days when Pentium processors were top of the range it was huge performance improvement, especially for graphics processing where 1/sqrt(x) is used a lot.  Even today it's still faster than general-purpose processors or libraries can achieve.

A good article on it is - https://medium.com/hard-mode/the-legendary-fast-inverse-square-root-e51fee3b49d9 - see if you can figure out how it works before reading the explanation  :) 
(Warning - code contains sweary word...)

Most general purpose processors now have square root or reciprocal square root instructions that are considerably faster and more accurate. x86 has had such instructions for over two decades now. This algorithm is mainly only of significance today for historical interest or small embedded processors. (And in those, even the Cortex M7 I'm writing code for now has a VSQRT instruction.)
Title: Re: Boeing Starliner
Post by: molesworth on September 08, 2021, 07:50:08 AM
I guess we've wandered way off topic from the thread's original subject, but I think the most outstanding piece of optimisation I've ever seen, and which still blows my mind, is the Fast Inverse Square Root.  The code looks like complete nonsense but gives a result within a percent or two of the actual value, and back in the days when Pentium processors were top of the range it was huge performance improvement, especially for graphics processing where 1/sqrt(x) is used a lot.  Even today it's still faster than general-purpose processors or libraries can achieve.

A good article on it is - https://medium.com/hard-mode/the-legendary-fast-inverse-square-root-e51fee3b49d9 - see if you can figure out how it works before reading the explanation  :) 
(Warning - code contains sweary word...)

Most general purpose processors now have square root or reciprocal square root instructions that are considerably faster and more accurate. x86 has had such instructions for over two decades now. This algorithm is mainly only of significance today for historical interest or small embedded processors. (And in those, even the Cortex M7 I'm writing code for now has a VSQRT instruction.)

Good point - hardware FPUs are a lot better nowadays, and GPUs are just mind-bogglingly fast.  I guess I've spent too much time working on what are really low-performance processors recently, not on more widely used systems...  :)

However, even as a historical artefact the algorithm is a great example of brilliant optimisation and ingenuity.
Title: Re: Boeing Starliner
Post by: jfb on September 08, 2021, 01:32:58 PM
To pull things back onto topic, can I point out a couple of Scott Manley videos in the last week, covering launch failures by a couple of new companies - Astra and Firefly (I assume you've all seen the sideways lift-off of the Astra). Which just goes to show that it's not just well-established companies that can suffer from rookie errors...

I'm still amazed at the Astra's ability to recover; that could have been a wrecked pad.  The guidance and control guys deserve a case of beer over that.

Agreed! I was so struck by what I saw that I showed the video to my kids. I explained to them that there are videos of spectacular rocket failures from the 1950s and 1960s where engine shutdowns straight after launch resulted in pretty much complete loss of control and big explosions close to the ground. Even they (after their fashion) were impressed by the Astra recovery - and laughed at Manley's "flamey-end down, pointy-end up" comment.

Having said that, as one commenter pointed out: just as well it wasn't the opposite-side engine that failed, otherwise the rocket would have slid sideways into the launch tower...

And regarding the Firefly launch, I was surprised the rocket survived its tumbling motion as long as it did. I thought rockets travelling other than straight ahead disintegrated quickly.

Yeah.  I don't remember if it was Manley or someone on another forum, but they had a comment to the effect that was a sign the stage was a bit over-engineered and could afford to shed some weight. 
Title: Re: Boeing Starliner
Post by: JayUtah on September 11, 2021, 12:35:05 PM
Publicly held companies can still take risks, try to reach new markets or start projects that let them improve their capabilities. Boeing used to do so, even as recently as the Delta IV which they initially had commercial ambitions for...

Good point. I worked on the Delta IV payload interface.  That was back when Boeing still had some of the original impetus.

Quote
They make commercial satellite busses, but they're content with minor variations on what they've done before. They no longer seem interested in developing new products or capabilities of their own initiative.

Okay, yes, that's a convincing argument that it's not just fiduciary conservatism.  The notion that the new Boeing does stuff on the cheap has been well explored.  But a more recent problem is the plundering of assets that many American corporations seem to suffer.  The shareholders want short-term profits, which is not consistent with a ten-billion-dollar investment in a ten-year development project that may or may not pay off.  Incremental improvements to existing products would seem to produce more profits consistently in the short term, but would not lead to a sustainable company.
Title: Re: Boeing Starliner
Post by: JayUtah on September 11, 2021, 12:56:56 PM
Optimizing a spin-delay function?

Yeah, that hits close to home.  We make a signal aggregator for telemetry that can capture up to 512 channels of data at 30 KS/s.  So its cycle time is just over 33 microseconds.  A lot of filtration and aggregation has to happen, so we absolutely cannot rely on a fixed execution time or a constant execution path.  I think it does use SIMD instructions on the Intel, though.  And yes, it has ASICs and FPGAs in it too.

One of the Easter eggs in it is that the error code it sends to the downstream recorder, in the event it falls behind and has to drop data, is numerical "1202."  I'm constantly amazed at how far ahead of its time the AGC seems to be.

Quote
And in general, you can usually make a much bigger difference by a change of approach at a higher level than you can in micro-optimizing every little piece of code. Implementing a more appropriate data structure or algorithm can make a bigger difference than any amount of time spent optimizing a search of an unsorted list.

All the guys who work on that signal aggregator used to be games programmers.  Some of them go back to cartridge game days, where everything was limited and you really had to know your stuff at all levels.  I love listening to these guys talk about all the innovative crazy stuff they had to do back in the day.
Title: Re: Boeing Starliner
Post by: cjameshuff on September 11, 2021, 04:50:54 PM
One of the crazier things I've seen is a firmware-based Gigabit Ethernet implementation on an XCore processor. There was a bare minimum of hardware to support the physical interface, and one 8-core "tile" devoted to handling Ethernet processing. One core was dedicated to TX, just moving data between the serdes and RAM and doing initial CRC calculations. There was no time for jump instructions or evaluating loop conditions, so there was just a long block of output, load, and CRC instructions sized for a maximum-sized packet. The transmit code just jumped in at the appropriate spot for the actual packet size:
https://github.com/xmos/lib_ethernet/blob/b2ab6c876feae4f179a90e74e0202dfb51eb97d5/lib_ethernet/src/rgmii_tx_lld.S#L194

RX was more complicated. I never really had to dig into it, but it somehow involved two cores taking turns pulling in data with similarly unrolled loops, and two more dedicated to managing buffers and communications with the first two.
Title: Re: Boeing Starliner
Post by: molesworth on September 12, 2021, 04:38:24 AM
All the guys who work on that signal aggregator used to be games programmers.  Some of them go back to cartridge game days, where everything was limited and you really had to know your stuff at all levels.  I love listening to these guys talk about all the innovative crazy stuff they had to do back in the day.
Oh, I could tell you lots of stories from those days :D  The analysis and debugging tools weren't that great so you needed a good understanding of how the hardware worked, and the compilers weren't as good as currently so hand-optimisation at assembler level wasn't unusual.  I think that's why (older) games developers fit in well in aerospace tech jobs.
Title: Re: Boeing Starliner
Post by: jfb on September 14, 2021, 12:52:52 PM
Once again, I am grateful to have been nothing but a dumb applications programmer.  I've rarely had to deal with any kind of hardware weirdness, and that was only to address portability issues, not speed.  Speed issues were dealt with by using the proper algorithms (or lookup tables), but not to the level of "I must be able to reliably execute this task in X +/- Y ms" kinds of precision. 

I have seen glimpses of that world and want nothing to do with it.