Author Topic: "The AGC wan't powerful enough"  (Read 1411 times)

Offline ka9q

  • Neptune
  • ****
  • Posts: 3009
Re: "The AGC wan't powerful enough"
« Reply #45 on: December 04, 2019, 11:33:15 AM »
A similar situation happened during the fatal Columbia re-entry.  As temperature sensors and other sensors started going offline, the flight controllers were looking for systemic commonalities.  It wasn't until much later in the troubleshooting process that they realized all those sensors were going offline because they were being destroyed -- the commonality was that they were in the rapidly heating part of the orbiter.
Again I think you're being hard on the controllers. They were looking for commonality because they knew that individual sensors, especially mechanical ones like temperature and pressure, often fail for many different reasons and it's difficult or impossible to incorporate fast, reliable self-test features. The last thing you want is to take some drastic action on the basis of false input, so the first thing you must do is to determine if the reading(s) are real. It's difficult to know whether a single non-nominal reading is real or not, so they rely on redundancy -- one of the most time-tested ways to detect and overcome random failures. If a set of nearby temperature sensors give similar out-of-range readings, and those sensors don't share a lot of support hardware ("no commonality"), the readings are probably real.

Of course, none of this really mattered in the Columbia disaster; there was absolutely nothing the ground could have done to prevent it once the entry began.
« Last Edit: December 04, 2019, 11:35:25 AM by ka9q »

Offline ka9q

  • Neptune
  • ****
  • Posts: 3009
Re: "The AGC wan't powerful enough"
« Reply #46 on: December 04, 2019, 11:43:17 AM »
I just can't pass up this opportunity...
Good on you.  I figured Confuse-A-Cat might have been too obscure a reference.
Not to a Python (Monty) fan like me.
Quote
Even better, this channel is disabled unless the crew explicitly switches the Uplink switch to Accept.  This too has tradeoffs.  What if the crew are incapacitated and can't enable ground control of the computer?
The uplink enable switch was put there for a specific reason -- preventing the Russians (or anyone else) from injecting false commands. It was a manual form of the (in)famous CRM-114 device in "Doctor Strangelove". Today we'd do it with cryptographic authentication, but that wasn't practical in the 1960s.

Offline JayUtah

  • Neptune
  • ****
  • Posts: 3195
    • Clavius
Re: "The AGC wan't powerful enough"
« Reply #47 on: December 04, 2019, 11:58:30 AM »
I thought P67 was a full "manual" landing?

It's billed as such, especially since the ROD switch operated as a direct throttle setting control in that program.  Someone will have to verify what mode the DAP was in, as opposed to either Auto or Att Hold for P66.  P67 was removed starting with Apollo 13.  It's important to realize the AGC was not out of the loop in P67 because, obviously, we're referring to an AGC program.

In P66, the ROD switch selects the descent rate.  In P67 it selects the throttle setting in discrete intervals.  Each still involves the computer, because at its heart the ROD switch is just a switch with two affirmative outputs:  up, or down.  How the two bits representing that input are interpreted is still a matter of software.  In P66, the up-bit, if newly set, is interpreted by the software as adjusting the set-point delta-h to be 0.5 foot per second slower than its current setting.  That value in turn is used in a software control law, along with other parameters, to derive a control output that represents a change in throttle setting.  The control law suddenly sees an "error" between the desired sink rate and the deduced sink rate.  It has to increase thrust in order to slow the sink rate, then decrease thrust again to maintain a steady descent once the desired sink rate is achieved.

In P67, a newly set up-bit bypasses the control law and simply adds a fixed incremental value to the register in the AGC that represents the throttle setting.  The reason this was initially contemplated is that it's very hard for something to go wrong with that logic in the computer, presuming the computer hardware is minimally functional.  An alternate implementation could conceivably bypass the computer altogether and connect the switch directly to the pintle position servo on the DPS, using appropriate electronics.  But in reality, if the AGC became entirely inoperative, the only real remedy was the Abort button, allowing AGS to take over and disabling PGNS completely.

In contrast, the joysticks on both spacecraft had a hardover mode.  Normally the position of the joystick was just a digital value given to the computer.  That is, it required a computer to interpret that digital value in terms of what mode the software was told to obey.  There's no one right way.  If I move the joystick to the right, it's clear that means some variant of "roll to the right."  But one of the possible questions is what happens when I let go and the joystick springs back to the detent?  Does that mean I've achieved the desired roll rate, or the desired roll attitude?  In case something goes wrong with the logic that lets that question be answered in whatever way the pilot selects, moving the joystick all the way to the right clicks the hardover switch on that extreme, and that's connected directly, electrically, to the wires that control the solenoid valves on the roll-specific RCS jets.  Unless something is very wrong with the spacecraft, it would be hard for that method of control to become inoperative.  Now it would be quite hard to fly the LM in that mode, but it could be done.

One of the questions ka9q and I are debating is the notion of flyability in the face of computer failure.  One theory of responding to an exigent crisis is to take whatever action leads you to the most achievable stable state.  What exactly that state looks like depends on what point of your mission you're in.  The space shuttle had an abort-to-orbit mode during the ascent.  This is because ascent is not a stable state.  Some degree of control must be actively maintained in order to avoid catastrophe.  Orbit is a stable state -- even a slapdash orbit.  It can be maintained for some time simply by passive physics.  The point is to achieve a state in which diagnosis and contemplation have time to occur, without constantly having to address exigencies.  Similarly Apollo 13 had an exigent life-support crisis; a stable state had to be achieved first before moving on.

The 120x alarms occurred early in the descent.  In the face of dire failure then, the most readily achievable stable state is a return to lunar orbit.  Not so later in the flight, where aborts become more dodgy.  If the LM remained minimally flyable (compare Apollo 10), it may have been a more stable situation to fly to a landing -- any landing.  Sitting comfortably on the lunar surface is a stable state.
"Facts are stubborn things." --John Adams

Offline JayUtah

  • Neptune
  • ****
  • Posts: 3195
    • Clavius
Re: "The AGC wan't powerful enough"
« Reply #48 on: December 04, 2019, 12:19:17 PM »
Again I think you're being hard on the controllers.

I'm not trying to be hard on anyone, and I agree the controllers acted appropriately.  We tend to think this way, even when we haven't been trained to do it.

Quote
The last thing you want is to take some drastic action on the basis of false input, so the first thing you must do is to determine if the reading(s) are real.

Correct.  Catastrophizing is the other extreme, and it's harmful too -- perhaps even more so.  Even more to the point, we prefer that operators do nothing if they don't understand the whole set of indications.  If the readings don't describe a coherent state of the system, don't change the state of the system so as to wreck what coherence there is, or may emerge.

My argument here is a bit hindsightedly circular.  I don't mean to say that operators shouldn't first vet the indications, nor that they shouldn't take clear action on the basis of some exigent subset of those indications.  In all the cases I'm looking at, where operator action mattered, the operators achieved an acceptable outcome, largely through quick-yet-methodical analysis.  (I haven't talked about Three Mile Island yet, though.)  But what happens inevitably is that people will start a deeper troubleshooting exercise while they're still dealing with exigency.  There isn't often a clear demarcation between when operators stop swerving and start trying to figure out what's wrong with the steering.  What we discover, when these types of situation do result in failure, is that the operators erred on the side of de minimis thinking when there were indications that a larger problem was afoot.  They downplayed nonconforming indications.  But this is only when they failed.  It's not a question of what operators typically do and why; it's a matter of what operators did wrong when they failed, and why.  They don't always fail, even when they start going down the de minimis hypothesization path.  Initially yes, you apply de minimis thinking because, as you say, most problems have minimal causes.  It's what you do when new indications stop adding up that interests me.

We don't tend to err on the side of catastrophization.  We tend to do too little, not too much.  This is really only an academic conclusion; it colors how we try to train people to operate complex systems, but it doesn't mean we habitually suck at it.
"Facts are stubborn things." --John Adams

Offline JayUtah

  • Neptune
  • ****
  • Posts: 3195
    • Clavius
Re: "The AGC wan't powerful enough"
« Reply #49 on: December 04, 2019, 12:28:52 PM »
It was a manual form of the (in)famous CRM-114 device in "Doctor Strangelove". Today we'd do it with cryptographic authentication, but that wasn't practical in the 1960s.

One of our in-house cryptographic software modules is titled "crm114".  But by the same token, that's why some of our nuclear warheads have purely mechanical fusing and detonation mechanisms.  There's no way to electronically fuzz something that doesn't rely on electronics to operate.  If I were defending against a nuclear attack, I'd certainly want to see if I could get the warhead to detonate early, at a relatively safe altitude.
"Facts are stubborn things." --John Adams

Offline Obviousman

  • Mars
  • ***
  • Posts: 476
Re: "The AGC wan't powerful enough"
« Reply #50 on: December 04, 2019, 03:30:48 PM »
If I move the joystick to the right, it's clear that means some variant of "roll to the right."  But one of the possible questions is what happens when I let go and the joystick springs back to the detent?  Does that mean I've achieved the desired roll rate, or the desired roll attitude?

Weren't they two of the modes available in the Mercury AFCS? One that simply stopped the thrust and the roll / pitch / yaw rate remained at whatever it was, and the other mode held the attitude when the hand controller was returned to a neutral position?

IIRC the astronauts preferred the latter mode but it used significantly more fuel. Also IIRC, for this reason I think it got removed from later missions.

Offline Abaddon

  • Saturn
  • ****
  • Posts: 1100
Re: "The AGC wan't powerful enough"
« Reply #51 on: December 04, 2019, 05:53:54 PM »
Good on you.  I figured Confuse-A-Cat might have been too obscure a reference.

The idea behind fuzzing is that you're not necessarily trying to exercise some specific means.  Take lock-picking, for example.  The overt approach is to twist the lock barrel to apply shear to the lock pins, then use a tiny pick to raise each pin to the appropriate position.  The constant shear holds it there while you work on the other pins.  But there's also a sawtooth tool you can just randomly slide in and out while you told the barrel in torsion, and it sort of randomly raises and lowers the pins.  It's often much faster than the explicit method, but it requires less skill.  This would be equivalent to fuzzing the lock.

That is amusing. I pick locks as a hobby. I have a nice set of Southord's picks and craft my own. To me, it is simply another form of puzzle. Your summary is spot on. The "sawtooth" tool to which you refer is known as a rake. While it is to me, a hobby (I have a long held interest in puzzles of all kinds) it occasionally comes in handy. "raking" will instantly open any piece of common office furniture. In extremis, I have improvised with a paper clip and a nail scissors. Still works like a charm.

Offline Obviousman

  • Mars
  • ***
  • Posts: 476
Re: "The AGC wan't powerful enough"
« Reply #52 on: December 04, 2019, 06:38:28 PM »
It was a manual form of the (in)famous CRM-114 device in "Doctor Strangelove". Today we'd do it with cryptographic authentication, but that wasn't practical in the 1960s.

One of our in-house cryptographic software modules is titled "crm114".  But by the same token, that's why some of our nuclear warheads have purely mechanical fusing and detonation mechanisms.  There's no way to electronically fuzz something that doesn't rely on electronics to operate.  If I were defending against a nuclear attack, I'd certainly want to see if I could get the warhead to detonate early, at a relatively safe altitude.

Off topic but I love some of the obscure homages you see in movies et al. One of my favourites is in Star Trek: First Contact, when they are trying to detach the deflector dish and have to use a manual control unit labelled 'AE35'.... which was the name of the transmitter component in 2001 - A Space Odyssey that HAL says was going to malfunction.

Offline JayUtah

  • Neptune
  • ****
  • Posts: 3195
    • Clavius
Re: "The AGC wan't powerful enough"
« Reply #53 on: December 04, 2019, 08:24:02 PM »
...a manual control unit labelled 'AE35'

As I get more and more into conceptual design for entertainment, I find more and more places to put such Easter eggs.  So do my colleagues.  When the crew of Serenity go to planet Miranda, the rescue ship is designated C57D.  A surprising number of designers are also big fans of Space: 1999.
"Facts are stubborn things." --John Adams

Offline Everett

  • Venus
  • **
  • Posts: 46
Re: "The AGC wan't powerful enough"
« Reply #54 on: December 05, 2019, 02:51:46 PM »
Actually, I'd say the astronauts 'did' manually land the LM, even if had a fly-by-wire system the controls were running through. If you consider they didn't simply because of that, then by that logic it's impossible to 'manually' land a fully functioning say, 777, even on a empty dry salt lake by looking out the window, since the pilot's controls are still going through a computer. The LM case is more complicated, since they're using modes that, for example, hold constant vertical speed, but they're also telling it exactly what to do and which way to fly at any given moment, but that's opposed to say, an autoland on a airplane, where the computer decides where to do to follow an ILS signal. To put it another way, on the LM the astronauts give the LM only-in-some-aspects commands, while an automatic landing would give a computer a task.

Am I making any sense?

Offline smartcooky

  • Uranus
  • ****
  • Posts: 1695
Re: "The AGC wan't powerful enough"
« Reply #55 on: December 05, 2019, 07:09:24 PM »
It was a manual form of the (in)famous CRM-114 device in "Doctor Strangelove". Today we'd do it with cryptographic authentication, but that wasn't practical in the 1960s.

One of our in-house cryptographic software modules is titled "crm114".  But by the same token, that's why some of our nuclear warheads have purely mechanical fusing and detonation mechanisms.  There's no way to electronically fuzz something that doesn't rely on electronics to operate.  If I were defending against a nuclear attack, I'd certainly want to see if I could get the warhead to detonate early, at a relatively safe altitude.

Off topic but I love some of the obscure homages you see in movies et al. One of my favourites is in Star Trek: First Contact, when they are trying to detach the deflector dish and have to use a manual control unit labelled 'AE35'.... which was the name of the transmitter component in 2001 - A Space Odyssey that HAL says was going to malfunction.

Some of the "messaging" can be more direct than that, with the programmer literally talking to who ever is using the program.

Back in the 1980's, I was quite heavily involved in programming Apple ][+ computers to analyse data from the output of a photoelectric photometer being used to observe variable stars. This programming was done in 6502 "assembly language" using a screen not unlike this one...



... and directly keying in sequences of hexadecimal numbers. I got quite good at "seeing" sequences of hex numbers, reading them as groups and recognizing them as program instructions. In the middle are the sequences of Hex numbers [00 to FF), with their ASCII interpretations in the right column./

One day, I was examining a subroutine written by another programmer for a different purpose, and trying to figure out how to integrate that subroutine into what I was doing - in other words, I was hacking his program and trying to pinch part of it to use it for my program. While doing this, I suddenly hit a sequence of hexadecimal numbers that I didn't immediately recognize as program instructions - they were letters, and when I looked over in the right column, I got a message from the programmer....




WHAT THE HELL ARE YOU DOING POKING AROUND IN HERE!

     
► What you can assert without evidence, I can dismiss without evidence
► When you argue with idiots you risk being dragged down to their level and beaten with experience.
► Conspiracism is a shortcut to the illusion of erudition

Offline bknight

  • Neptune
  • ****
  • Posts: 2891
Re: "The AGC wan't powerful enough"
« Reply #56 on: December 05, 2019, 10:45:45 PM »
Caught with your hand in the cookie jar.  :o
Truth needs no defense.  Nobody can take those footsteps I made on the surface of the moon away from me.
Eugene Cernan

Offline raven

  • Saturn
  • ****
  • Posts: 1472
Re: "The AGC wan't powerful enough"
« Reply #57 on: December 06, 2019, 06:39:59 PM »
I laughed, thank you, a delightful little Easter egg.