OFJ Field Journal from Steven Tyler - 12/12/95
ANOMALY ON THE SPACECRAFT!So, what am I thinking about? Well, I'm thinking that it sure will be an easy day. Our January orbit trim maneuver has been cancelled. I guess I can start thinking about our March maneuver that will change our orbit dramatically. Right now, our orbit takes us within 215,000 kilometers of Jupiter's cloudtops. We want to raise that minimum distance (the "perijove") to about 715,000 kilometers. Otherwise, we'll absorb a lot of radiation in each orbit, and after a couple of orbits our spacecraft will probably be incapacitated. As a matter of fact, our present orbit is no good for another reason. When we are far from Jupiter in the orbit, the gravitational pull of Jupiter is less, and the pull from the Sun becomes more noticeable. The result is that we would be only about 35,000 kilometers from the Jupiter cloud tops at perijove if we would stay in our present path. On the following orbit, we'd crash into Jupiter. So we'll do a "Perijove Raise" maneuver (or PJR for short) to fix all this.
December looks like it won't be such a busy month. Next week, I'll host a pot-luck party for the group I'm in. Then, maybe I'll have some time to spend with my family! We can try training our almost year old dog (a Basenji named Pepper). We'll go to a couple of UCLA basketball games, an LA Philharmonic concert, and maybe the Santa Anita race track...
But now I'm at work, and I abruptly awake from my daydreaming when I realize that there is an anomaly -- a problem -- on the spacecraft. It is nearly 8 AM. At 9:30 there will be a meeting to discuss it. We may need to send a command.
What does this mean? Is there a real problem? The Sun is just about in between us and the spacecraft, and with the solar radio interference, the telemetry from the craft may not be too good. Normally, we could "read out" parts of Galileo's computer memory to check this, but we can't do this while we are getting back our Probe data.
If there is a minor problem, we might not want to do anything. The Orbiter is close enough to being behind the Sun that sending a command might be risky. We'd rather not send a command until December 29. That's 17 days away!
Luckily, it seems to be a minor problem...our power margin (which tells us how much power is left between what the spacecraft is currently using, and the maximum it has to draw on) is a little lower than expected. That could be a symptom of something serious, but in this case, we think we know what the problem is. Our Relay Radio hardware, the equipment that the Orbiter used to communicated with the probe, is in the wrong "mode."
What are we doing using the Relay Radio anyway, you may ask? Didn't the Probe vaporize more than two days ago?
Yes, the Relay Radio receivers, heroes of the Probe Relay. And now, how the mighty have fallen! We're using a Relay Radio receiver as a lowly heater! It chews up 23.1 watts in "wideband" mode. 15.5 watts in standby. 1.8 watts get used for the oscillator (which provides a frequency for the receiver to lock on to). It is just the right thing to supply enough heat to protect the computer hardware against extreme cold...
But isn't there some problem with using equipment for other than its intended duty? I remember when I first considered using this Relay Radio Hardware (RRH for short) as a heater. I saw two problems. First, one has to turn on the oscillator and let it warm up for twenty minutes, otherwise you can't be sure how much power it will use. You might be off by a couple of watts (maybe as much as 5 watts). And you need to know *exactly* how much power is being used, because the spacecraft only produces about 500 watts of power total. We can't use more than that.
Second, if you only need 15.5 watts, you would like to go straight to Standby Mode when you turn on the RRH. That saves power, and protects against overheating the computer hardware. But you can't do that. It will come on only in Wideband mode, at 23.1 watts. You'd better be sure that you can survive being in Wideband mode for a while.
And now, I'm being told that Wideband mode is where we are. The power margin has gone down by an amount that corresponds to switching from standby to Wideband mode. The temperatures in the propellant tanks are dropping appropriately (any extra power is basically shunted to the tanks, and the amount of extra power has gone down, so the tanks are cooling off). The Probe Engineering Team (you guessed it, PET for short) confirms that the RRH has switched modes (this report seems unnecessarily dramatic..."no, we are no longer in Standby, yes, we are in Wideband..." But this is fair, since they are reporting on two different pieces of data). Well, that isn't so bad. Just a few weeks ago, I helped determine our power margin strategy for this time period. I know that using an extra 7.6 watts is not ideal, but it is acceptable. We won't have to do anything about it. But I'm asking myself a few questions. What happened? WHY did the RRH change modes unexpectedly? Why wasn't our Team a little more suspicious of using the RRH as a heater?
In seconds I'm guessing about some of the answers. The answer to the final question is easiest. All of us subconsciously expected the RRH to work the right way. We expect equipment to stay in the state we put it in. After all, we wouldn't be able to fly spacecraft if equipment did anomalous things like this all the time.
It isn't that we wouldn't be prepared for an anomaly. Nor would we be shocked if a piece of equipment failed and turned itself off. But I'm not used to wondering about whether or not a 7.6 watt load will unexpectedly go ON, especially in the absence of a short circuit!
But now that it has happened, I'm not surprised at all. After all, the RRH was not supposed to be used as a heater. It was supposed to receive the Probe data! And that meant there was one awful thing it was never, ever supposed to do. It was *never* supposed to be in Standby during Probe Relay. That could have destroyed the entire Probe mission. No, the RRH was supposed to be in Wideband during the probe mission.
Hmm. Since it could be a disaster for the RRH to be in Standby, engineers designed the RRH so that any power glitch (and maybe any software glitch as well) would automatically put the RRH in Wideband mode. Wideband is where it wanted to be, and Wideband is where it would stay.
That meant that using the RRH in Wideband as a heater was probably reasonable. But putting it in Standby and expecting it to remain there might be asking for trouble.
In addition, we didn't have much real spacecraft experience with using the RRH in Standby mode before now. There was no reason to use the RRH before Probe Relay except for four or five sessions of checking it out. Any more and we would have been afraid of breaking it. So we had it on in 1989, eight days after we launched, in 1990, twice in 1992, and a couple more times this year. My guess was that the RRH had been in Standby only a couple of times since launch, and not for very long either time.
There is plenty of time before our meeting about the anomaly, so I go down from my eighth floor office to the fifth floor Mission Support Area to see how the spacecraft is doing. As I said, the data are going to be poor, because the Sun is "in the way." (Did you know that "data" is plural? When I was in high school, I did. But I always said "the data IS bad" when I really meant "the data ARE bad").
Sure enough, the first things I see on the Systems screen are a bunch of alarms. (In red, of course. That makes sense.) Now, are any of them real? The alarms say that we requested a standard response to a problem on the spacecraft -- "Safing." (As the name suggests, "Safing" puts the spacecraft in a "safe" state after it has developed some problem, which should keep anything else from going wrong.) But we can't just trust the alarms. Are we really in Safing? It would be no surprise to me if we *were* in Safing. It is easy to believe that the problem that caused our RRH to go to Wideband mode could trigger such a response.
Kauser Dar is the System Fault Protection engineer, and I'm her backup. At the 9:30 anomaly meeting, she'll be asked if the spacecraft is really in Safing. That means both of us ought to have the answer. We are looking at the screens and checking what they mean in our System Fault Protection Design books. The data say yes, we have requested Safing. But it doesn't make sense. Safing would cancel our request for Probe data. And we're still getting back Probe data!
Kauser quickly reminds me that if we were in Safing, the Power margin should go up by a couple of watts, not down by about 8 watts. At first, I'm not convinced. There are a number of details I'm not sure of (such as the RRH state in Safing, and some other indicators of executing Safing that might be available). We ask the engineers who work on the spacecraft's main computers what they think is happening. Just then, almost as if the spacecraft is replying, a new Probe data readout begins. That simply can not happen if we are in Safing, so the discussion is ended.
At 9:30, there is an Anomaly Team meeting. Bob Barry, who did such a fine job in leading the design of the Jupiter Orbit Insertion sequence, is leading it. The evidence is presented for the RRH being in Wideband mode. It is pretty convincing. Now, what should we do about it?
Normally, we would quickly decide to do...nothing. But there is a problem. A few months ago, we realized that a valve to one of our tanks (one that holds oxidizer, which is needed to burn the spacecraft fuel) might be leaking oxidizer (not into space, but into a line which leads to the fuel tank). If we are not careful about what temperature we keep our tanks at, we might get more oxidizer leaking into the fuel tank (not enough to be a hazard, we think). Is it safe to let the tanks cool down? This is a question for the engine people. They agree that it is safe to cool down.
Now we've got a few minutes before the Project management asks us for our recommendation. There are four choices:
1) Do nothing (send no immediate commands to our spacecraft). We'll cool off the propellant a little more, but this might be completely safe and at worst provides a very small future additional risk.
2) Put the RRH back to Standby mode. This warms the propellant up right away. However, the propellant is already more than halfway to the final temperature it will reach if we simply leave things alone. If there is a risk, it will be in warming back up. Also, it is getting dangerous to send a command since the spacecraft is almost directly "behind" the Sun in the sky, and the Sun is interfering with radio communications.
Besides, if we put the RRH back to Standby, aren't we just asking for it to return to Wideband after the next glitch? We don't want to have this same problem pop back up!
3) Switch to other, smaller heaters, so that we stop cooling down, but so that we don't warm up either. This sounds tricky, doesn't it? But engineers like us ought to be able to figure out how to do it fairly quickly. The problem is that we have already cooled down a lot, so we save very little if anything with this plan. We also do not have a lot of time to figure out which heaters to use. It would be easy to make a mistake. We might pick a heater that works now. But there's just been a problem on the spacecraft, and it's not safe to assume that everything is working perfectly. What if there's a nasty surprise, and we wind up in an unsafe state? What if we miscalculate, and choose heaters that mess up our Safing strategy?
4) Quit sending back Probe data, so that we can watch the spacecraft more. But this doesn't help us much, and the Probe data is very high priority.
The Project meeting is fairly short. To nobody's surprise, we decide to send no immediate commands to the craft. After December 29, when the spacecraft comes out from behind the Sun, it may be completely safe to send some commands to go back to our previous power margin. If not (or if we aren't sure), we'll wait until it is safer, namely at Perijove raise in March.
But the Orbiter Engineering Team still has more work to do. We still have to figure out why we think the RRH changed state. And there is a smoking gun. There was a problem with how power was being distributed around the spacecraft at about the same time as we changed RRH states (we can't be sure whether it was at the same time, because rate at which data is being sent from the spacecraft is slow right now, and we get a report on the RRH state only once every two hours). Tomorrow, we'll look at the spacecraft's circuit diagrams to see if this could actually affect the RRH. It's a start in figuring out what went wrong.