OFJ Field Journal from Steven Collins - 11/21/95It's Tuesday the 21st of November. I'm writing this from the MSA. The MSA (Mission Support Area) is the "control room" for the Galileo spacecraft. From here we send commands and monitor the telemetry from critical events on the spacecraft.
It's a smallish room at the corner of the 5th floor of building 264 here at JPL in Pasadena. Out one set of windows is a view of JPL's huge, cylindrical Space Simulator where Galileo was tested before beginning its long journey to Jupiter. Out the other windows is a hazy but beautiful view of the San Gabriel mountains, a great place to ride mountain bikes (but you had better like to *climb*). A third window provides a view of *us* from the Viewing Room. The managers sit in there to keep an eye on things when a "Big Event" (tm) is taking place.
On two walls of the MSA, a pair of digital clocks with big, red letters show the current day and time: UTC 325/23:33:27. A third clock counts down the time till Jupiter Orbit Insertion (JOI): JOI -16/01:46:00. Yikes! only sixteen days left!!
The room is arranged as a set of computer displays and work areas for different groups of engineers. A little blue sign hangs over each area: Telecom , Systems, SFP (system fault protection), CDS (command and data subsystem), Mission Control. Each station has a computer with a pair of screens to display telemetry and voice-net terminal we can use to talk to each other. When we need to say something official like, "Roger Systems, you are go to send command package "X-ray Zulu Three Four Niner."
The sign over *my* head reads AACS (Attitude and Articulation Control System). I'm here today to watch the final test of our troublesome tape recorder before JOI. The tape recorder is really part of the CDS, and normally an AACS guy like me wouldn't be involved, but when the recorder anomaly occurred, I volunteered to help the CDS unit and the anomaly team collect and plot the recorder data.
To be honest, I sort of crammed my help down their throats. I have a lot of experience using the telemetry system and have spent many hours searching through old flight data trying to solve problems for AACS, so I just started plotting up recorder data and giving it to people. My plots seemed to be helping folks and pretty soon I found myself being invited to anomaly team meetings and given work to do. When a major problem like this comes up, everyone on the team contributes where they can.
For a couple of weeks after the anomaly, everyone on the team was working 7 days a week and late into the night to try and figure out what was going on. It was *very* exciting to be involved. It's sort of like trying to solve a Sherlock Holmes mystery, where you have to find pieces of evidence and then fit them together into a theory about what happened and how to fix things.
Sometimes you get false clues and they make you pick the wrong "suspect". For example, we have a computerized simulation of the spacecraft that we call "The Testbed". It's built out of our spare flight computers and hardware. We use it to test command sequences on the ground before we send them to the spacecraft and to help us figure out problems.
Now get this: on the *very same day* that the spacecraft recorder anomaly happened, the spare recorder in our testbed also failed! People were saying that it was a million to one odds that both recorders would fail on the same day, so our first "prime suspect" was some kind of problem with the computer software that controls the tape recorder.
After looking much closer at the telemetry from the flight recorder and eventually taking the testbed recorder apart and looking inside, it is now clear that the two failures were really unrelated! The software is fine and we just beat the odds and had two independent hardware failures on the same day!!!
Anyway, back to the MSA. By getting involved in the anomaly, I learned a lot about how the recorder works and build some tools for decoding and plotting recorder data. That's what I'm doing today. We are doing our fifth test of the recorder, running it for a couple of seconds to make sure that it still works and that the tape isn't getting stuck to anything.
My job is to collect some data on the recorder and plot it up so the real tape recorder experts can make sure everything is working ok. This is our last chance to check on it before we use the recorder to save the science data coming up from the Galileo Probe on 7 Dec.
The data I want is coming down as a MRO (Memory Read Out). That's where the spacecraft just sends down a copy of a little section of its computer memory instead of its normal stream of telemetry channels. At my workstation under the blue AACS sign, I set start up a program to capture and save the MRO data as they come down.
There's the data, right on time. On my screen can watch the data being printed out as I type this. It looks kind of like this:
ERT: 95/325-22:18:16.670 BUC: 8b Module: DBUM-1B SCLK: 03186602.81.0.0 5100: ffff ffff ffff ffff ffff ffff ffff ffff ffff ffff 8121 1a21 121a 8b81 8b1b 8121 1a21 121a 8b81 8b1b 8120 1a20 121a 8b81 8b1b 8121 1a21 121a 8b81 8b1b 8122 1a21 121a 8b81 8b1b 8121 1a21 121a 8b81 8b1bIt takes a while for all the data I need to come down, maybe 15 minutes. When I have it all, I copy it to a file and run my little program on it. It slices it up, moves it around, figures out what data happened when. Finally, it makes a set of plots that I can hear dropping out of the printer. As far as I can tell, everything looks just fine. But that's for the real recorder experts to decide. I call the beeper number I've been given and Greg Levanas (our version of Sherlock Holmes) responds, He says he'll be up in a few minutes to have a look.
I use the time to Xerox copies of the plots for him and finish this report.