Out-of-memory problem caused Mars rover's glitch

Computerworld - A shortage of memory on board the Spirit Mars rover is what caused it to become unresponsive on the Martian surface on Jan. 22, raising fears that the Martian mission might end almost before it began in earnest.

Mike Deliman, a technical staff member at Wind River Systems Inc., which provided the real-time embedded operating system used in the mission, said the problem has been re-created in testing on Earth and appears to be entirely memory-related.

"It's not a software bug, it's not an application bug, and it's not a hardware bug," Deliman said. "It's a system constraint that we ran up against."

The Spirit rover dedicates 32MB of its 128MB of RAM to the onboard Wind River VxWorks operating system and a host of science applications, and as the mission progresses, technicians are scheduled to periodically delete old files and directories to clear out the memory for reuse, he said.

But with all the excitement after the Mars landing on Jan. 3, and with data being returned to Earth by the rover, that step was not performed quickly enough by mission technicians.

"We just ran out of memory, ran out of RAM," Deliman said. "This is why we initially lost contact" with the rover. The six-wheeled vehicle runs hundreds of tasks simultaneously in normal operations, with each operation using its own chunk of RAM, he said.

The VxWorks operating system was embedded in a specially prepared, radiation-hardened 20-MHz PowerPC CPU installed on each of the rovers, along with 128MB of RAM (see story). The hardware was cutting-edge back when it was chosen in the mid-1990s, but then it had to be treated to ensure its reliability in the radiation of deep space -- a process that takes five to 10 years.

"It's like having an old Windows machine that has a very little bit of disk space [remaining]," Deliman said. "When you run up against the end of your disk, if you don't clean it up, your system becomes unstable."

For about a week, scientists worked to figure out why the rover wasn't responding to commands from Earth and feared that a hardware problem could halt the rover in its tracks.

Technicians were eventually able to correct the problem when the rover went into a diagnostic mode, Deliman said. Diagnostic commands were beamed up to the machine, and a series of files and folders were deleted from a flash-memory-based file system board, allowing the rover to resume normal operations.

The Spirit rover is now continuing to take photographs on the Martian surface and conducting experiments for NASA. A second Mars rover, Opportunity, landed on Jan. 24 and has also been operating on the surface of the red planet. A minor glitch with a heater that won't shut off on Opportunity's robotic arm is the only problem experienced by that machine so far.