Gemini North Recovers from a Power Outage and an Earthquake

Gemini North is recovering from some significant downtime caused by a problem with the secondary mirror control system which was exacerbated by an earthquake. 

A technical issue with the secondary mirror control computer, known as the CEM, first appeared on the night of Sunday 4 July 2021, manifesting itself as continual failures to hold guiding for longer than 10 minutes. Two members of our day crew went to Maunakea on the Independence Day holiday on Monday to troubleshoot. As they were setting up, a 5th-magnitude earthquake just north of the island brought one phase of our commercial three-phase power down. That problem persisted until after 8 p.m. HST, and caused a slew of other issues. Once power was restored, our staff members were able to leave the facility in a safe state for Monday night, and recovery was completed by Tuesday afternoon. We were then able to continue troubleshooting the CEM problem, which turned into one of the most intense troubleshooting efforts in recent years, involving staff from most departments as well as support from staff at Gemini South.

The CEM is a complex system based on very old technology and is the subject of one of the main projects in Gemini’s Obsolescence Program. Unfortunately, the CEM does not provide the diagnostics one would expect with a more up-to-date system. After swapping out some key components, we went back on sky on the night of Thursday 8 July, but the system still hung up periodically so the night’s observations were eventually abandoned. Regrouping on Friday, the team reviewed an extensive list of possible causes along with their fixes and associated risks. It was decided that CPU cooling, which had been noted as a possibility the day before, should be the top priority. The team found that the CPU fan was intermittent and replaced it. The team also purged the CPU’s glycol system and removed some air pockets. 

These efforts were successful and once back on sky on the night of Friday 9 July, the system behaved very well. The night crew executed nine separate science programs spanning multiple Gemini Participants, Subaru exchange time, and Director’s Discretionary time, and in the end we lost very little time on sky. 

Some side effects of the earthquake persist, however. Most seriously, after the power failure, NIFS slowly warmed up. Our various efforts failed to re-establish its cooling, so we are now warming it all the way up for a look at the cryogenic cold head. We will work to iron out the residual issues in the coming week or two.