Summary of Hydra-CTIO Maintenance Work (2001)

November 2001

Knut Olsen, on behalf of the people who work on Hydra: Rolando Cantarutti, Rodolfo Cardemil, Manuel Martinez, Daniel Maturana, Andres Montane, Javier Rojas, Oscar Saa, Nick Suntzeff, Ricardo Venegas

The Hydra spectrograph and positioner have had a history of problems since its commissioning. In the months July-September 2001, good progress was made towards fixing the worst problems. Below are summaries of the problems we have encountered and the work done to fix them.

PROBLEMS W/ HYDRA-CTIO REPORTED THROUGH October 2001

POSITIONER:

1. Z-motion limit switches frequently tripped, loss of time ~0.5 hour per occurrence.
    
2. Frequent trips to the cage to remove fibers stuck in the gripper jaws or crossed with other fibers, loss of time ~0.5 hour per occurrence.
    
3. During June/July 2001 block, once per field GUI would report that it could not pick up a button, even though the TV camera showed the gripper to be positioned immediately above the button. Restarting the configuration would solve the problem, so minimal loss of time.
    
4. During early part of October 2001 block, many fields generated "transition" errors (sometimes several per field), which required parking fibers in the setup file and reconfiguring. Loss of time a few minutes per occurrence.
    
5. Prior to June/July 2001 block, frequent reports that guiding appeared to drive the guide stars to the edges of FOPS fibers. This problem may in the worst cases have reduced the efficiency by ~50%.
    
6. Occasional errors of "xy motion failure: stages not at destination". Restarting the configuration generally succeeded.
    
7. Many different versions of Hydra simulator and concentricities file on various mountain computers.

The problems with the positioner resulted in the loss of often several hours per night.

COMPARISON SYSTEM:

8. Frequent burnout of Ne penray lamp.
    
9. Penray line ratios not well balanced.
    
10. In echelle mode, many reports of lines being weak and/or difficult to identify. Nick noted that HeNe lamp contained neither He or Ne, but instead Ar.

BENCH SPECTROGRAPH:

11. Frequent problems with shutter moving to wrong position.
     
12. Many light sources in the coude room, as measured both through Hydra dark frames and images taken by Roger & Ramon with a cooled CCD camera placed in the room.
     
13. Region spanning ~100 pixels at one end of the CCD rolls off sharply in intensity, suggesting vignetting in the spectrograph.
     
14. Bias jumps of several ADU, sometimes occurring during night but most frequent right after dewar refill.
     
15. No RTD with 1x1 binning.
     
16. Solid N2 needed to cool CCD sufficiently, requires constant pumping.
     
17. Fiber mount can't be tilted far enough to straighten spectra in echelle mode.

FIXES AND MAINTENANCE

Many thanks to Sam Barden, Phil Massey, Dave Sawyer, and Bezhad Aredeshi for their advice on fixing Hydra's troubles.

POSITIONER:

Our top priority for Hydra before the October 2001 observing block was to reduce the failure rate of the gripper. Many of the problems experienced in previous runs, listed above, have now been fixed:

1. At first, the problem of the Z-limit switch tripping inadvertently was thought to mainly be due to a short circuit between closely packed wires inside the switch. In 2000, Javier Rojas and Esteban Parkes partially disassembled the switch, then wrapped and separated the wires to prevent their contact. After this work, the problem appeared solved, but then reappeared several days later. In April(?) 2001, Ricardo Venegas (and others?) noticed that the travel allowed by the switch in the positive Z direction was very small. Since their tuning of the switch to increase the range of possible Z motion, the problem has disappeared.
    
2. & 3. Early discussions of the problem of occasional crossed fibers came to the conclusion that the buttons were being gripped too tightly during transport; under normal operation, the gripper holds the button in a "relaxed grip" while moving fibers (aided by the magnets in each of the gripper jaws), only closing tightly while picking up or placing buttons on the plate. This conclusion appeared corroborated by the observation that fibers moved at large angles from the plate radius were bent into increasingly severe S-curves. The immediate solution was to limit the angle through which the fibers could bend to 3 - 5 degrees, which solved most, but not all, of the problems with crossed/colliding fibers.

However, with the appearance of another problem with the gripper (#3), it became clear that a complete overhaul of the gripper was necessary. Andres Montane (hardware) and Rolando Cantarruti (software) jointly took on this work, with Ricardo Venegas supporting.

In order to uncover the cause of the gripper troubles, Andres & Co. needed to understand how the gripper behaves during the sequence of picking up a button, moving it, and putting it back onto the plate. What is supposed to happen? From Sam Barden:

• The gripper closes to sense that it does not have a button.
• The stages move over the position of the button. The gripper opens.
• The gripper is lowered over the button. The gripper closes.
• The gripper is raised to cruise height. A test is made to see if the button was picked up.
• The gripper moves to the relaxed position.
• After the stages have moved to the new position, the gripper goes to the closed position.
• The gripper lowers button to plate. Gripper jaws open.
• The gripper raises to cruise height. Gripper jaws close, to check that the button was released from the gripper jaws.

The gripper thus cycles through three states while placing a button: open, closed, and relaxed. (A fourth state, "open wide", is also available, but is used mainly when the gripper is initialized). How does the gripper achieve these states? In Fig. 1 above, the three teeth of the gripper are mounted on the black wheel with red LEDs attached. When this wheel rotates, the teeth tilt about their mounting pins and the jaw opens or closes. The motor which provides the power to rotate the wheel is located immediately above the label "C". "A"-"D" indicate four mechanical switches. From our observations of the gripper in April 2001 and conversation with Sam Barden, we established that:

• A is activated when the gripper is closed.
• B is activated when the gripper opens "wide".
• C adjusts the gripper tension for the relaxed position.
• D defines the open position of the gripper.

Andres&Co. independently established these functions of the switches. Their real insight, however, came from Andres' understanding of the mechanics of the system. Andres first observed that the turning of the gripper motor compresses the spring in switch C and opens the gripper. The closing of the gripper is then entirely driven by the stored tension in the spring. Second, given the distance by which the teeth are supposed to move in order to reach the open position, Andres calculated the distance through which the motor needs to turn. He found that the motor was stopping before it reached the full open position, held back by excessive force from the spring and by a spring in switch B. After he adjusted the positions of switches B and C, the gripper was able to complete the opening of the jaws. (Incidentally, Rolando discovered that the setting of switch B is never used in the software control, so that it actually serves no purpose). Third, Andres noted that when closing the gripper with a button in the jaw, the teeth continued to rotate even after they were fully closed around the gripper. By tuning switch A, he was able to stop the closing of the jaw before the teeth imparted rotation to the button.

Following Andres', Rolando's, and Ricardo's work on the gripper, the October observing block passed with no trips to the cage to untangle problems with the fibers, and virtually zero problems with the gripper. Their work leaves us with two valuable lessons:

• The gripper motor and the springs in limit switches B and C are in a delicate balance, and need to be looked at prior to each Hydra block
• Our earlier observation that fibers were often twisted into S-curves is likely not due to their failure to rotate with the jaw in its relaxed position, but instead to switch A being out of tune, which caused the buttons to rotate as the jaw closed

Andes will refine the process of tuning the gripper before the next Hydra block.

4. As an additional preventative measure to avoid crossed fibers, Rolando rewrote the Hydra positioning software to compute fiber trajectories as straight, rather than curved, lines. Observers who configured their fields with the old simulator thus experienced a number of "transition errors" when moves which were allowed by the old version of the software were found to be illegal by the new version. The errors were also seen in fields which were observed at significantly different sidereal times from the ones for which they were configured (as is to be expected). In fields with a number of transition errors, recovery was time-consuming, since the problem fibers had to be parked manually and the errors popped up one at time. Rolando has modified the software so that it checks for ALL problem fibers in a single step, reports the errors, and offers the option to leave the problem fibers parked while configuring the rest. This modification saves a great deal of time when transition errors are encountered. We are also taking steps to maintain the mountain computers with up-to-date simulators and to advertise the latest versions visibly.
    
5. In April 2001, Ricardo, Rolando, and Knut remeasured the centroids of the FOPS fibers and their boundary boxes. For many of the fibers, they found significantly different centroids from those used since the commissioning of Hydra. Fig. 2 demonstrates what happened. At some point, the guide camera was rotated with respect to the FOPS fiber set, causing the guider to guide off-center with respect to the guide star. Indeed, Sam Barden recalls having to rotate the camera slightly in order to mount it properly.

6. This problem has been traced to a problem with the Galil box which controls the motion of the gripper stage. Unfortunately, the spare Galil box also has a problem which renders it unusable. The spare has been sent back to the manufacturer for repairs.
    
7. Software and concentricities files are now being kept up-to-date. See 4.

COMPARISON SYSTEM:

8. Javier has checked the Ne penray power supply, and found it supplies the same voltage as does the Xe. He thus finds it unlikely that the power supply is burning out the lamp. A more likely possibility is that the lamps have a gas leak. Needs more work. Investigating the comparison system will be given high priority during the next engineering run.
    
9. Nick and Knut will decide on what filters could be placed over the penrays to balance the strengths of their lines.
    
10. Javier and Ricardo have extensively tested the ThAr lamp. They have compared the effects of placing it where the Argus lamp used to sit and of using the Argus lamp power supply. Although some differences appear when using different power supplies, qualitatively the features do not change. The major difference is the much lower brightness of the Hydra system, but it is exactly as expected considering that the light is diffused over a 40 arcminute field. The HeNe lamp system in the box in the cage remains to be tested in similar fashion.

BENCH SPECTROGRAPH:

8. Prior to the October block, we experienced frequent problems with the SMC shutter control losing its fiducial, which would result in the shutter going to the wrong position. This would generally happen at the beginning of each observing block, although sometimes in the middle, and required Rodolfo Cardemil to adjust the system. Rodolfo has now disabled shutter control from the SMC, and Manuel Martinez has designed and installed a simple system using a motor with two mechanical positions. The new shutter system worked flawlessly throughout the October observing block.
    
9. Roger & Ramon have expended great energy in reducing stray light inside the Coude. They have imaged the room with a cooled CCD camera, and identified many light sources and light leaks in the room. As a result of their efforts, a number of steps have been taken to get the CCD dark current below 1 e-/pix/hour--see Hydra Dark Current II for a summary. However, this dark current is only achieved at night with the fibers covered. More effort is needed to allow astronomers to measure the dark current during the day. In particular:

• Seal the two entrance doors to the Coude, as light leaks in underneath both.
• Continue imaging the room with the CCD camera, to identify the source of the light leaking in at roof level.
• Measure the impact of fluorescence from the corrector on nighttime observations.
• Replace the back cloth covering the spectrograph with non-translucent material.

8. Ricardo has examined the entire optical path through the spectrograph, but found no source of vignetting. It remains possible that the vignetting could arise in the CCD housing. An electronic effect has also not been completely ruled out.
    
9. After Ramon replaced the video card in the Arcon, the bias jumps have been seen less frequently during nighttime exposures. However, the jumps still occur immediately following dewar refills.
    
10. Marco has traced the lack of RTD with 1x1 binning as likely arising because of limited video memory in the Sun acquisition computer, which means we'll have to live with it.
     
11. Roger supposes that the need for solid N2 to keep the CCD's thermal current acceptably low arises from poor thermal contact in the CCD mounting. We have decided to pump the dewar constantly to maintain the low temperature rather than perform the risky undertaking of opening the dewar and identifying and correcting the problem. Originally, a small pump in a room next to the Coude was used. Before the October run, Oscar and the mountain staff rerouted pumping to a larger pump further from the Coude. Before this new system can be used, however, we need to install a valve to control the flow, and perhaps turn off pumping while filling the dewar. We found during the engineering run that LN2 poured into the flask ended up in the tube leading to the pump rather than inside the dewar; the tubing consequently froze and broke.
     
12. There is a software limit that prevents the fiber mount from tilting as far as it needs to in echelle mode. The limit was originally installed to prevent the fiber mount from colliding with the camera baffle. However, since the baffle has been modified, the limit is unnecessary. Rodolfo is working on this.