Troubleshooting
The DECam observer system is quite stable. This section lists several of the most common problems that can appear during a night, and its possible solutions. If a problem appears find help from your Telescope Operator/ Observer Support Person/ TelOps Electronics-Data support person /Support Astronomer.
The best way of getting a diagnosis of a possible problem is to look at the Alarm History GUI. The different levels of alarms are:
- Warning - The system is still running, but something isn't quite right.
- Alert - Something's gone wrong enough that image taking is compromised. An example of this level of alarm is one of the software components going down.
- Critical - Hardware failure.
These alarms will produce a distinctive sound to make the observer/operator aware of a potential problem.
The following is a list of common errors and its solutions:
Unresponsive GUIs
No Guider Stars Found
Broken Interlock
DTS Stops
DECAL not ready
Shutter Error
Stars with weird shape
It is pretty common that some GUIs stop responding during the night. Sometimes they will turn out black. Before panicking, just reload the GUIs and they will come back to life. If you cannot see the refresh button on the top part of the windows, just press the F5 key to refresh.
In some occasions a message appears about the "Unresponsive Page" providing two options: "Kill page" or "Wait". Use "Kill page".
If the GUIs are still unresponsive, all GUIs can be restarted by pressing the DES icon (in the top bar of the top left screen) and choose close all GUIs (red logo). Using the same DES icon (dark logo), all GUIs can be restarted.
In the case that the users have no control of the SISPI observer control, a releaseMasterConsole command needs to be executed by the observer support or the telescope operator, after shutting down the GUIs and before restarting them using the method described above.
First, check the sky. You may not have guider stars because of clouds.
The default parameters of the guider worked very well in most conditions. However, sometimes it won't find suitable stars for guiding in any of the CCDs. If this is the case, a warning will appear (which you will see in the Alarm History GUI) and it will make a distinctive sound. The exposure will continue without guiding.
You may want to repeat the observation and help the Guider to find guider stars using any (or both) of these options:
1. Increase/Decrease the exposure time of the guider. The default exposure time is 600ms. For example, during twilight the sky is still bright and you may need to increase the exposure time. On the other hand, in very crowded fields you may want to reduce the exposure time. The exposure time may be changed directly in the Guider GUI by choosing a value from the drop-down menu. Another possibility for changing the exposure time is the following:
- Go to the Architect Console
- Find the System Control Box
- Replace the boxes that says: "Component Device Command: Press enter when complete" with "GCS GCS set guideexptime nnn" (where nnn is the exposure time in ms).\
2. Change the search and isolation range (in arcsec) in the Architect Console: find the GUIDER role and make the following change:
- GUIDER "set search_range 5"
- GUIDER "set isolation_range 7"
The defaults values are:
- GUIDER "set search_range 6"
- GUIDER "set isolation_range 12"
A broken interlock will produce and Alert and you will see a red light in the System Status. Observations cannot continue until the failure is solved. These types of failure can be very simple or quite complex. Simple solutions should be tried first. Complex ones will require an expert. This is the protocol to follow in case of a broken interlock:
1. Try Reset (Observer Console, System Control Tab). If successful, continue (takes only a few seconds). Many problems are solved just with this.
2. If you can identify the role with the problem, try to restart (it also takes only a few seconds but some roles depend on others which could cause problems in rare occasions). To identify the role with the problem go to the "Interlock Viewer" page, which is one of the GUIs up on the observer3 console. Click on the SISPI box at the bottom of the upper half of the page. This should fill the lower half of the page with SISPI Interlocks. Make a note of which of the Interlocks is highlighted in red.
2.1. OCS Status -- if the OCS Status is the *only* one that is red, go to the Observer Console GUI, click on the System Control tab, and hit the "Reset" button.
2.2. If the OCS Status + Another Process (eg., FCS_STATUS) shows up in red, go to the "Alarm History" page, which is the other tab in the same GUI. Identify what the specific alarm is (for example, check the time of the alarm to isolate the alarm that caused the interlock to break). In most cases, the Alarm History page will provide more details than the Interlock Viewer page.
Once you have identified the component that has failed, go to the "Architect Console" page, which is one of the GUIs up on the observer3 console, typically next to the interlock viewer page.
Find the component that has failed on the left side of the page, and highlight it. Click on the "Full Log" button on the middle, right part of the page. A terminal will pop up with a log history. Scroll all the way to the bottom of the terminal and verify that there is indeed a problem with the component.
Then, hit the "restart" button under "Role Control" in the top, right part of the Architect Console. Make sure that a red cross appears before the component that was just restarted--this means that the restart worked. Wait a few seconds for the red cross to become a green check mark.
Lastly, go to the Observer Console GUI, click on the System Control tab, and hit the "Configure" button.
3. Try configure (if you restarted a role or not). If successful, continue (takes about 20 seconds)
None of this 3 options above will affect the queue and the setup.
4. If above actions don't fix the problem restart SISPI. If successful, continue. Your telescope operator or support astronomer can help you with this task (takes about 4 min, requires reloading the setup information, configuring and load again the exposure scripts). In a few occasions SISPI has to be restarted 2 or 3 times before the system gets to work again.
5. If restarting SISPI does not work, and expert must be called. One way to do it is by locating people online in the skype account open in observer3.
6. If for some reason the Telescope Control System (TCS) has to be completely restarted then SISPI ought to be restarted afterwards.
Images are built but not delivered (last button never gets green). A Warning will be issued but observations continue. Continue observations and report it in the morning so the people at the NOAO archive recovers your data.
If the Alarm page shows a message about DECAL component not being ready, please tell the telescope operator. He will need to reset the DECAL (calibration) computer which is located in the MZ floor. After that, hit Reset.
A few shutter errors have appeared in the past. If the Alarm page indicates this type of error, tell the operator. He will perform a hardware reset which takes only a few seconds. Then, on the Observer Console, click on Reset.
In rare occasions the telescope mirror may loose the air support. This situation usually sets an alarm for the telescope operator. On the observer side, stars will show a triple-image pattern like the one shown in the figure.
Alert immediately the operator for restoring the air support and stop observations. This procedure involves slewing the telescope to the zenith position. If SISPI also issued an alarm (likely related with a MAMAC failure), you may need to press Reset before resuming observing.
Updated on April 18, 2024, 10:12 am