SOME NOTES ON OBSERVING WITH SuperAAPS

SuperAAPS is a DramaTCL GUI that links together several of the AAT’s systems which are used for Planet Search observing. It aims to combine them together into an integrated observing system that improves the efficiency, reliability and quality of AAPS data. The systems that SuperAAPS makes use of are

    1. The ODC (ie. Detector Controller),

    2. PTCS (Portable Telescope Control System),

    3. Guider,

    4. Exposure Meter,and

    5. UCLES (via its underlying task known as UMIT)

  1. However, they were not exactly designed with automated interaction in mind. Interoperability has improved in the last few years, and saaps now works reliably about 95% of the time. However, to make further improvements, failures need to be reported to me (with times and detailed descriptions) so I can track down the remaining instabilities in the system. If I don't find out what's going wrong, it will never get fixed!

Chris T. 13/02/2011 .




WHAT'S NEW

August 2011

  1. The Apogee CCD in the exposure meter has been replaced, and the meter has been recommissioned and is back in operation.

  2. CYCLOPS or UCLES - The SuperAAPS system can now run both regular AAPS observing with the UCLES slit, and CYCLOPS observing with the fibre bundle. saaps will try to work out whether you are using CYCLOPS of regular slit mode from the drama version you have started. If it gets this wrong, you can change the system being used from the UCLES_CYCLOPS menu in saaps.

  3. Observing Run Codes - Please use run code 'w' from now on for regular AAPS observing, run code 'c' for CYCLOPS observing, run code 's' for sub-giant planet search observing (Rob Wittenmyers' program) and any other letter (except for 'u') for other programs.

  4. Tracking Tool - The tracking tool 'Configure Runs to Track' option now sorts the runs by their run code (in reverse order) so 'w' runs appear the top and 'c' runs appear at the bottom, making it easier to select recent runs for each program

  5. Target Catalogues - The loading of your own target catalogues now works, as does the use of that catalogue with the tracking tool.

  6. Beware of Pattern Noise - The CCD Controllers developed a pattern noise problem during the Monster Run in 2009. This is intermittent, and causes excess noise to appear in each image - something that the Doppler analysis code picks up. So check for excess noise in the overscan region have been implemented in the usual post-processing - if the noise is within nominal limits, you'll get a green report on the post-processing window.If its excessive, the report is red, and a note appears in the observing log. When this happens you need to get technical help to (1) reset the controllers, or (2) have the cards from the controllers removed and jiggled - it seems to be some sort of annoying intermittent connection bug.

  7. A bug appeared in which (on 5 times on one night) the EXPOSED keyword in the FITS files (which is supposed to give us the exposure time) was set to 0. I have implemented a check for this in the post-processing script, and if the EXPOSED keyword is different from UTEND-UTSTART time by more than a few seconds, then UTEND-UTSTART is used to give the exposure time instead. In any case, the exposure meter mid-time is still correct.

  8. The EXPOSURE_METER continues to have a subtle bug we can't identify that causes the EXPOSURE METER to freeze during an exposure, and the saaps tool to then stop in its tracks after the exposure has read out. That exposure will be fine, but it will NOT have a mid-time correctly calculated in it. So you will need to shut everything down and start again. Changes to the underlying code seem to have reduced (but not eliminated) the incidence of this bug.

  9. The GUIDER has been much more robust in its automatic acquisition in the last year or so - nonetheless, it is not perfect, so glancing at it when a new target is slewed to is always worthwhile!

    At present it looks like this bug is due to a cumulative "memory leak" style problem, so that it just happens after a certain number of exposure meter exposures. It is being examined.

             



AT THE START OF A RUN

Pre-slit area hardware - at the beginning of an AAPS run, AAT technical staff will begining the set-up for all the 'standard' AAT equipment we use. However, you should check that the additional hardware we use in the pre-slit area of UCLES is all connected abd powered up. These are

  1. 1.Power up rotating pick-off mirror power supply and set to 1.6V. The rotating pick off is powered by a dark coloured variable power supply. It needs to be plugged in and set to deliver 1.6V to the rotating mirror, which sends 4% of the light exiting the slit o a CCD camera for monitoring by the exposure meter. (NB: Not needed while exposure meter is not operational).

  2. 2.Power up Exposure Meter CCD by connecting its plug-pack. The exposure meter uses an Apogee CCD camera to collect light picked off from the beam exiting the slit. It doesn't have a power switch, and so is powered up by connecting its supply. (NB: Not needed while exposure meter is not operational).

  3. 3.Connect up Iodine Cell temperature controller - make sure the temperature controller is plugged in. In general you'll want the cell to be cold when doing the set-up, so don't power it up just yet.


Start up UCLES : the SuperAAPS observing process

  1. 1. Connect to aatlxx (if not already connected) from the triple-headed Linux desktop (aatxdb) being used as an X-display and log in (username and passwords on the wall)

  2. 2.  Start drama. Logging into aatlxx will automatically start drama. You may need to use a 'special' version (if so instructed by AAT staff or so noted on the white-board). For example as at August 2011, we are using ucles_cyclops_latest, so enter this command to start the new drama

       drama -v ucles_cyclops_latest

    otherwise, just enter

      drama

    The following is not necessary any more

    (Then enter
       setenv EM_SAVE_IMAGE 0
       setenv EM_CAL_FILE "/instsoft/instusers/aatinst/saaps/spectral.cal"

    to save time by not having the exposure meter write out every image it takes, and to get the right spectral calibration file.)

  3. 3. The following is not necessary any more.

    Up until recently it has been necessary to use a 'private' version of the exposure meter task, that fixed a variety of flaws in the drama-standard version. To make this happen use the following commands (easiest to cut and paste from here)

      setenv EXPOSUREMETER_DIR /home/cgt/acmm/ExposureMeter-2-20
      setenv EXPOSUREMETER_LIB /home/cgt/acmm/ExposureMeter-2-20
      setenv EXPOSUREMETER_DEV /home/cgt/acmm/ExposureMeter-2-20

    Hopefully the need to do this will have been removed by the next run.

  4. 4.Start UCLES with an ucles1 or ucles2 command (depending which controller is in use - see the white board) and the following extra options which will ensure you can talk to the exposure meter and guider

       > cd ~
       > ucles2 guider em

    If the exposuremeter is not operational for any reason you need to remove the "em"

       > cd ~
       > ucles2 guider

    Similarly, if the GUIDER is not operational from the telescope console and you want to run without it (say during the day to do calibrations) use

       > cd ~
       > ucles2 em

  5. 5.After entering the ucles1/2 command, the System Loader window will appear and all the tasks needed to run UCLES for SuperAAPS will commence, with the observing GUI, and Exposure Meter window and a Skycat window appearing last. Note that the cd ~ is important. If you try to start ucles from anywhere except the home directory, it will crash with a really unhelpful error message.

    When the exposure meter first starts up after a power-up, it will need to cool-down to its operating temperature of -15C from room temperature. This can take up to half-an-hour.

    It might be useful to know what the options to the ucles1 or ucles2 commands are

  6. em - start the exposure meter task

  7. guider  - talk to the guider task on the new AAT TCS

  8. ptcsSim - start a PTCS simulator. This option means you can do pretend observing with a pretend telescope

  9. To do observing or tests with a pretend guider, you need to start a special 'guider simulator' before you enter the ucles1/ucles2 commands above, as follows

    > /instsoft/instusers/aatinst/guisim &

  10. 6.Start a GAIA window for examining images as they are taken (it offers additional functionality for measuring seeing and SNR over the Skycat window)  and a ptcs window so you can see where the telescope is.

        > gaia &
        > tel

  11. 7. Start SuperAAPS

        > cd saaps ; saaps &    

    This will start the SuperAAPS GUI. The first thing you'll see is a pop-up informing you that the "Guider will attempt to acquire objects from now on". Just Click OK, and the GUI will appear.

  12. 8.If this is the second or subsequent night of observing, use the Commands->Reconfigure menu pull down in the System Loader to advance the first run number past the last run number from last night.

  13. 9. Then 'Initialise' a new night to start a new logsheet in saaps with 'Commands->Initialise new Night'. You will get a pop-up saying "Are you sure you want to fnalise and archive this log?"  Make sure the name of the log-file is correct for the last nights observing and say yes. If you are unsure say no and check the files out in ~/saaps/final_logs/.

    After you say 'Finalise' the old log will be changed to be read-only, so it can't be accidentally deleted in the future.

    Make sure you then update the 'UT Date' entry correctly in the log header for the next night - you will not be able to write a log file until that entry is a legitimate one.




TCS Issues , Daytime Staff, and the End of the Night

It may be that AAT staff will want to do testing on the TCS during the day that will require killing and re-starting the TCS and/or guider. This will cause the UCLES ODC to stop taking data (eg. flats) because it needs to talk to a TCS task to fill in file headers etc. To stop this becoming a problem, I often shut down saaps, and start it back up in simulation mode, before I start the 40-60 wideflats we take at the end of the night.


  1. Shut down saaps (File->Close), and then shut down the ODC (File->Exit - making sure to  NEVER park the echelles - then when prompted cleanup 'Local Nodes' at the pop-up).

  2. Restart ODC as above, but with the following commands to start a fake TCS and a fake guider


    cd ~

    /instsoft/instusers/aatinst/guisim &

    ucles2 guider em ptcsSim

    cd saaps

    saaps &


You can then happily take flats regardless of what happens to the real TCS.  Of course, the following afternoon (usually after dinner), you will need to shutdown saaps & ODC  again, then restart ODC and saaps as per usual.


Start up UCLES : the SuperAAPS observing process

  1. 1. Shutdown SuperAAPS. Make sure there are no queue observations running, and use the "File->Exit" menu or Ctrl-Q in the SuperAAPS window.

  2. 2. Shutdown UCLES from the System Loader. You must shut-down UCLES from the "AAO CCD Loader" window using the "File->Exit" menu pull down. Shutting down from the observing GUI itself doesn't work.

    You will be asked "Do you really want to do this?" - say yes.

    You may then be asked "Do you want to park the echelles?" - say "NO". This is very important. If you accidentally say yes, you'll have to re-do the fine positioning of the AAPS set-up again. This pop-up is a recent addition, and we don't like it, and are trying to get rid of it. Until we do, "Just Say NO to Echelle Parking"

    Then you'll be asked do you want to do a clean up. Select "Local Nodes" and do the cleanup.

    Everything should then clean-up.

  3. 3.Emergency Shutdown. If the CCD Loader window is unresponsive, or the clean-up won't work, you can use a cleanup command in the same terminal window you started ucles from. But this is not the preferred mode of shutdown.

  4. 4.Warning - After you've exited SuperAAPS, you'll be sitting in ~/saaps. If you try to restart UCLES from this directory, you'll get an error message like "Can't talk to aatvme15" which arises because you're starting UCLES not in the home directory. So do the cd ~ and try again.


Start up UCLES : the "old" observing process

  1. For the record, here's how do to start up and shutdown when not using SuperAAPS.

  2. 1. Connect to aatlxx from the triple-headed Linux desktop (aatxdb) being used as an X-display and log in (username and passwords on the wall)

  3. 2. This will automatically log you into drama. If you have been told you need to need use a 'special' version (if so instructed by AAT staff or so noted on the white-board) then enter (eg.)

       > drama -v ucles_cyclops_latest

  4. 3. Start UCLES with an ucles1 or ucles 2 command (depending which controller is in use - see the white board)

       > ucles2

    after which the System Loader window will appear and all the tasks needed to run UCLES will commence, with the observing GUI and a Skycat window appearing last.

    NB: If this is the second or subsequent night of observing, use the Commands->Reconfigure menu pull down in the System Loader to advance the first run number past the last run number from last night.

  5. 4.Start a GAIA window for exmaing images as they are taken (it offers additional functionality for measuring seeing and SNR over the Skycat window)

       > gaia &

  6. 5.Start observing from the GUI

Shutdown UCLES : the "old" observing process

  1. 1. Shutdown UCLES from the System Loader. You must shut-down UCLES from the "AAO CCD Loader" window using the "File->Exit" menu pull down. Shutting down from the observing GUI itself doesn't work.

    You will be asked "Do you really want to do this?" - say yes.

    You may then be asked "Do you want to park the echelles?" - say "NO". This is very important. If you accidentally say yes, you'll have to re-do the fine positioning of the AAPS set-up again. This pop-up is a recent addition, and we don't like it, and are trying to get rid of it. Until we do, "Just Say NO to Echelle Parking"

    Then you'll be asked do you want to do a clean up. Select "Local Nodes" and do the cleanup.

    Everything should then clean-up.

  2. 2.Emergency Shutdown. If the CCD Loader window is unresponsive, or the clean-up won't work, you can use a cleanup command in the same terminal window you started ucles from. But this is not the preferred mode of shutdown.





 

Observing with SuperAAPS

 

OBSERVING WITH SuperAAPS

Overview


SuperAAPS has 10 major components in three major windows. Sitting in front of the triple headed monitor you might see something like this with the AAO CCD Loader on the far left, then the ODC GUI, then SuperAAPS, then the SuperAAPS exposure post-processing results, then GAIA.


In the main SuperAAPS GUI we have


  1. 1. A Catalogue of target stars and calibration observations.

  2. 2. A set of Queue Actions ("Start/Stop", "Pause/Continue" , "Abort") that control the queue

  3. 3. A set of current exposure actions ("Stop","Change","Abort") to stop, change the time of, or abort the current exposure.

  4. 4. A bunch of status information on what everything is doing, and a few more action buttons (Comment and Note buttons).

  5. 5.A queue of objects loaded from the Catalogue in 3 panels "Now Observing", "Next Target" and "Rest of Queue". Objects will be observed from "Next Target" and "Rest of Queue" in the sequence you provide.

  6. 6. A Messages window that tells you what the queue thinks its doing, and where errors and problems will be reported.

  7. 7. An Observing Log window with information on completed observations including the photon-weighted mid-time, seeing, SNR and comments you've added on individual runs.

  8. 8. A Notes window for notes you enter in things that happen during the night.

  9. 9. Menu options for less commonly used actions (Finalising the current log into a human friendly text output like that we used to keep in u85.logsheet3 format, Initialising a new log at the start of the night, Asking for images to have their seeing/SNR re-estimated, etc)

    In addition to those GUI components, there's also




  10. 10.An exposure meter progress window, showing real-time SNR, count-rates and current exposure meter temperature








  11. 11.A perl/PGPLOT pop-up window where SNR and seeing estimates are shown for the last exposure taken. The code that plots this also works out the mid-time, gets the photon-weighted mid-time from the file (if present) and flags if the two are significantly different. (below)






















General Principles.

In general, the idea is that you select a set of stars you want to observe, rearrange their order, adjust their exposure times, and hit "Start". Observing should begin, with the telescope slewing to the first target, setting up UCLES, beginning guiding and starting an exposure. When an exposure starts, it gets promoted from the "Queue" window to the "Now Observing" window, and the next planned observation gets promoted to the "Next" window.

When an exposure finishes, the "Now Observing" window will grey out, and readout will start. If the "Next" object is at a different position from the "Now Observing" one the telescope will start slewing to it. Once the readout (and slew and acquisition if needed) is finished, the "Next" object will get promoted to "Now Observing", and so on.

As each observation is written to disk, it will be post-processed by a perl script to produce a new display of seeing, SNR and some useful cuts through the image in the PGPLOT window, and the log window will be updated.

Setting time automatically (A)

The exposure meter not only determines the photon-weighted mid-point of exposures, but also estimates the accumulated SNR per wavelength pixel for our target stars in the iodine region. In principle this can be used to adjust the length of exposures to obtain a desired SNR.

The GUI also has a model for observing efficiency of AAPS stars as a function of V magnitude, seeing and S/N target. It uses this to estimate how long you'll need to observe a target (given the last measured seeing) to reach the S/N=100, 200 or 400 target set (for each object) in the catalogue. It then tries to break down that exposure into exposures that will be less than the maximum allowed exposure time (currently 450s, though that is a parameter you can change from the Observing pull-down menu).

If the exposure meter is being used to set times, it then works out how that split breaks up the S/N requirement, and sets a S/N target per exposure.

You can play with the specific target for an object by selecting it (once its in the queue) and hitting modify. By Entering the object in the pop-up as (A)uto and providing a S/N target (say 400 in 1 exposure, or equivalently 200 in 4 exposures), it will calculate a new split of the times to reach that target.

Setting time semi-automatically (S)

Sometimes that split is not quite right - you can override it and simply tell the GUI to do 3 S/N=200 exposures by using "Modify" with the (S)emi-auto tag. Semi-auto objects will still get their exposure times determined by the exposure meter ... but you get to determine the target S/N and how many exposures yourself.

Setting times manually (M)

In practice Auto and Semi-auto modes just don't work for some targets, and you'll want to hard-wire an exposure time and a number of exposures. For example on the Monster Runs, where every object gets observed for 20 minutes, its easiest just to manually put in exposures that add up to 20 minutes.

Use "Modify" and the (M)anual option for this - note that some objects are flagged in the input catalogue as only to be observed manually - usually these are faint objects like M-dwarfs where we just have to take what we can get in 1800s.

Manipulating the queue

Accessibility: The Catalogue gets the telescope's current position once a minute and updates the HA and airmass each target would have if observed now. It 'greys out' objects that will be too far over to observe (though you can still select them and try to queue them).

Status and SNR Target: Each object in the catalogue has a 'status' flag - 'K' for known planet, 'H!' for hot (things we have flagged in the past for observing every night), 'P!' for interesting (get once per run). Each object also has a SNR target '4' for OMPS stars (SNR=400), '3' for TMPS stars (SNR=300), '2' for everything else (SNR=200) except for faint object like M-dwarfs, where the number is a maximum exposure time to use.

Some stars are also flagged 'No' under the Q? column - these are objects where user interaction is required to ensure you observe the right one, so they won't be acquired automatically - instead a pop-up will appear and you'll have to dismiss it to indicate the telescope is guiding on the right object.

Queued Objects: Once an object is queued the actual exposure time proposed (or the manual ones you've over-ridden its estimates with) are shown in the 'Time' and 'N' columns. And entry in the 'M' column means you've indicated the times are manual ones, and the exposure meter will not truncate or extend exposures based on SNR - it will just observe for those times.

You'll also see columns for 'UT', 'HA', 'AM' and 'Slw'. 'UT' is the UT time when its estimated this target will get observed, based on things ahead of it in the queue, 'HA' & 'AM' are the HA and AM it will have then, and 'Slw' is the slew time from the previous object in the queue to this object. You can use this information to play with the 'travelling salesman' problem of minimising slew times for objects in the queue.

Once objects are in the queue, you can select them and promote or demote them in the queue (up and down arrows), remove them ("Remove"), remove everything ("Clear"). There is also a button "Make Next" which will promote the selected object into being next in the queue. (Noting that if no slew was initiated when the readout started at the end of the current exposure,  then making a new object 'Next" will result in the slew to the new target not starting until after the readout is finished. OR if the target you've just 'Made Next' is different form the one currently being slewed to, you'll have to wait for it to finish getting to the old target, before it will slew to the new one.)

Calibrations

The most commonly used calibrations for AAPS observing are listed at the top of the catalogue - Flats, Arcs, and Iodines. These can be queued, modified and rearranged like any other observation. There is also a "PAUSE" you can put in the queue that will make it stop and wait until you hit "Continue".

Changing your mind

So you've started observing an object, and you want to do something different or stop observing. What can you do? Well you actually have a lot of options. Common situations are

Cloud has come over and you want to stop observing, but wait and be ready to go again - well you can't close the shutter on an AAPS exposure and open it later ... your exposure mid-time would be useless. So what you should do is "Pause" the queue, then either "Stop" the current exposure (which will read out the data obtained to date) or "Abort" the current exposure (which is somewhat riskier). Then when the cloud parts, you can just "Continue" the queue.

The object I had next in the queue was incorrect, and when the shutter closed the telescope started slewing to the wrong object - this one is easy. Just "Pause" the queue, select the object you DO want to do next, hit "Make Next" which will put it into the "Next Object" entry. Then hit "Continue" - the GUI checks that the object in "Next" is the same as where the telescope is pointed before it starts an exposure. If its different, it will move the telescope to the correct target. Of course, you'll have slewed twice, but I can't help that ...

Maintaining the logbook & post-processing

At the end of every exposure, SuperAAPS runs a perl script (~/saaps/saaps_postproc.pl) which does some simple image processing to estimate seeing and SNR in a 'standard' box in the iodine region of the spectrum on the detector where our AAPS spectra should always lie. It also gets the photon-weighted mid-point and open-close mid-point, compares them and flags if they are significantly different.

That information (mid-time, seeing, SNR) is captured and used to create a logbook line in a format very similar to that we made by hand with the mtfits.pro IDL script. SuperAAPS maintains a 'working' set of this log information in an internal file format, as well as in human readable text format.

In addition, you can add comments to any one of these lines with the 'Add Comment' button, and add a timed 'Note' for the night with the "Add Note" button.

And finally, you can enter all the information we put at the top of our manual logs with the 'Observing->View/Update Log Header' menu.

To initialise these logs at the start of the night use 'Observing->Initialise New Night's Observing Log' from the menu bar. This will do three things

  1. 1. 'Finalise' the log data already there - ie. copy the text version of all the log information into a single file called uRR.YYMMDD.log in ~/saaps/final_logs. This is then the equivalent of the uRR.logsheetN files we currently make. You will be warned when you finalise a log. Usually this is OK, since all its doing is overwriting its previous idea of what the final log was with its updated idea. But its worth checking the date in the final log name to make sure it really is the right log for last night. Once finalised the log is changed to read-only so it can;t be accidentally modified in future.

  2. 2. Prompt you for new header details - at minimum these will usually need changing for the UT date.

  3. 3.Clear the log out and start a new one. When it does so, it creates a date-stamped copy of the working files, so it is always possible to 'go back' to an old state by exiting SuoerAAPS, copying the contents of ~/saaps/final_logs_working/YYMMDD into ~/saaps/working_logs, and restarting SuperAAPS. You will probably have to play with permissions when you do this as well - it is deliberately clunky. Overwriting a finalised log should only be done in extremus!

If during the night you create a crap file, and decide to use 'Reconfigure' to reset the run number and so create a new file with the same run number, SuperAAPS will not (by itself) notice this is a new file and so create a new log entry for it. You can force it to post-process the new object again with 'Observing->Reprocess Images into log'. Indeed you can use this to reprocess as many files as you like. One use for this is that if the 'standard' post-processing box is out by a few pixels, you can tweak it with 'Observing->Adjust post-processing box location', and then reprocess files. Note that reprocessing keeps your old comments. So if they are not relevant delete them by suing "Add Comment" to make the comment for that run empty.

Setting GUIDER Acquisition and EXPOSURE_METER modes

By default the exposure meter will try to set exposure times based on a SNR request. To change this default to using all manual exposure times (and to change back) there are pull down menu items in 'Observing' that will also determine whether the exposure meter is used (a) at all, (b) only to get mid-times and (c) in the default exposure time setting mode.

Similarly, the default behaviour of the GUIDER task is to try and have the GUIDER automatically acquire targets. If you are finding this is unreliable, then you can use 'Observing->Don't use guider to acquire' to turn this behaviour off. In this mode a pop-up will appear when you slew to each new target saying 'Dismiss me when the target is acquired'.

Related options in 'Observing' are those which reset flags telling SuperAAPS to stop polling the GUIDER and EXPOSURE_METER. (If communication with these stops for any reason, continued polling just makes things worse - the SuperAAPS stops polling them after they've failed once. Its then your job to get them connected again by restarting the tasks and/or resetting the AAO CCD Loader, before restarting polling by using the 'Observing->Try ???? again' menu items)

The Tracking Tool

When observing we often want to know what data we've collected for targets on previous nights in this run, on recent runs, or indeed within the last year or so. The Tracking Tool provides a way to get at that information. It reads all the log-sheets that have been made with the saaps system, and makes up a summary in a seperate window. The tracking tool come up automatically when you start saaps. It can be hidden and recreated with the "Tracking Tool->Show..." and "Tracking Tool->Hide..." menubar items.



Within the tracking tool the most important option is the "Configure->Select Runs to Track" menu item. By default, the tool will summarise the status of observations from the current run, putting a coloured box in each row and column for each night on the current run that summarises the S/N achieved on each night. However, you can tell it (as has been done in the example above) to track both this run and and previous runs as the 'current run', so that you can ensure (for example) that all objects get observed at least once over the course of two nearby runs.

The tool also shows you how many epochs (ie. nights) each target has been observed on over the last four months, and also in the last year.

The tool automatically updates if you read in a new target catalogue in the main window.



OBSERVING - SKYCAT and "THE OLD FASHIONED WAY"

Controlling the Telescope from SKYCAT

If observing with SuperAAPS is not working, it is straightforward to go back to the old fashioned mode of observing. Indeed you can do it even while SuperAAPS is running, controlling everything you need from the GUI.

However, what you DO need in this mode is a way of sending telescope co-ordinates. The easiest way to do that these days is with Skycat (NB this doesn't work in GAIA). If you've minimised Skycat then bring it up (and if you killed it then start it with > cd ~ ; skycat &).

Then use 'Data Servers->Local Catalogs->AAPS_Nov07_Target.txt' to bring up our target catalogue. To list the whole catalogue enter a very large number in "Max Radius:". Slewing to a target is then as simple as selecting it, the using 'Telescope->Slew to Selected Object', and responding 'Slew' to the resulting pop-up.

If skycat loses contact with PTCS, you can re-establish it with 'Telescope->Reset Server Connection'

You can use GAIA to do SNR and seeing calcs on our targets with the 'View->Pick Spectrum...' function, which requires you to draw a spectral box on the display (choose an order near X=705 in the centre of the I2 region), and then provides seeing estimates and SNR estimates for the relevant X and Y cuts.


LIMITATIONS


WHAT GOES WRONG (and how to fix it)

Shutdown/Resetting and Restarting/Resetting : The start-up and shutdown notes above contain most of the relevant useful information. Important tricks are

  1. 1. If there's a problem communicating with the GUIDER, EXPOSURE_METER or PTCS then the first thing to try is resetting UCLES from the AAO CCD Loader ("Commands->Reset"). This will establish communication with tasks that are still running, and restart any tasks that have died. So its usually faster than a full on kill and restart of UCLES. (But one common side-effect is that it breaks communication of the ODC GUI with the ODC, so you stop seeing the progress bars and times on the ODC GUI - though you will still see them on SuperAAPS)

  2. 2.After you've killed the UCLES tasks from the AAO CCD Loader, don't forget to cd ~ before re-starting UCLES. It will fail if you try to start anywhere except the home directory.

  3. 3.If you start in ~ and then see a 'Cannot communicate with AATVME15 error' then aatvme15 may need to be reset. If you don't know how to do that, ring for an afternoon technician.




MOST COMMON FAULTS

DITS Orphanned transaction messages : This is a problem that we have been seeing during the night (especially when the GUIDER is being used). It seems to be much less prevalent in March 2008, so not using the sequencer may have solved it.  The symptom is that there are message on the SuperAAPS console saying "DITS Orphaned transaction errors". Sometimes these have no effect at all. Sometimes they cause SuperAAPS to crash.

Just start SuperAAPS again with the saaps command - if it won't let you start a new observation, then its time to restart the entire UCLES/ODC system by exiting saaps, and shutting everything down from the System Loader

GUIDER, EXPOSURE_METER, UCLES or PTCS dies: If one of these tasks dies, SuperAAPS should be able to continue on ... though obviously with somewhat reduced functionality! (If the EXPOSURE_METER dies in the middle of an exposure, you won't be getting an exposure mid-time, and the exposure will run out to the maximum length. If the guider dies your object may drift off the slit.)

SuperAAPS makes an effort to continue on if these tasks die, and produces pop-ups, when it gets to the stages in the queue where it needs the dead task, that say things like "Please fix this and hit CONTINUE to go on or STOP to halt queue." You'll also see messages in the startup window like "Skipped polling of GUIDER/EXPOSURE_METER", which just indicate  SuperAAPS has noted the relevant task has died and has stopped trying to talk to it).

When this happens you need to fix the problem by (1) getting the dead task running again, and (2) getting it connected to Drama, with the 'Commands->Reset' function in the System Loader, then (3) hit CONTINUE on the pop-up to continue observing. Or use STOP to stop observing and fix everything, then restart observing,

May 2008 - changes in the way these pop-ups work should now make them more reliable. Prior to May 2008 they tended to lock-up the whole SuperAAPS GUI. They don't do that anymore.


PAST FAULTS (May 2008)

An examination of the error logs from March 2008 observing indicated three major bugs that occurred on multiple nights

  1. 1.Space in Drama's ring buffer for communication with the detector (RCT) task filled up (6 times in 5 nights). This repeats over the course of a minute, and eventually seems to result in the exposure meter stopping. In May 2008, I increased this buffer size to see if it helped.

  2. 2.When GUIDER acquisition failed, the signal that this had happened was not being correctly picked up (6 times in 5 nights), resulting in the queue having to be killed when it was about to start the next exposure. I believe I have (a) identified the possible cause of this bug and (b) made the pop-ups, that come up when the GUIDER is noticed to not be guiding properly before an exposure stars, work properly.

  3. 3.The EXPOSURE_METER spontaneously dies (3 times in 5 nights) which was causing SuperAAPS to have to be restarted. I have not yet identified the cause of the spontaneous EXPOSURE_METER deaths, but I have fixed the pop-up system, so that when EXPOSURE_METER does die, SuperAAPS will much more smoothly handle it.