OBSERVING WITH SuperAAPS

Overview

SuperAAPS has 10 major components in three major windows. Sitting in front of the triple headed monitor you might see something like this with the AAO CCD Loader on the far left, then the ODC GUI, then SuperAAPS, then the SuperAAPS exposure post-processing results, then GAIA.

In the main SuperAAPS GUI we have

1. 1. A Catalogue of target stars and calibration observations.
   

2. 2. A set of Queue Actions ("Start/Stop", "Pause/Continue" , "Abort") that control the queue
   

3. 3. A set of current exposure actions ("Stop","Change","Abort") to stop, change the time of, or abort the current exposure.
   

4. 4. A bunch of status information on what everything is doing, and a few more action buttons (Comment and Note buttons).
   

5. 5.A queue of objects loaded from the Catalogue in 3 panels "Now Observing", "Next Target" and "Rest of Queue". Objects will be observed from "Next Target" and "Rest of Queue" in the sequence you provide.
   

6. 6. A Messages window that tells you what the queue thinks its doing, and where errors and problems will be reported.
   

7. 7. An Observing Log window with information on completed observations including the photon-weighted mid-time, seeing, SNR and comments you've added on individual runs.
   

8. 8. A Notes window for notes you enter in things that happen during the night.
   

9. 9. Menu options for less commonly used actions (Finalising the current log into a human friendly text output like that we used to keep in u85.logsheet3 format, Initialising a new log at the start of the night, Asking for images to have their seeing/SNR re-estimated, etc)
   
   

   

   In addition to those GUI components, there's also
   
   
   
   
   

10. 10.An exposure meter progress window, showing real-time SNR, count-rates and current exposure meter temperature
    
    
    
    
    
    
    
    
    

11. 

    11.A perl/PGPLOT pop-up window where SNR and seeing estimates are shown for the last exposure taken. The code that plots this also works out the mid-time, gets the photon-weighted mid-time from the file (if present) and flags if the two are significantly different. (below)
    

General Principles.

In general, the idea is that you select a set of stars you want to observe, rearrange their order, adjust their exposure times, and hit "Start". Observing should begin, with the telescope slewing to the first target, setting up UCLES, beginning guiding and starting an exposure. When an exposure starts, it gets promoted from the "Queue" window to the "Now Observing" window, and the next planned observation gets promoted to the "Next" window.

When an exposure finishes, the "Now Observing" window will grey out, and readout will start. If the "Next" object is at a different position from the "Now Observing" one the telescope will start slewing to it. Once the readout (and slew and acquisition if needed) is finished, the "Next" object will get promoted to "Now Observing", and so on.

As each observation is written to disk, it will be post-processed by a perl script to produce a new display of seeing, SNR and some useful cuts through the image in the PGPLOT window, and the log window will be updated.

Setting time automatically (A)

The exposure meter not only determines the photon-weighted mid-point of exposures, but also estimates the accumulated SNR per wavelength pixel for our target stars in the iodine region. In principle this can be used to adjust the length of exposures to obtain a desired SNR.

The GUI also has a model for observing efficiency of AAPS stars as a function of V magnitude, seeing and S/N target. It uses this to estimate how long you'll need to observe a target (given the last measured seeing) to reach the S/N=100, 200 or 400 target set (for each object) in the catalogue. It then tries to break down that exposure into exposures that will be less than the maximum allowed exposure time (currently 450s, though that is a parameter you can change from the Observing pull-down menu).

If the exposure meter is being used to set times, it then works out how that split breaks up the S/N requirement, and sets a S/N target per exposure.

You can play with the specific target for an object by selecting it (once its in the queue) and hitting modify. By Entering the object in the pop-up as (A)uto and providing a S/N target (say 400 in 1 exposure, or equivalently 200 in 4 exposures), it will calculate a new split of the times to reach that target.

Setting time semi-automatically (S)

Sometimes that split is not quite right - you can override it and simply tell the GUI to do 3 S/N=200 exposures by using "Modify" with the (S)emi-auto tag. Semi-auto objects will still get their exposure times determined by the exposure meter ... but you get to determine the target S/N and how many exposures yourself.

Setting times manually (M)

In practice Auto and Semi-auto modes just don't work for some targets, and you'll want to hard-wire an exposure time and a number of exposures. For example on the Monster Runs, where every object gets observed for 20 minutes, its easiest just to manually put in exposures that add up to 20 minutes.

Use "Modify" and the (M)anual option for this - note that some objects are flagged in the input catalogue as only to be observed manually - usually these are faint objects like M-dwarfs where we just have to take what we can get in 1800s.

Manipulating the queue

Accessibility: The Catalogue gets the telescope's current position once a minute and updates the HA and airmass each target would have if observed now. It 'greys out' objects that will be too far over to observe (though you can still select them and try to queue them).

Status and SNR Target: Each object in the catalogue has a 'status' flag - 'K' for known planet, 'H!' for hot (things we have flagged in the past for observing every night), 'P!' for interesting (get once per run). Each object also has a SNR target '4' for OMPS stars (SNR=400), '3' for TMPS stars (SNR=300), '2' for everything else (SNR=200) except for faint object like M-dwarfs, where the number is a maximum exposure time to use.

Some stars are also flagged 'No' under the Q? column - these are objects where user interaction is required to ensure you observe the right one, so they won't be acquired automatically - instead a pop-up will appear and you'll have to dismiss it to indicate the telescope is guiding on the right object.

Queued Objects: Once an object is queued the actual exposure time proposed (or the manual ones you've over-ridden its estimates with) are shown in the 'Time' and 'N' columns. And entry in the 'M' column means you've indicated the times are manual ones, and the exposure meter will not truncate or extend exposures based on SNR - it will just observe for those times.

You'll also see columns for 'UT', 'HA', 'AM' and 'Slw'. 'UT' is the UT time when its estimated this target will get observed, based on things ahead of it in the queue, 'HA' & 'AM' are the HA and AM it will have then, and 'Slw' is the slew time from the previous object in the queue to this object. You can use this information to play with the 'travelling salesman' problem of minimising slew times for objects in the queue.

Once objects are in the queue, you can select them and promote or demote them in the queue (up and down arrows), remove them ("Remove"), remove everything ("Clear"). There is also a button "Make Next" which will promote the selected object into being next in the queue. (Noting that if no slew was initiated when the readout started at the end of the current exposure,  then making a new object 'Next" will result in the slew to the new target not starting until after the readout is finished. OR if the target you've just 'Made Next' is different form the one currently being slewed to, you'll have to wait for it to finish getting to the old target, before it will slew to the new one.)

Calibrations

The most commonly used calibrations for AAPS observing are listed at the top of the catalogue - Flats, Arcs, and Iodines. These can be queued, modified and rearranged like any other observation. There is also a "PAUSE" you can put in the queue that will make it stop and wait until you hit "Continue".

Changing your mind

So you've started observing an object, and you want to do something different or stop observing. What can you do? Well you actually have a lot of options. Common situations are

Cloud has come over and you want to stop observing, but wait and be ready to go again - well you can't close the shutter on an AAPS exposure and open it later ... your exposure mid-time would be useless. So what you should do is "Pause" the queue, then either "Stop" the current exposure (which will read out the data obtained to date) or "Abort" the current exposure (which is somewhat riskier). Then when the cloud parts, you can just "Continue" the queue.

The object I had next in the queue was incorrect, and when the shutter closed the telescope started slewing to the wrong object - this one is easy. Just "Pause" the queue, select the object you DO want to do next, hit "Make Next" which will put it into the "Next Object" entry. Then hit "Continue" - the GUI checks that the object in "Next" is the same as where the telescope is pointed before it starts an exposure. If its different, it will move the telescope to the correct target. Of course, you'll have slewed twice, but I can't help that ...

Maintaining the logbook & post-processing

At the end of every exposure, SuperAAPS runs a perl script (~/saaps/saaps_postproc.pl) which does some simple image processing to estimate seeing and SNR in a 'standard' box in the iodine region of the spectrum on the detector where our AAPS spectra should always lie. It also gets the photon-weighted mid-point and open-close mid-point, compares them and flags if they are significantly different.

That information (mid-time, seeing, SNR) is captured and used to create a logbook line in a format very similar to that we made by hand with the mtfits.pro IDL script. SuperAAPS maintains a 'working' set of this log information in an internal file format, as well as in human readable text format.

In addition, you can add comments to any one of these lines with the 'Add Comment' button, and add a timed 'Note' for the night with the "Add Note" button.

And finally, you can enter all the information we put at the top of our manual logs with the 'Observing->View/Update Log Header' menu.

To initialise these logs at the start of the night use 'Observing->Initialise New Night's Observing Log' from the menu bar. This will do three things

1. 1. 'Finalise' the log data already there - ie. copy the text version of all the log information into a single file called uRR.YYMMDD.log in ~/saaps/final_logs. This is then the equivalent of the uRR.logsheetN files we currently make. You will be warned when you finalise a log. Usually this is OK, since all its doing is overwriting its previous idea of what the final log was with its updated idea. But its worth checking the date in the final log name to make sure it really is the right log for last night. Once finalised the log is changed to read-only so it can;t be accidentally modified in future.
   

2. 2. Prompt you for new header details - at minimum these will usually need changing for the UT date.
   

3. 3.Clear the log out and start a new one. When it does so, it creates a date-stamped copy of the working files, so it is always possible to 'go back' to an old state by exiting SuoerAAPS, copying the contents of ~/saaps/final_logs_working/YYMMDD into ~/saaps/working_logs, and restarting SuperAAPS. You will probably have to play with permissions when you do this as well - it is deliberately clunky. Overwriting a finalised log should only be done in extremus!
   

If during the night you create a crap file, and decide to use 'Reconfigure' to reset the run number and so create a new file with the same run number, SuperAAPS will not (by itself) notice this is a new file and so create a new log entry for it. You can force it to post-process the new object again with 'Observing->Reprocess Images into log'. Indeed you can use this to reprocess as many files as you like. One use for this is that if the 'standard' post-processing box is out by a few pixels, you can tweak it with 'Observing->Adjust post-processing box location', and then reprocess files. Note that reprocessing keeps your old comments. So if they are not relevant delete them by suing "Add Comment" to make the comment for that run empty.

Setting GUIDER Acquisition and EXPOSURE_METER modes

By default the exposure meter will try to set exposure times based on a SNR request. To change this default to using all manual exposure times (and to change back) there are pull down menu items in 'Observing' that will also determine whether the exposure meter is used (a) at all, (b) only to get mid-times and (c) in the default exposure time setting mode.

Similarly, the default behaviour of the GUIDER task is to try and have the GUIDER automatically acquire targets. If you are finding this is unreliable, then you can use 'Observing->Don't use guider to acquire' to turn this behaviour off. In this mode a pop-up will appear when you slew to each new target saying 'Dismiss me when the target is acquired'.

Related options in 'Observing' are those which reset flags telling SuperAAPS to stop polling the GUIDER and EXPOSURE_METER. (If communication with these stops for any reason, continued polling just makes things worse - the SuperAAPS stops polling them after they've failed once. Its then your job to get them connected again by restarting the tasks and/or resetting the AAO CCD Loader, before restarting polling by using the 'Observing->Try ???? again' menu items)

The Tracking Tool

When observing we often want to know what data we've collected for targets on previous nights in this run, on recent runs, or indeed within the last year or so. The Tracking Tool provides a way to get at that information. It reads all the log-sheets that have been made with the saaps system, and makes up a summary in a seperate window. The tracking tool come up automatically when you start saaps. It can be hidden and recreated with the "Tracking Tool->Show..." and "Tracking Tool->Hide..." menubar items.

Within the tracking tool the most important option is the "Configure->Select Runs to Track" menu item. By default, the tool will summarise the status of observations from the current run, putting a coloured box in each row and column for each night on the current run that summarises the S/N achieved on each night. However, you can tell it (as has been done in the example above) to track both this run and and previous runs as the 'current run', so that you can ensure (for example) that all objects get observed at least once over the course of two nearby runs.

The tool also shows you how many epochs (ie. nights) each target has been observed on over the last four months, and also in the last year.

The tool automatically updates if you read in a new target catalogue in the main window.

OBSERVING - SKYCAT and "THE OLD FASHIONED WAY"

Controlling the Telescope from SKYCAT

If observing with SuperAAPS is not working, it is straightforward to go back to the old fashioned mode of observing. Indeed you can do it even while SuperAAPS is running, controlling everything you need from the GUI.

However, what you DO need in this mode is a way of sending telescope co-ordinates. The easiest way to do that these days is with Skycat (NB this doesn't work in GAIA). If you've minimised Skycat then bring it up (and if you killed it then start it with > cd ~ ; skycat &).

Then use 'Data Servers->Local Catalogs->AAPS_Nov07_Target.txt' to bring up our target catalogue. To list the whole catalogue enter a very large number in "Max Radius:". Slewing to a target is then as simple as selecting it, the using 'Telescope->Slew to Selected Object', and responding 'Slew' to the resulting pop-up.

If skycat loses contact with PTCS, you can re-establish it with 'Telescope->Reset Server Connection'

You can use GAIA to do SNR and seeing calcs on our targets with the 'View->Pick Spectrum...' function, which requires you to draw a spectral box on the display (choose an order near X=705 in the centre of the I2 region), and then provides seeing estimates and SNR estimates for the relevant X and Y cuts.

LIMITATIONS

WHAT GOES WRONG (and how to fix it)

Shutdown/Resetting and Restarting/Resetting : The start-up and shutdown notes above contain most of the relevant useful information. Important tricks are

1. 1. If there's a problem communicating with the GUIDER, EXPOSURE_METER or PTCS then the first thing to try is resetting UCLES from the AAO CCD Loader ("Commands->Reset"). This will establish communication with tasks that are still running, and restart any tasks that have died. So its usually faster than a full on kill and restart of UCLES. (But one common side-effect is that it breaks communication of the ODC GUI with the ODC, so you stop seeing the progress bars and times on the ODC GUI - though you will still see them on SuperAAPS)
   

2. 2.After you've killed the UCLES tasks from the AAO CCD Loader, don't forget to cd ~ before re-starting UCLES. It will fail if you try to start anywhere except the home directory.
   

3. 3.If you start in ~ and then see a 'Cannot communicate with AATVME15 error' then aatvme15 may need to be reset. If you don't know how to do that, ring for an afternoon technician.
   

MOST COMMON FAULTS

DITS Orphanned transaction messages : This is a problem that we have been seeing during the night (especially when the GUIDER is being used). It seems to be much less prevalent in March 2008, so not using the sequencer may have solved it.  The symptom is that there are message on the SuperAAPS console saying "DITS Orphaned transaction errors". Sometimes these have no effect at all. Sometimes they cause SuperAAPS to crash.

Just start SuperAAPS again with the saaps command - if it won't let you start a new observation, then its time to restart the entire UCLES/ODC system by exiting saaps, and shutting everything down from the System Loader

GUIDER, EXPOSURE_METER, UCLES or PTCS dies: If one of these tasks dies, SuperAAPS should be able to continue on ... though obviously with somewhat reduced functionality! (If the EXPOSURE_METER dies in the middle of an exposure, you won't be getting an exposure mid-time, and the exposure will run out to the maximum length. If the guider dies your object may drift off the slit.)

SuperAAPS makes an effort to continue on if these tasks die, and produces pop-ups, when it gets to the stages in the queue where it needs the dead task, that say things like "Please fix this and hit CONTINUE to go on or STOP to halt queue." You'll also see messages in the startup window like "Skipped polling of GUIDER/EXPOSURE_METER", which just indicate  SuperAAPS has noted the relevant task has died and has stopped trying to talk to it).

When this happens you need to fix the problem by (1) getting the dead task running again, and (2) getting it connected to Drama, with the 'Commands->Reset' function in the System Loader, then (3) hit CONTINUE on the pop-up to continue observing. Or use STOP to stop observing and fix everything, then restart observing,

May 2008 - changes in the way these pop-ups work should now make them more reliable. Prior to May 2008 they tended to lock-up the whole SuperAAPS GUI. They don't do that anymore.

PAST FAULTS (May 2008)

An examination of the error logs from March 2008 observing indicated three major bugs that occurred on multiple nights

1. 1.Space in Drama's ring buffer for communication with the detector (RCT) task filled up (6 times in 5 nights). This repeats over the course of a minute, and eventually seems to result in the exposure meter stopping. In May 2008, I increased this buffer size to see if it helped.
   

2. 2.When GUIDER acquisition failed, the signal that this had happened was not being correctly picked up (6 times in 5 nights), resulting in the queue having to be killed when it was about to start the next exposure. I believe I have (a) identified the possible cause of this bug and (b) made the pop-ups, that come up when the GUIDER is noticed to not be guiding properly before an exposure stars, work properly.
   

3. 3.The EXPOSURE_METER spontaneously dies (3 times in 5 nights) which was causing SuperAAPS to have to be restarted. I have not yet identified the cause of the spontaneous EXPOSURE_METER deaths, but I have fixed the pop-up system, so that when EXPOSURE_METER does die, SuperAAPS will much more smoothly handle it.