Instructions for reducing APT data
		----------------------------------

Last updated 30 May 2006


Very quick guide
----------------

To reduce a run's worth of data

* run aptproc.pl in each night's directory until sky flats are processed

* select good sky flats and combine to form master flat

* reducce one good night's data using new master flat

* cread master image and catalogue for each new field from good night's data

* reduce remaining nights with aptproc.pl

* for each field, generate lightcurves with res2lc.pl



Details
-------

I have started every line where user input is requred with '**'. Stuff in 
between is just additional information.


** copy data from CD/DVD to /data2/jessiec/apt on Wink

    A separate directory is used for each night's data, and each directory is 
    named after the date at the start of the night.
    eg. /data2/jessiec/apt/030303/
	/data2/jessiec/apt/030304/
	/data2/jessiec/apt/030305/
	etc...


** > source /home/jessiec/mike/apt_setup

   This will define various useful aliases for programs, scripts, etc...  You 
may find it useful to put this line into your .cshrc file, so that these 
definitions are automatically loaded every time you open a new shell.


** open a new xterm and cd to a raw data directory, eg 030324/ (this is so that
several directories can be processed at once - unfortunately the pipeline
doesn't seem to run as a background job).


** > aptproc 

   This will will run for an hour or two (from raw images to photometry
results) and do the following:

    - Unzip files, if there are any .gz or .bz2 files in the directory.

    - Set permission to 664 for all .fits files

    - Update image headers, adding the keywords TRIMSEC and BIASSEC required
for later processing, also changing FILTERS keyword to FILTER

    - Sort the images into the subdirectories sky/ and star/ based on the
IMAGETYP header keyword. Also write a list of images for each field in a file
star/l<fieldname>

    - If there are any sky flats, process them (subtract bias, trim overscan
region and correct for non-linearity). Form a nightly master sky flat by 
combining the processed sky flats (creating sky/msky.fits).

    - At this point, unless a master sky flat has been specified with the '-f'
option (see below), the process will stop and ask for a flat to be specified,
offering the msky.fits from the current night as default (though it may not
exist). However, in general this is not what we want to use. Instead, once we
have reached this point for all the nights within a run (or even a couple of
consecutive runs), we want to combine all the processed twilight images into 
a single master flat, and use that same flat to flatfield all the images from 
that run (or that season).

      A wonderfully friendly Perl script called mflat.pl will make this much 
easier. Once you have enough processed twilights for a season (say beginning
with nights 0501??), go to /data1/mgh/master/ and do this:

** > mflat.pl 0501skyI.fits /data2/jessiec/apt/0501*/sky

    where 'I' is the filter used. Make sure you only include flats that were
taken with that filter. This will generate a master flat (0501skyI.fits), and
a text file (0501skyI.l) listing the individual processed twilight flats that
were combined to make the master.

** Inspect the new master flat (eg. in DS9) to check for any 'strange'
features, such as stars or images of the APT entrance pupil due to
scattered light (it happens!), or in fact any small-scale
features. Actually, scattered light problems seem to be unavoidable,
but make sure they are not present in the processed nightly flats (see
below). Smooth gradients across the field are not a problem as they
are handled by later processing. There are also a few 'spots', an
obvious vertical line and some vignetting in the corners that will
always be there (see other master flats in /data1/mgh/master for
examples).

**  You should also check the individual, nightly combined flats (msky.fits), 
after flatfielding them with the newly created master flat. To help you do 
this, mflat.pl automatically creates a tmp/ subdirectory in the location where
you ran it (usually /data1/mgh/master/), links all the msky.fits files in
there (with names like sky050106.fits), then processes them with the new
master flat.

The resulting psky050106.fits (etc...) file should be 'flat', with
the exception of shallow, smooth gradients. In particular, the spots,
vertical line and vignetting should be absent. Stars should also be absent!
Some flatfield features seem to be quite difficult to actually fully remove, 
but the amplitude of any residual features in these images should be less 
than 0.1% of the overall flux level. An easy way to check is to go into IRAF 
and use commands like 

  > imarith 'a/median(a)' nsky06 psky050106

... to divide each image by its median value. If any of these images
look bad, the corresponding night's sky/ subdirectory should be 
renamed to something like 'badsky' (or the pim*.fits files deleted)
and then mflat.pl re-run without them.


**  go back to each aptproc process and specify the master flat to use.  Again,
make sure you reduce images with the correct master flat (right filter and 
from twilights taken within a few months of the data you're reducing. 
Most of our images are now in I band, but there may be some other data too.

    aptproc then goes on to...

    - Decide which fields need to be processed. By default, all fields will be
processed, but fields can be selected or excluded (see below for command line
options).

    - Trim, bias subtract, linearise and flatfield star images.

    - Re-run AptAstrom on every processed image, updating the headers with the
correct world coordinate system information (this is needed as trimming the
overscan region off the images changes the image coordinates, upon which the
old WCS transformation was based). Processed images with smag<-1 
are deleted, as they don't tend to give useful photometry. The 
corresponding raw images are not removed. The smag limit can be adjusted by 
changing the values in the %smaglim hash in aptproc.pl.

    - (The original version fo the pipeline used to run and IRAF script to add
ST, UT, JD and AIRMASS to image headers. The AptAstrom program now takes care 
of this, so I have turned this off. However it could be turned back on for 
older data if need be.)

    - Move processed images to separate directories for each field, renaming
them from .fits to .fit (Mike's photometry programs use .fit for images and
.fits for objects catalogues in FITS table format). Any images that were not 
copied over because the AptAstrom fit failed or the 'smag' was less than -1 
(i.e. cloudy and probably useless) are listed in a file called 'bad'.

    - Generate object catalogue for each image. For each .fit image, this will
create a _cat.fits table and also a _cat.ell ascii listing (which can be used
to check what objects are in the catalogue using the regions feature in ds9).

    - If a master catalogue for the current field has not been chosen yet (by
default this is in the directory /data1/mgh/master), select the lowest airmass
image from the present night and copy it, along with its image catalogue to the
master directory. (This is only used when we first want to quickly reduce data
on a new field. In practice we then generate a master image "manually", by 
stacking a dozen or two good images from a single night and then generate the 
master catalogue from this stacked image.)

    - Use the individual object catalogues to find six-parameter linear 
coordinate transformations from each image to the master image. These are 
stored in a .trans file for each image.

    - Transform the master catalogue objects into the frame of each image and
measure these stars. The photometry apertures used in each image are written to
a _list.ell file for easy plotting. The results of the photometry itself are
output in binary fits tables with names ending _list.fits

    - For compatibility with earlier versions of the pipeline, the data is 
then read out of the _list.fits files and converted to a single binary 
(though readable as ascii) file called 'resultsn'. This file contains 5760-byte
records. The first of these lists the JD (-2450000) from each image (the first
four numbers are dummies). Each subsequent record contains data for a star form
the master list. The first four numbers are the x & y coordinate, the median
mag of the star, and something else i'm not sure about. The numbers after that
are magnitude-airmass pairs corresponding to the JDs in the first record. The
values in this file are instrumental magnitudes (2.5*log(flux in ADU)).

    - Perform a frame-to-frame magnitude zero-point correction. This is done by
the program 'cllc', which allows the zero-point to vary as a linear function of
x, y, xy and y**2 (x and y being the pixel coordinates). The final output is a
'resultsn.xyy' file, in the same format as the original resultsn file. 
      This program also uses Tycho stars (in a master.ty file linked in from
the /data1/mgh/master directory) to calibrate the magnitude zero-point. In I 
band, this involves a guess at each Tycho star's I magnitude based on its B and
V mags (using tables for main-sequence stars from AQ5/Cox2000). 
      The output magnitudes are still instrumental, but adjusted so that a 
fixed zero-point of 22.5 will turn them into real mags (this is done by 
res2lc, below).

    - After the last iteration, cllc checks the magnitude rms for bright stars 
in each image, and if it is ever greater than 15mmag, outputs the sequence 
number of that image to the file fort.33. aptproc.pl renames this to 
resultsn.ex, and re-runs cllc, which now excludes the noisy images from the 
calculations (and from resultsn.xyy).

    - Write a detailed log file, including the output from each step, to a
'LOG' file. (Previous LOG files are saved as LOG001, LOG002, etc...)


    aptproc has now finished running, but the fun's not over yet!. From here
on, data is treated one field at a time. Individual nights of data can be 
quickly inspected using:

**  > lightcurves  resultsn.xyy

    This program will first calculate the overall magnitude offset of each
image from the median, as well as the RMS error on the magnitues of bright
stars in the image. These values are printed in columns 2 and 3
respectively. The offsets should be within a few millimag of 0, since they have
already been corrected for, and the RMS should be around 3-5mmag, but up to
9-10mmag is still ok.

    After pressing Enter a few times, it will display a plot of RMS vs. mag, 
along with a theoretical limit (Poisson noise - for this to match the data,
the values of the sky flux and magnitude zero-point need to be recorded in a
file named 'lcpar' - see an example in /data2/jessiec/apt). Stars brighter 
than about I=7.7 will have higher RMS values since they are saturated (in 
60sec images).

    It will then display the image offsets and RMS values for each image (the 
same information it printed to the terminal), followed by star lightcurves in 
order of decreasing magnitude, starting with the saturated ones. Images for 
which the RMS was greater than 9mmag are have their corresponding datapoints 
highlighted by red circles in all plots. The only way to avoid looking through
all lightcurves is to do a Ctrl-C.

    If there are still images you think should be removed from the lightcurves,
add their sequence numbers (i.e. 1 for the first image, etc..) to the
'resultsn.ex' file. You can also specify negative numbers to count from the 
end (eg. -1 is the last image). To help you with this, the lightcurves program 
outputs such a list, containing all the images for which RMS>9mmag, to a file 
called 'fort.33'. Remember, however, that the sequence numbers in resultsn.ex 
should be relative to the complete set if images in that directory, while the
image numbers output by lightcurves are only the ones it found in resultsn.xyy.
So if there was already a resultsn.ex there, you'll have to do a bit of mental
arithmetic...

    If all of the data in this directory is useless, rename the directory from
'fld<fieldname>' to 'bad<fieldname>', and/or delete its contents.

   Once all the data for a particular field has been reduced to 'resultsn.xyy'
files, we're ready to create some lightcurve files!

** > cd /data2/jessiec/apt/
** > mkdir <fieldname>           (Jan06_1 for example)
** > res2lc.pl  Jan06_1/  14.0  */fldJan06_1/resultsn.xyy

   This will read all the 'resultsn.xyy' files and write a lightcurve file
(Jan06_1/nnnnn.lc) for each star brighter than 14.0 (set this to -1 to include
all stars) and sufficiently far from the CCD edges and corners (parameters for
this are hard-coded into res2lc). 'nnnnn' is the star's id, which is its
sequence number in the master catalogue for the field. The catalogue is sorted
in order of increasing y coordinate (approximately). A log is written to 
Jan06_1/res2lc.log so that we can easily check later on where those 
lightcurves came from.

   In practice we generate several sets of lightcurves for each field, adding 
more nights of data as they are taken, so within Jan06_1/ there will be 
subdirectories like 'upto060126_14', 'upto060201_13', etc... (the number at the
end is the magnitude limit).


And you're ready to do some eyeballing... or whatever you wish!


-------------------------------------------------------------------------------


Command line options for aptproc
--------------------------------

+all		do everything (this is the default)
-all		do nothing, except tasks individually turned on by subsequent
                '+' switches

+pre		do preprocessing (update headers + sort images to sky/ & star/)
-pre		skip preprocessing

+sky		process and combine sky images to form master sky flat
-sky		don't process sky flats

+star		process star images and move to field dir
-star		don't process star images

+cat		generate object catalogues for all images in each field dir
-cat		don't generate object catalogues

+phot		do photometry on images in each field dir using master list
		(and generate resultsn and resultsn.xyy files)
-phot		don't do photometry

+recal		Repeat calibration after excluding bad images (done by default
		with +all or +phot)
-recal		Don't recalibrate

-lin		skip linearity correction when processing images

-f flat.fits	use flat.fits to flatfield star images (instead of master flat
		from the current dataset)

-d f1,f2,...	process only the fields in the list (comma-separated)

-s f1,f2,...	skip fields in the list (comma-separated)

-m master	If master is a *_cat.fits file, use it as the master catalogue
		for photometry. If it is a directory, look for master catalogue
		files in that directory, with the name <fieldname>_cat.fits

-h field	Here: only do catalogue generation and photometry on images in
		the current directory, which are of the field specified. If
		aptproc is run from a directory ending in /fld<fieldname>/
		this option is automatically enabled and the field name is
		read from the directory name.

-r rcore	Set the photometry aperture radius in pixels. The default is 3.
		(The software measures stars with apertures of radius rcore, 
		and also factors of 1/sqrt(2), sqrt(2), 2, and 2sqrt(2) times
		rcore. These mags are all stored in the _list.fits files)

-------------------------------------------------------------------------------


Quick guide to files and directories used in the reduction process
------------------------------------------------------------------

* In each night's directory (/data2/jessiec/apt/YYMMDD/), you'll find:

aptYYMMDD.log   observer's log (brief comment on night, sometimes absent)
obsYYMMDD.log	output from observing script
obs<fieldname>	copy of observing script used
LOG             log file from most recent run of aptproc.pl
LOG???          previous log files
sky/		containing raw (im*), processed (pim*) and combined (msky.fits)
		twilight images
star/           containing raw star images (and lists of image names belonging
		to each field observed)
fldF1/		reduced data for field F1


* the each field directory (eg. fldF1/) will have:

LOG*		log files from aptproc run just on this field/night
bad             list of non-photometric images that were not processed
fort.*          debugging output from fortran programs (not really important)
images          list of processed images
master_cat.fits copy of master catalogue for field
master.ty	copy of list of Tycho stars in field 
pim*.fit        processed image
pim*.trans      coefficients of coordinate transformation to master frame
pim*_cat.ell    for plotting catalogue objects (DS9 region file)
pim*_cat.fits   fits catalogue of objects found in this image
pim*_list.ell   for plotting photometry apertures used to measure image
pim*_list.fits	bits table output from photometry program (contains mags 
		measured with several aperture sizes, and more...)
resultsn        base-aperture-size (3pix) photometry compiled in single file
resultsn.ex     sequence numbers of datapoints to exclude from lightcurves
resultsn.xyy    results file after zero-point correction
tttt            temporary file (not important)

(some older data may also have:
results_0.ps	plot of rms vs. mag and residual extinction coeffs vs. mag
results_1.ps	as above, after residual extinction has been corrected for
)


* Also used in the reduction are the following files in the /data1/mgh/master/
directory:

YYMMskyI.fits 	master sky img in I filter (combined from twilights obtained
		on several nights within the run, listed in YYMMskyI.l)

F1.fit		image of field F1 from which master catalogue was obtained
F1.l		list of images stacked to create F1.fit
F1.ty         	list of Tycho stars matched to stars in the master catalogue
                (columns: star id, Tycho id, x, y, V mag, B-V colour, 
                match distance in pixels)
F1_cat.ell	DS9 region file for master catalogue
F1_cat.fits	master catalogue for field F1
F1_cat.rd	same list as _cat.ell, but with RA and Dec instead of x & y
F1_mat.ell	DS9 region file of matched objects (Tycho)



-------------------------------------------------------------------------------

Programs and scripts used in the data reduction process
-------------------------------------------------------

Note: most of these are aliased (in /home/jessiec/mike/apt_setup) so that you 
can run them from any directory.


* scripts & programs in /home/jessiec/mike/apt/

aptproc.pl		The main pipeline script
translate_keywords.pl	Change FILTERS keyword in FITS header to FILTER
sortdata.pl		Sort images into sky/ and star/ subdirs
combflat.pl 		Combine twilight flats, wrapper for fitsio_combine_flat
combstar.pl 		Combine star images, wrapper for fitsio_combine_star
tycho.pl 		Find Tycho stars in the field and match them up with
			Mike's object catalogue
lot_merge.pl 		Calculate coordinate transformations to master frame
dolistr			Shell script wrapper used to run imcore_list on all
			processed images in the current directory (this used
			to be done by lot_list.pl). Takes rcore as an input.
cllc			Perform relative photometric calibration on a results
			file to remove atmospheric extinction variations
lightcurves		Display RMS vs. mag and individual lightcurves from
			results file
res2lc			Generate individual .lc files from results file 
res2lc.pl		Wrapper for res2lc, saves log of output
			

* Mike's Perl scripts in /home/jessiec/mike/processing/

hedit.pl		Edit FITS headers
imcore.pl 		Generate object catalogues for images (wrapper for 
			imcore_conf)
lot_classify.pl		Classify detected objects


* Mike's Fortran programs in /home/jessiec/mike/soft/fitsio/

fitsio_combine		Stack images (newer version, but not currently used)
fitsio_combine_flat	Combine flatfield images, rejecting those with less 
			than 10000 counts on average (based on an earlier 
			version by Mike)
fitsio_combine_star	Combine star images, rejecting those with more than 
			5000 counts on average (based on an earlier version)
fitsio_ccdproc		Process (flatfield, debias, etc...) an image
imcore_conf		Generate object catalogue for and image
imcore_list		Do photometry on an image, using an input object list


* IRAF scripts in /home/mgh/iraf/scripts/new/

AptAstrom	Find Tycho stars in image, do quick astrometry & photometry on
		them to set WCS info in header & estimate sky transparency.
		Also adds ephemeris info such as JD, UT, ST, AIRMASS, etc...

(../setheader	used to be for seting JD, UT, ST, in headers , but no longer 
		needed now that AptAtrom does this)