Communicating DECam Exposures to Ignore to the NOIRLab Community Pipeline

There are a variety of situations where bad exposures occur or exposures are taken that are not of interest to the pipeline or the archive.  Bad exposures include electronic problems, tracking errors, saturation, and complete cloud cover (i.e. no sources).  Ignorable data are things such as engineering or diagnostic data which are not taken under the engineering proposal identifier.  The main concern for the pipeline and archive is not individual calibrated data products but the effect of such data on stacks; one trailed exposure in a stack is annoying and generally requires pipeline reprocessing to fix.  What is not intended is to identify exposures which the observing program plans to ignore.  Such images will still have archival value.

Note that what is described here only applies to the processing of exposures and the archiving of the data products.  The raw exposures are archived regardless of anything discussed here.

There are two mechanisms for identifying these ignorable exposures to the Community Pipeline (CP).  The first is to include the string junk (case and position insensitive) in the object title.  One might think to use test, but there are sometimes programmatic reasons observers use this word and so it is not (currently) excluded.  Other words in the title that cause the pipeline to ignore an exposure are focus, pointing, and donut.

Of course using junk in the title only works when the observer has foreknowledge that the exposure is ignorable.  After the exposure the NOIRLab data handling system whisks the data on its way to the archive and the CP.  To identify data to ignore after the exposure, observers should keep a text file of SISPI exposure numbers.  This file must have a name beginning with CPignore and ending with .txt .  Anything in-between is allowed and might be used to create multiple files by day, hour, field, etc.  To communicate with the CP send this file as an attachment to community.pipelines@noirlab.edu.  It is also desirable that the subject include the attachment name, though if it is forgotten it doesn't affect the receipt of the file.  The body of the email may include additional information such as observing problems and data problems that might be useful for the CP operator to know.  This is not, however, to request special handling and processing.

The CP must receive the file prior to when it runs.  This is typically after the end of the run though long runs or split night scheduling may alter this.  So observers should ideally send the file by the end of each night.  As noted previously, multiple messages during the night are also acceptable.

It might help to understand the process in a little more detail.  When mail is received it is automatically searched for attachments whose names meet the rule of beginning with CPignore and ending with .txt.  These attachments are extracted and the contents are added to a list of exposures to be ignored.  When the pipeline stages data it matches the acquisition name, say DECam_00323844.fits.fz, against the strings in the file.  Hence the prefix 'DECam_00' may or may not be included.  The concern is to not have extraneous material that might inadvertantly match; e.g.  'DECam' by itself.  That is why the request is for the file to consist of only the 6-8 digit exposure sequence counter.  Note there is no facility for ranges; each exposure needs to be identified separately on a line.

An example attachment might be:

We hope these mechanisms prove easy to use and useful for CP data calibration and archiving.  Any comments on this or issues during observing can be sent to the community.pipelines@noirlab.edu address as well.

The NOIRLab-DECam Pipeline Group