CSDC/MSO Pipelines

NSF's National Optical-Infrared Research Laboratory (NOIRLab) Community Science and Data Center (CSDC) is charged with development and/or support of pipelines for NOIRLab's Mid-Scale Observatories (MSO) affilliated instruments by the CSDC/MSO pipeline group. The pipeline group has developed software and pipeline infrastructure -- e.g. the NOAO High-Perforamnce Pipeline System (NHPPS) -- for several instruments -- e.g. DECam, MOSAIC, and NEWFIRM. The NHPPS-based pipelines are highly efficient, distributed and parallel applications. CSDC/MSO has also been involved with collaborative pipelines; to be provided to and operated by the CSDC, e.g. Dark Energy Camera Community Pipeline (DCP), or to provide pipeline software, e.g. the WIYN One Degree Imager (ODI) Pipeline, Portal, and Archive project.

Living software evolves with time and so a document library will contain documents of different vintages. Also documents may refer to the MSO predecessor NOAO and various programs -- Data Products Program (DPP) and Science Data Management (SDM). In this library, those documents which are still reasonably accurate, though out-of-date in some details, may be found. Documents are in various formats with PDF and HTML being most common.

Recent Documents

PL206: The DECam Community Pipelines Instrumental Signature Removal, Valdes, 2021 (Version 1.0, August 2021, html)
PL203: Instrumental Signature Removal for the Legacy Surveys, Valdes, 2021 (DRAFT IN PROGRESS, html)
PL028: Operator's Guide to the DECam Community Pipeline, Valdes, 2020 (V1 August 2020, pdf)

Pipeline Technical Document Series:

NOAO High Performance Pipeline System (NHPPS) Documents

PL001: The NOAO High-Performance Pipeline System, Valdes et al., 2006 (html)
PL002: The NHPPS Pipeline Definition Language (PDL), Valdes et al., 2012 (html)
PL008: The NOAO High-Performance Pipeline System: Architecture Overview, Cline et al., 2006
PL010: The NOAO Mosaic Pipeline Architecture, Pierfederici et al., 2003
PL011: FLIPPER: A FLexIble PiPeline framEwoRk, Pierfederici, 2005
PL027: The NHPPS Simple, Light-Weight Message Protocol, Valdes, et al., Jan. 2012

DECam Community Pipeline (DCP) and Legacy Surveys Pipelines

The Dark Energy Camera Community Pipeline (DCP) is a currently active (and evolving) MSO pipeline for the instrumental calibration of all DECam observations. The Legacy Surveys, which cover a large fraction of the sky, makes use of the DCP as well as pipelines specifically developed for the surveys and the Mosaic-3 and Steward Observatory Bok camera. The latter pipelines are documented in one of the documents in this section.

PL206: The DECam Community Pipelines Instrumental Signature Removal, Valdes, 2021 (Version 1.0, August 2021, html)
PL201: Preliminary Description of the DECam Community Pipeline, Valdes, (January 2014, html)
PL202: The DECam Community Pipeline - an overview, Valdes, Gruendl, and the DES Project, 2014 (ADASS preprint)
PL203: Instrumental Signature Removal for the Legacy Surveys, Valdes, 2021 (DRAFT IN PROGRESS, html)
PL028: Operator's Guide to the DECam Community Pipeline, Valdes, 2020 (V1 August 2020, pdf)
BOK 90prime Pipeline (pptx): Overview of the BOK 90prime Pipeline (BCP) at the 2015 Legacy Surveys team meeting, Valdes, 2015
PL301: Calibrating Astronomical Exposures Contaminated by Out-of-Focus (Pupil) Camera Reflections, Valdes, 2013. Instrumental Signature Removal for the Legacy Surveys, Valdes, 2021 (DRAFT IN PROGRESS, html)
The NOAO Data Handbook (pdf): Includes information about instruments, pipelines, and data formats. There is a chapter on the CP circa 2015, V2.2 May 2015
PL204: Integrating the DESDM-CP Into NOAO/E2Ea, Valdes, 2012.
PL205: The NOAO DECam Community Pipeline Option, Valdes and Swaters, 2011.

Mosaic1 and Mosaic2

PL003: The NOAO High-Performance Pipeline System: The Mosaic Camera Pipeline Calibrations, Valdes, Swaters, Huard, February 1, 2008 (Incomplete)
PL004: Mosaic Pipeline Operator's Manual, Valdes and Zarate
PL007: The NOAO High-Performance Pipeline System: The Mosaic Camera Pipeline Algorithms, Valdes and Swaters
PL009: The NOAO High-Performance Pipeline System: The Mosaic Camera Pipeline, Swaters and Valdes

MSO Specific Documents

PL012: NOAO Science Pipelines Operations Model, Valdes et al.
PL014: NEWFIRM KTM, Observing Tools, and Quick-Reduce Pipeline, Valdes, Swaters, and Dickinson
PL016: NEWFIRM Linearity Calibration and Correction, Mark Dickinson
PL020: The NOAO Calibration Library System, Valdes, 2018
PL024: Remediation in the NOAO Pipelines, Valdes and Swaters
PL026: Pipeline Scheduling Operator Interface, Valdes

ODI Documents

NOAO collaborated with the WIYN Consortium, and the Indiana Supercomputer Center to develop an NHPPS pipeline for the first version of the One Degree Imager that would be hosted at the supercomputer center and make use of the Open Grid Computing Environment (OGCE) as a meta-orchestrator. After considerable development work the WIYN Consortium decided to take a different path to an ODI calibration pipeline. So the articles here are largely historical but contain some interesting concepts and documentation.

PL013: ODI Pipeline Data Flow Design, Valdes and Swaters
PL019: Transforming NHPPS Pipeline Applications into Grid Applications, Valdes
PL021: The ODI Demonstration Tier 1 Pipeline, Valdes and Marru
PL023: The Marriage of Mario (NHPPS) and Luigi (OGCE), Valdes and Marru
PL101: Automatic Calibration Pipeline Application Design, Valdes
PL102: High Level ODI Pipeline Workflow Architecture, Valdes (HTML)
PL103/CDR-45: The ODI-PPA Calibration Library System, Valdes, Swaters, Sun, Nov. 2011
PL104/CDR-46: The ODI Pipeline Scheduling Agent ICD, Valdes, Swaters, Dec. 2011
PL105/CDR-18: ODI Automatic Calibration Pipeline Design Elements, Swaters, Valdes, Feb. 2012
PL106/CDR-19: ODI Automatic Pipeline Data Products, Swaters, Valdes, Oct. 2011

ADASS/SPIE

The NEWFIRM Observing Software: From Design To Implementation
Daly, et al., 2008 (SPIE 7019-40)
This includes a brief description of the NEWFIRM Quick Reduce Pipeline and triggering of the pipeline by the data handling system.
The NOAO High-Performance Pipeline System: The Mosaic Camera Pipeline Algorithms
Valdes and Swaters, 2006
NOAO SDM Document PL007, ADASS preprint
The NOAO High-Performance Pipeline System: Architecture Overview
Cline, et al. 2006
NOAO SDM Document PL008, ADASS preprint
The NOAO High-Performance Pipeline System: The Mosaic Camera Pipeline
Swaters and Valdes 2006
NOAO SDM Document PL009, ADASS preprint
The NOAO Mosaic Pipeline Architecture
Pierfederici, et al. 2003
NOAO SDM Document PL010, ADASS preprint
FLIPPER: a FLexIble PiPeline framEwoRk
Pierfederici 2004
NOAO SDM Document PL011, ADASS preprint
The DECam Community Pipeline
Valdes, Gruendl, and the DES Project, 2014
ADASS preprint