CSDC/MSO Pipelines

NSF's National Optical-Infrared Research Laboratory (NOIRLab) Community Science and Data Center (CSDC) is charged with development and/or support of pipelines for NOIRLab's Mid-Scale Observatories (MSO) affilliated instruments by the CSDC/MSO pipeline group. The pipeline group has developed software and pipeline infrastructure -- e.g. the NOAO High-Perforamnce Pipeline System (NHPPS) -- for several instruments -- e.g. DECam, MOSAIC, and NEWFIRM. The NHPPS-based pipelines are highly efficient, distributed and parallel applications. CSDC/MSO has also been involved with collaborative pipelines; to be provided to and operated by the CSDC, e.g. Dark Energy Camera Community Pipeline (DCP), or to provide pipeline software, e.g. the WIYN One Degree Imager (ODI) Pipeline, Portal, and Archive project.

Living software evolves with time and so a document library will contain documents of different vintages. Also documents may refer to the MSO predecessor NOAO and various programs -- Data Products Program (DPP) and Science Data Management (SDM). In this library, those documents which are still reasonably accurate, though out-of-date in some details, may be found. Documents are in various formats with PDF and HTML being most common.

Recent Documents

PL206: The DECam Community Pipelines Instrumental Signature Removal, Valdes, 2021 (Version 1.0, August 2021, html)

PL203: Instrumental Signature Removal for the Legacy Surveys, Valdes, 2021 (DRAFT IN PROGRESS, html)

PL028: Operator's Guide to the DECam Community Pipeline, Valdes, 2020 (V1 August 2020, pdf)

Pipeline Technical Document Series:

NOAO High Performance Pipeline System (NHPPS) Documents

PL001: The NOAO High-Performance Pipeline System, Valdes et al., 2006 (html)

PL002: The NHPPS Pipeline Definition Language (PDL), Valdes et al., 2012 (html)

PL008: The NOAO High-Performance Pipeline System: Architecture Overview, Cline et al., 2006

PL010: The NOAO Mosaic Pipeline Architecture, Pierfederici et al., 2003

PL011: FLIPPER: A FLexIble PiPeline framEwoRk, Pierfederici, 2005

PL027: The NHPPS Simple, Light-Weight Message Protocol, Valdes, et al., Jan. 2012

DECam Community Pipeline (DCP) and Legacy Surveys Pipelines

The Dark Energy Camera Community Pipeline (DCP) is a currently active (and evolving) MSO pipeline for the instrumental calibration of all DECam observations. The Legacy Surveys, which cover a large fraction of the sky, makes use of the DCP as well as pipelines specifically developed for the surveys and the Mosaic-3 and Steward Observatory Bok camera. The latter pipelines are documented in one of the documents in this section.

PL206: The DECam Community Pipelines Instrumental Signature Removal, Valdes, 2021 (Version 1.0, August 2021, html)

PL201: Preliminary Description of the DECam Community Pipeline, Valdes, (January 2014, html)

PL202: The DECam Community Pipeline - an overview, Valdes, Gruendl, and the DES Project, 2014 (ADASS preprint)

PL203: Instrumental Signature Removal for the Legacy Surveys, Valdes, 2021 (DRAFT IN PROGRESS, html)

PL028: Operator's Guide to the DECam Community Pipeline, Valdes, 2020 (V1 August 2020, pdf)

BOK 90prime Pipeline (pptx): Overview of the BOK 90prime Pipeline (BCP) at the 2015 Legacy Surveys team meeting, Valdes, 2015

PL301: Calibrating Astronomical Exposures Contaminated by Out-of-Focus (Pupil) Camera Reflections, Valdes, 2013. Instrumental Signature Removal for the Legacy Surveys, Valdes, 2021 (DRAFT IN PROGRESS, html)

The NOAO Data Handbook (pdf): Includes information about instruments, pipelines, and data formats. There is a chapter on the CP circa 2015, V2.2 May 2015

PL204: Integrating the DESDM-CP Into NOAO/E2Ea, Valdes, 2012.

PL205: The NOAO DECam Community Pipeline Option, Valdes and Swaters, 2011.

Mosaic1 and Mosaic2

PL003: The NOAO High-Performance Pipeline System: The Mosaic Camera Pipeline Calibrations, Valdes, Swaters, Huard, February 1, 2008 (Incomplete)

PL004: Mosaic Pipeline Operator's Manual, Valdes and Zarate

PL007: The NOAO High-Performance Pipeline System: The Mosaic Camera Pipeline Algorithms, Valdes and Swaters

PL009: The NOAO High-Performance Pipeline System: The Mosaic Camera Pipeline, Swaters and Valdes

MSO Specific Documents

PL012: NOAO Science Pipelines Operations Model, Valdes et al.

PL014: NEWFIRM KTM, Observing Tools, and Quick-Reduce Pipeline, Valdes, Swaters, and Dickinson

PL016: NEWFIRM Linearity Calibration and Correction, Mark Dickinson

PL020: The NOAO Calibration Library System, Valdes, 2018

PL024: Remediation in the NOAO Pipelines, Valdes and Swaters

PL026: Pipeline Scheduling Operator Interface, Valdes

ODI Documents

NOAO collaborated with the WIYN Consortium, and the Indiana Supercomputer Center to develop an NHPPS pipeline for the first version of the One Degree Imager that would be hosted at the supercomputer center and make use of the Open Grid Computing Environment (OGCE) as a meta-orchestrator. After considerable development work the WIYN Consortium decided to take a different path to an ODI calibration pipeline. So the articles here are largely historical but contain some interesting concepts and documentation.

PL013: ODI Pipeline Data Flow Design, Valdes and Swaters

PL019: Transforming NHPPS Pipeline Applications into Grid Applications, Valdes

PL021: The ODI Demonstration Tier 1 Pipeline, Valdes and Marru

PL023: The Marriage of Mario (NHPPS) and Luigi (OGCE), Valdes and Marru

PL101: Automatic Calibration Pipeline Application Design, Valdes

PL102: High Level ODI Pipeline Workflow Architecture, Valdes (HTML)

PL103/CDR-45: The ODI-PPA Calibration Library System, Valdes, Swaters, Sun, Nov. 2011

PL104/CDR-46: The ODI Pipeline Scheduling Agent ICD, Valdes, Swaters, Dec. 2011

PL105/CDR-18: ODI Automatic Calibration Pipeline Design Elements, Swaters, Valdes, Feb. 2012

PL106/CDR-19: ODI Automatic Pipeline Data Products, Swaters, Valdes, Oct. 2011

ADASS/SPIE

The NEWFIRM Observing Software: From Design To Implementation

Daly, et al., 2008 (SPIE 7019-40)
This includes a brief description of the NEWFIRM Quick Reduce Pipeline and triggering of the pipeline by the data handling system.

The NOAO High-Performance Pipeline System: The Mosaic Camera Pipeline Algorithms

Valdes and Swaters, 2006
NOAO SDM Document PL007, ADASS preprint

The NOAO High-Performance Pipeline System: Architecture Overview

Cline, et al. 2006
NOAO SDM Document PL008, ADASS preprint

The NOAO High-Performance Pipeline System: The Mosaic Camera Pipeline

Swaters and Valdes 2006
NOAO SDM Document PL009, ADASS preprint

The NOAO Mosaic Pipeline Architecture

Pierfederici, et al. 2003
NOAO SDM Document PL010, ADASS preprint

FLIPPER: a FLexIble PiPeline framEwoRk

Pierfederici 2004
NOAO SDM Document PL011, ADASS preprint

The DECam Community Pipeline

Valdes, Gruendl, and the DES Project, 2014
ADASS preprint

Updated on December 30, 2021, 5:59 am