New Science Data Archive Launched

Last month, the Community Science and Data Center (CSDC) rolled out a new data archive platform, the NOIRLab Astro Data Archive, which will replace the current NOAO Science Archive software system later this year. The Astro Data Archive provides access to data taken with more than 40 telescope and instrument combinations from NOIRLab’s CTIO and KPNO facilities, including those operated in partnership with the WIYN, SOAR, and SMARTS consortia. Both raw data and pipeline-reduced data products from the DECam, Mosaic, and NEWFIRM imagers are available, as well as advanced data products from teams carrying out surveys and other large observing programs with NOIRLab facilities.

The holdings are extensive:

• Comprising 1.23 petabytes of raw and reduced data in total, 89% of the holdings are from DECam (1095 terabytes), with Mosaic (68 terabytes) and NEWFIRM (19 terabytes) contributing significant fractions.
• The DECam and Mosaic science exposures together currently cover approximately 90% of the sky. (DECam covers 74% and Mosaic 22% of the sky.)
• Large coherent survey datasets include those of the Dark Energy Survey, the DESI Legacy Imaging Surveys, the DECam Local Volume Exploration (DELVE) Survey, and other NOAO Survey Programs.

The welcome page of the Astro Data Archive offers information and multiple ways to access the archive

The new system continues in its primary role of serving public and proprietary pixel-based data products to PI and survey programs. Upgrades include a modern high-performing back-end and near-instantaneous search/filter capabilities through the web interface. In addition, to meet the demands of data-intensive science, a new API (application program interface) is available. The API can be accessed through Python notebooks as well as through the NOIRLab Astropy/Astroquery module. Astroquery is a set of tools for querying astronomical web forms and databases. The Astro Data Archive website offers example Python Jupyter notebooks, as well as basic examples illustrating the use of the NOIRLab astroquery module.

To facilitate a smooth transition of Archive users and data services, the Astro Data Archive was deployed as a limited release and will be operated in parallel with the legacy Archive through the 2020A semester. The limited release offers a feature-complete system, prioritizing access to contemporary PI and survey instrument programs; access to some historic data is pending. The complete peta-scale collection of data products is expected to be available in October 2020, coinciding with the retirement of the legacy system.