The IRMA Community
Newsletters
Research IRM
Click a keyword to search titles using our InfoSci-OnDemand powered search:
|
Scalable Data Mining, Archiving, and Big Data Management for the Next Generation Astronomical Telescopes
|
|
Author(s): Chris A. Mattmann (California Institute of Technology, USA), Andrew Hart (California Institute of Technology, USA), Luca Cinquini (California Institute of Technology, USA), Joseph Lazio (California Institute of Technology, USA), Shakeh Khudikyan (California Institute of Technology, USA), Dayton Jones (California Institute of Technology, USA), Robert Preston (California Institute of Technology, USA), Thomas Bennett (SKA South Africa Project, South Africa), Bryan Butler (National Radio Astronomy Observatory (NRAO), USA), David Harland (National Radio Astronomy Observatory (NRAO), USA), Brian Glendenning (National Radio Astronomy Observatory (NRAO), USA), Jeff Kern (National Radio Astronomy Observatory (NRAO), USA)and James Robnett (National Radio Astronomy Observatory (NRAO), USA)
Copyright: 2016
Pages: 27
Source title:
Big Data: Concepts, Methodologies, Tools, and Applications
Source Author(s)/Editor(s): Information Resources Management Association (USA)
DOI: 10.4018/978-1-4666-9840-6.ch100
Purchase
|
Abstract
Big data as a paradigm focuses on data volume, velocity, and on the number and complexity of various data formats and metadata, a set of information that describes other data types. This is nowhere better seen than in the development of the software to support next generation astronomical instruments including the MeerKAT/KAT-7 Square Kilometre Array (SKA) precursor in South Africa, in the Low Frequency Array (LOFAR) in Europe, in two instruments led in part by the U.S. National Radio Astronomy Observatory (NRAO) with its Expanded Very Large Array (EVLA) in Socorro, NM, and Atacama Large Millimeter Array (ALMA) in Chile, and in other instruments such as the Large Synoptic Survey Telescope (LSST) to be built in northern Chile. This chapter highlights the big data challenges in constructing data management systems for these astronomical instruments, specifically the challenge of integrating legacy science codes, handling data movement and triage, building flexible science data portals and user interfaces, allowing for flexible technology deployment scenarios, and in automatically and rapidly mitigating the difference in science data formats and metadata models. The authors discuss these challenges and then suggest open source solutions to them based on software from the Apache Software Foundation including Apache Object-Oriented Data Technology (OODT), Tika, and Solr. The authors have leveraged these solutions to effectively and expeditiously build many precursor and operational software systems to handle data from these astronomical instruments and to prepare for the coming data deluge from those not constructed yet. Their solutions are not specific to the astronomical domain and they are already applicable to a number of science domains including Earth, planetary, and biomedicine.
Related Content
|
.
© 2023.
34 pages.
|
|
.
© 2023.
15 pages.
|
|
.
© 2023.
15 pages.
|
|
.
© 2023.
18 pages.
|
|
.
© 2023.
24 pages.
|
|
.
© 2023.
32 pages.
|
|
.
© 2023.
21 pages.
|
|
|