Pipeline Collector: Gathering performance data for distributed astronomical pipelines
Access Status
Authors
Date
2018Type
Metadata
Show full item recordCitation
Source Title
Remarks
© 2018. This manuscript version is made available under the CC-BY-NC-ND 4.0 license http://creativecommons.org/licenses/by-nc-nd/4.0/
Collection
Abstract
Modern astronomical data processing requires complex software pipelines to process ever growing datasets. For radio astronomy, these pipelines have become so large that they need to be distributed across a computational cluster. This makes it difficult to monitor the performance of each pipeline step. To gain insight into the performance of each step, a performance monitoring utility needs to be integrated with the pipeline execution. In this work we have developed such a utility and integrated it with the calibration pipeline of the Low Frequency Array, LOFAR, a leading radio telescope. We tested the tool by running the pipeline on several different compute platforms and collected the performance data. Based on this data, we make well informed recommendations on future hardware and software upgrades. The aim of these upgrades is to accelerate the slowest processing steps for this LOFAR pipeline. The pipeline_collector suite is open source and will be incorporated in future LOFAR pipelines to create a performance database for all LOFAR processing.
Related items
Showing items related by title, author, creator and subject.
-
Mechev, A.P.; Shimwell, T.W.; Plaat, A.; Intema, Huib ; Varbanescu, A.L.; Rottgering, H.J.A. (2019)LOFAR is a leading aperture synthesis telescope operated in the Netherlands with stations across Europe. The LOFAR Two-meter Sky Survey (LoTSS) will produce more than 3000 14 TB data sets, mapping the entire northern sky ...
-
De Gasperin, F.; Dijkema, T.J.; Drabent, A.; Mevius, M.; Rafferty, D.; Van Weeren, R.; Brüggen, M.; Callingham, J.R.; Emig, K.L.; Heald, G.; Intema, Huib ; Morabito, L.K.; Offringa, A.R.; Oonk, R.; Orrù, E.; Röttgering, H.; Sabater, J.; Shimwell, T.; Shulevski, A.; Williams, W. (2019)Context. New generation low-frequency telescopes are exploring a new parameter space in terms of depth and resolution. The data taken with these interferometers, for example with the LOw Frequency ARray (LOFAR), are often ...
-
Stappers, B.; Hessels, J.; Alexov, A.; Anderson, K.; Coenen, T.; Hassall, T.; Karastergiou, A.; Kondratiev, V.; Kramer, M.; van Leeuwen, J.; Mol, J.; Noutsos, A.; Romein, J.; Weltevrede, P.; Fender, R.; Wijers, R.; Baehren, L.; Bell, M.; Broderick, J.; Daw, E.; Dhillon, V.; Eisloeffel, J.; Falcke, H.; Griessmeier, J.; Law, C.; Markoff, S.; Miller-Jones, James; Scheers, B.; Spreeuw, H.; Swinbank, J.; ter Veen, S.; Wise, M.; Wucknitz, O.; Zarka, P.; Anderson, J.; Asgekar, A.; Avruch, I.; Beck, R.; Bennema, P.; Bentum, M.; Best, P.; Bregman, J.; Brentjens, M.; van de Brink, R.; Broekema, P.; Brouw, W.; Brueggen, M.; de Bruyn, A.; Butcher, H.; Ciardi, B.; Conway, J.; Dettmar, R.; van Duin, A.; van Enst, J.; Garrett, M.; Gerbers, M.; Grit, T.; Gunst, A.; van Haarlem, M.; Hamaker, J.; Heald, G.; Hoeft, M.; Holties, H.; Horneffer, A.; Koopmans, L.; Kuper, G.; Loose, M.; Maat, P.; McKay-Bukowski, D.; McKean, J.; Miley, G.; Morganti, R.; Nijboer, R.; Noordam, J.; Norden, M.; Olofsson, H.; Pandey-Pommier, M.; Polatidis, A.; Reich, W.; Roettgering, H.; Schoenmakers, A.; Sluman, J.; Smirnov, O.; Steinmetz, M.; Sterks, C.; Tagger, M.; Tang, Y.; Vermeulen, R.; Vermaas, N.; Vogt, C.; de Vos, M.; Wijnholds, S.; Yatawatta, S.; Zensus, J. (2011)Low frequency radio waves, while challenging to observe, are a rich source of information about pulsars. The LOw Frequency ARray (LOFAR) is a new radio interferometer operating in the lowest 4 octaves of the ionospheric ...