Curtin University Homepage
  • Library
  • Help
    • Admin

    espace - Curtin’s institutional repository

    JavaScript is disabled for your browser. Some features of this site may not work without it.
    View Item 
    • espace Home
    • espace
    • Curtin Research Publications
    • View Item
    • espace Home
    • espace
    • Curtin Research Publications
    • View Item

    Pipeline Collector: Gathering performance data for distributed astronomical pipelines

    75149.pdf (1.904Mb)
    Access Status
    Open access
    Authors
    Mechev, A.
    Plaat, A.
    Oonk, J.
    Intema, Hubertus
    Röttgering, H.
    Date
    2018
    Type
    Journal Article
    
    Metadata
    Show full item record
    Citation
    Mechev, A. and Plaat, A. and Oonk, J. and Intema, H. and Röttgering, H. 2018. Pipeline Collector: Gathering performance data for distributed astronomical pipelines. Astronomy and Computing. 24: pp. 117-128.
    Source Title
    Astronomy and Computing
    DOI
    10.1016/j.ascom.2018.06.005
    Remarks

    © 2018. This manuscript version is made available under the CC-BY-NC-ND 4.0 license http://creativecommons.org/licenses/by-nc-nd/4.0/

    URI
    http://hdl.handle.net/20.500.11937/74864
    Collection
    • Curtin Research Publications
    Abstract

    Modern astronomical data processing requires complex software pipelines to process ever growing datasets. For radio astronomy, these pipelines have become so large that they need to be distributed across a computational cluster. This makes it difficult to monitor the performance of each pipeline step. To gain insight into the performance of each step, a performance monitoring utility needs to be integrated with the pipeline execution. In this work we have developed such a utility and integrated it with the calibration pipeline of the Low Frequency Array, LOFAR, a leading radio telescope. We tested the tool by running the pipeline on several different compute platforms and collected the performance data. Based on this data, we make well informed recommendations on future hardware and software upgrades. The aim of these upgrades is to accelerate the slowest processing steps for this LOFAR pipeline. The pipeline_collector suite is open source and will be incorporated in future LOFAR pipelines to create a performance database for all LOFAR processing.

    Related items

    Showing items related by title, author, creator and subject.

    • Scalability model for the LOFAR direction independent pipeline
      Mechev, A.P.; Shimwell, T.W.; Plaat, A.; Intema, Huib ; Varbanescu, A.L.; Rottgering, H.J.A. (2019)
      LOFAR is a leading aperture synthesis telescope operated in the Netherlands with stations across Europe. The LOFAR Two-meter Sky Survey (LoTSS) will produce more than 3000 14 TB data sets, mapping the entire northern sky ...
    • Systematic effects in LOFAR data: A unified calibration strategy
      De Gasperin, F.; Dijkema, T.J.; Drabent, A.; Mevius, M.; Rafferty, D.; Van Weeren, R.; Brüggen, M.; Callingham, J.R.; Emig, K.L.; Heald, G.; Intema, Huib ; Morabito, L.K.; Offringa, A.R.; Oonk, R.; Orrù, E.; Röttgering, H.; Sabater, J.; Shimwell, T.; Shulevski, A.; Williams, W. (2019)
      Context. New generation low-frequency telescopes are exploring a new parameter space in terms of depth and resolution. The data taken with these interferometers, for example with the LOw Frequency ARray (LOFAR), are often ...
    • Observing pulsars and fast transients with LOFAR
      Stappers, B.; Hessels, J.; Alexov, A.; Anderson, K.; Coenen, T.; Hassall, T.; Karastergiou, A.; Kondratiev, V.; Kramer, M.; van Leeuwen, J.; Mol, J.; Noutsos, A.; Romein, J.; Weltevrede, P.; Fender, R.; Wijers, R.; Baehren, L.; Bell, M.; Broderick, J.; Daw, E.; Dhillon, V.; Eisloeffel, J.; Falcke, H.; Griessmeier, J.; Law, C.; Markoff, S.; Miller-Jones, James; Scheers, B.; Spreeuw, H.; Swinbank, J.; ter Veen, S.; Wise, M.; Wucknitz, O.; Zarka, P.; Anderson, J.; Asgekar, A.; Avruch, I.; Beck, R.; Bennema, P.; Bentum, M.; Best, P.; Bregman, J.; Brentjens, M.; van de Brink, R.; Broekema, P.; Brouw, W.; Brueggen, M.; de Bruyn, A.; Butcher, H.; Ciardi, B.; Conway, J.; Dettmar, R.; van Duin, A.; van Enst, J.; Garrett, M.; Gerbers, M.; Grit, T.; Gunst, A.; van Haarlem, M.; Hamaker, J.; Heald, G.; Hoeft, M.; Holties, H.; Horneffer, A.; Koopmans, L.; Kuper, G.; Loose, M.; Maat, P.; McKay-Bukowski, D.; McKean, J.; Miley, G.; Morganti, R.; Nijboer, R.; Noordam, J.; Norden, M.; Olofsson, H.; Pandey-Pommier, M.; Polatidis, A.; Reich, W.; Roettgering, H.; Schoenmakers, A.; Sluman, J.; Smirnov, O.; Steinmetz, M.; Sterks, C.; Tagger, M.; Tang, Y.; Vermeulen, R.; Vermaas, N.; Vogt, C.; de Vos, M.; Wijnholds, S.; Yatawatta, S.; Zensus, J. (2011)
      Low frequency radio waves, while challenging to observe, are a rich source of information about pulsars. The LOw Frequency ARray (LOFAR) is a new radio interferometer operating in the lowest 4 octaves of the ionospheric ...
    Advanced search

    Browse

    Communities & CollectionsIssue DateAuthorTitleSubjectDocument TypeThis CollectionIssue DateAuthorTitleSubjectDocument Type

    My Account

    Admin

    Statistics

    Most Popular ItemsStatistics by CountryMost Popular Authors

    Follow Curtin

    • 
    • 
    • 
    • 
    • 

    CRICOS Provider Code: 00301JABN: 99 143 842 569TEQSA: PRV12158

    Copyright | Disclaimer | Privacy statement | Accessibility

    Curtin would like to pay respect to the Aboriginal and Torres Strait Islander members of our community by acknowledging the traditional owners of the land on which the Perth campus is located, the Whadjuk people of the Nyungar Nation; and on our Kalgoorlie campus, the Wongutha people of the North-Eastern Goldfields.