In-Transit Molecular Dynamics Analysis with Apache Flink
Use este enlace para citar
http://hdl.handle.net/2183/32255Coleccións
Metadatos
Mostrar o rexistro completo do ítemTítulo
In-Transit Molecular Dynamics Analysis with Apache FlinkData
2018-11Cita bibliográfica
Henrique C. Zanúz, Bruno Raffin, Omar A. Mures, and Emilio J. Padrón. 2018. In-transit molecular dynamics analysis with Apache flink. In Proceedings of the Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization (ISAV '18). Association for Computing Machinery, New York, NY, USA, 25–32. https://doi.org/10.1145/3281464.3281469
Resumo
[Abstract] In this paper, an on-line parallel analytics framework is proposed to process and store in transit all the data being generated by a Molecular Dynamics (MD) simulation run using staging nodes in the same cluster executing the simulation. The implementation and deployment of such a parallel workflow with standard HPC tools, managing problems such as data partitioning and load balancing, can be a hard task for scientists. In this paper we propose to leverage Apache Flink, a scalable stream processing engine from the Big Data domain, in this HPC context. Flink enables to program analyses within a simple window based map/reduce model, while
the runtime takes care of the deployment, load balancing and fault tolerance. We build a complete in transit analytics workflow, connecting an MD simulation to Apache Flink and to a distributed database, Apache HBase, to persist all the desired data. To demonstrate the expressivity of this programming model and its suitability for HPC scientific environments, two common analytics in the MD field have been implemented. We assessed the performance of this framework, concluding that it can handle simulations of sizes used in the literature while providing an effective and versatile tool for scientists to easily incorporate on-line parallel analytics in their
current workflows.
Palabras chave
In-transit analysis
Apache Flink
Molecular Dynamics
Online analysis
Apache Flink
Molecular Dynamics
Online analysis
Descrición
SC18's workshop
Versión do editor
ISBN
978-1-4503-6579-6