BDWatchdog: real-time monitoring and profiling of Big Data applications and frameworks
View/ Open
Use this link to cite
http://hdl.handle.net/2183/21722
Except where otherwise noted, this item's license is described as Atribución-NoComercial-SinDerivadas 3.0 España
Collections
- GI-GAC - Artigos [192]
Metadata
Show full item recordTitle
BDWatchdog: real-time monitoring and profiling of Big Data applications and frameworksDate
2018-10Citation
Jonatan Enes, Roberto R. Expósito, Juan Touriño, BDWatchdog: Real-time monitoring and profiling of Big Data applications and frameworks, Future Generation Computer Systems, Volume 87, 2018, Pages 420-437, ISSN 0167-739X, https://doi.org/10.1016/j.future.2017.12.068.
Abstract
[Abstract] Current Big Data applications are characterized by a heavy use of system resources (e.g., CPU, disk) generally distributed across a cluster. To effectively improve their performance there is a critical need for an accurate analysis of both Big Data workloads and frameworks. This means to fully understand how the system resources are being used in order to identify potential bottlenecks, from resource to code bottlenecks. This paper presents BDWatchdog, a novel framework that allows real-time and scalable analysis of Big Data applications by combining time series for resource monitorization and flame graphs for code profiling, focusing on the processes that make up the workload rather than the underlying instances on which they are executed. This shift from the traditional system-based monitorization to a process-based analysis is interesting for new paradigms such as software containers or serverless computing, where the focus is put on applications and not on instances. BDWatchdog has been evaluated on a Big Data cloud-based service deployed at the CESGA supercomputing center. The experimental results show that a process-based analysis allows for a more effective visualization and overall improves the understanding of Big Data workloads. BDWatchdog is publicly available at http://bdwatchdog.dec.udc.es.
Keywords
Big Data
Monitoring
Profiling
Time series
Flame graphs
Process-based analysis
Monitoring
Profiling
Time series
Flame graphs
Process-based analysis
Description
This is a post-peer-review, pre-copyedit version of an article published in Future Generation Computer Systems. The final authenticated version is available online at: https://doi.org/10.1016/j.future.2017.12.068
Editor version
Rights
Atribución-NoComercial-SinDerivadas 3.0 España
ISSN
0167-739X
1872-7115
1872-7115