BDWatchdog: real-time monitoring and profiling of Big Data applications and frameworks
Not available until 2020-11-01
Use this link to citehttp://hdl.handle.net/2183/21722
Except where otherwise noted, this item's license is described as Atribución-NoComercial-SinDerivadas 3.0 España
- GI-GAC - Artigos 
MetadataShow full item record
TitleBDWatchdog: real-time monitoring and profiling of Big Data applications and frameworks
Jonatan Enes, Roberto R. Expósito, Juan Touriño, BDWatchdog: Real-time monitoring and profiling of Big Data applications and frameworks, Future Generation Computer Systems, Volume 87, 2018, Pages 420-437, ISSN 0167-739X, https://doi.org/10.1016/j.future.2017.12.068.
[Abstract] Current Big Data applications are characterized by a heavy use of system resources (e.g., CPU, disk) generally distributed across a cluster. To effectively improve their performance there is a critical need for an accurate analysis of both Big Data workloads and frameworks. This means to fully understand how the system resources are being used in order to identify potential bottlenecks, from resource to code bottlenecks. This paper presents BDWatchdog, a novel framework that allows real-time and scalable analysis of Big Data applications by combining time series for resource monitorization and flame graphs for code profiling, focusing on the processes that make up the workload rather than the underlying instances on which they are executed. This shift from the traditional system-based monitorization to a process-based analysis is interesting for new paradigms such as software containers or serverless computing, where the focus is put on applications and not on instances. BDWatchdog has been evaluated on a Big Data cloud-based service deployed at the CESGA supercomputing center. The experimental results show that a process-based analysis allows for a more effective visualization and overall improves the understanding of Big Data workloads. BDWatchdog is publicly available at http://bdwatchdog.dec.udc.es.
This is a post-peer-review, pre-copyedit version of an article published in Future Generation Computer Systems. The final authenticated version is available online at: https://doi.org/10.1016/j.future.2017.12.068
Atribución-NoComercial-SinDerivadas 3.0 España