BDWatchdogFaaS: A Tool for Monitoring and Analysis of Functions-as-a-Service in Cloud Environment

Abstract

BDWatchdog is a framework to assist in the in-depth and real-time analysis of the execution of Big Data frameworks and applications. BDWatchdog was originally developed to monitor Hadoop ecosystems deployed on serverless containers, in order to detect bottlenecks and spot certain patterns that frameworks or applications may have. In thisworkwe shift the focus to monitoring serverless functions in the public cloud, by proposing an extension of BDWatchdog which captures, transforms and analyzes logs from both AWS Cloudwatch and Azure Application Insights, which store the logs from AWS Lambda and Azure Functions (respectively), the FaaS (Function-as-a-Service) solutions of the two main public cloud providers, AWS and Azure. The extension, called BDWatchdogFaaS, builds and stores a common model to both providers, allowing to consult, analyze and monitor function logs from AWS and Azure indistinctly. The transformation of logs into the common model is done by a FaaS of the corresponding provider, which in near real-time ingests, processes and sends the data to a common storage. In addition, the data is forwarded to a Power BI dashboard so that the serverless functions can be monitored easily