A novel framework for generic Spark workload characterization and similar pattern recognition using machine learning

Use este enlace para citar
http://hdl.handle.net/2183/36284
Excepto si se señala otra cosa, la licencia del ítem se describe como Atribución-NoComercial 3.0 España
Colecciones
- Investigación (FIC) [1615]
Metadatos
Mostrar el registro completo del ítemTítulo
A novel framework for generic Spark workload characterization and similar pattern recognition using machine learningFecha
2024-07Cita bibliográfica
M. Garralda-Barrio, C. Eiras-Franco, and V. Bolón-Canedo, "A novel framework for generic Spark workload characterization and similar pattern recognition using machine learning", Journal of Parallel and Distributed Computing, Vol. 189, 104881, Jul. 2024, doi: 10.1016/j.jpdc.2024.104881
Resumen
[Abstract]: Comprehensive workload characterization plays a pivotal role in comprehending Spark applications, as it enables the analysis of diverse aspects and behaviors. This understanding is indispensable for devising downstream tuning objectives, such as performance improvement. To address this pivotal issue, our work introduces a novel and scalable framework for generic Spark workload characterization, complemented by consistent geometric measurements. The presented approach aims to build robust workload descriptors by profiling only quantitative metrics at the application task-level, in a non-intrusive manner. We expand our framework for downstream workload pattern recognition by incorporating unsupervised machine learning techniques: clustering algorithms and feature selection. These techniques significantly improve the process of grouping similar workloads without relying on predefined labels. We effectively recognize 24 representative Spark workloads from diverse domains, including SQL, machine learning, web search, graph, and micro-benchmarks, available in HiBench. Our framework achieves a high accuracy F-Measure score of up to 90.9% and a Normalized Mutual Information of up to 94.5% in similar workload pattern recognition. These scores significantly outperform the results obtained in a comparative analysis with an established workload characterization approach in the literature.
Palabras clave
Apache spark
Big data
Machine learning
Pattern recognition
Workload characterization
Big data
Machine learning
Pattern recognition
Workload characterization
Versión del editor
Derechos
Atribución-NoComercial 3.0 España