Semantic Hierarchical Classification Applied to Anomaly Detection Using System Logs with a BERT Model
![Thumbnail](/dspace/bitstream/handle/2183/38332/Carneiro_Victor_2024_Semantic_Hierarchical_Classification_Applied_to_Anomaly_Detection_Using_System_Logs_BERT_Model.pdf.jpg?sequence=5&isAllowed=y)
Use this link to cite
http://hdl.handle.net/2183/38332
Except where otherwise noted, this item's license is described as Atribución 4.0 Internacional (CC-BY 4.0)
Collections
Metadata
Show full item recordTitle
Semantic Hierarchical Classification Applied to Anomaly Detection Using System Logs with a BERT ModelDate
2024-06Citation
Corbelle, C.; Carneiro, V.; Cacheda, F. Semantic Hierarchical Classification Applied to Anomaly Detection Using System Logs with a BERT Model. Appl. Sci. 2024, 14, 5388. https://doi.org/10.3390/app14135388
Abstract
[Abstract]: The compaction and structuring of system logs facilitate and expedite anomaly and cyberattack detection processes using machine-learning techniques, while simultaneously reducing alert fatigue caused by false positives. In this work, we implemented an innovative algorithm that employs hierarchical codes based on the semantics of natural language, enabling the generation of a significantly reduced log that preserves the semantics of the original. This method uses codes that reflect the specificity of the topic and its position within a higher hierarchical structure. By applying this catalog to the analysis of logs from the Hadoop Distributed File System (HDFS), we achieved a concise summary with non-repetitive themes, significantly speeding up log analysis and resulting in a substantial reduction in log size while maintaining high semantic similarity. The resulting log has been validated for anomaly detection using the “bert-base-uncased” model and compared with six other methods: PCA, IM, LogCluster, SVM, DeepLog, and LogRobust. The reduced log achieved very similar values in precision, recall, and F1-score metrics, but drastically reduced processing time.
Keywords
System logs
Anomaly detection
BERT model
Hierarchical codes
Semantic similarity
Anomaly detection
BERT model
Hierarchical codes
Semantic similarity
Editor version
Rights
Atribución 4.0 Internacional (CC-BY 4.0)
ISSN
2076-3417