BERTbek: A Pretrained Language Model for Uzbek
Title
BERTbek: A Pretrained Language Model for UzbekDate
2024-05Citation
Elmurod Kuriyozov, David Vilares, and Carlos Gómez-Rodríguez. 2024. BERTbek: A Pretrained Language Model for Uzbek. In Proceedings of the 3rd Annual Meeting of the Special Interest Group on Under-resourced Languages @ LREC-COLING 2024, pages 33–44, Torino, Italia. ELRA and ICCL. https://aclanthology.org/2024.sigul-1.5
Abstract
[Abstract]: Recent advances in neural networks based language representation made it possible for pretrained language models to outperform previous models in many downstream natural language processing (NLP) tasks. These pretrained language models have also shown that if large enough, they exhibit good few-shot abilities, which is especially beneficial for low-resource scenarios. In this respect, although there are some large-scale multilingual pretrained language models available, language-specific pretrained models have demonstrated to be more accurate for monolingual evaluation setups. In this work, we present BERTbek - pretrained language models based on the BERT (Bidirectional Encoder Representations from Transformers) architecture for the low-resource Uzbek language. We also provide a comprehensive evaluation of the models on a number of NLP tasks: sentiment analysis, multi-label topic classification, and named entity recognition, comparing the models with various machine learning methods as well as multilingual BERT (mBERT). Experimental results indicate that our models outperform mBERT and other task-specific baseline models in all three tasks. Additionally, we also show the impact of training data size and quality on the downstream performance of BERT models, by training three different models with different text sources and corpus sizes.
Keywords
BERT
language modeling
low-resource languages
natural language processing
Uzbek language
language modeling
low-resource languages
natural language processing
Uzbek language
Description
All the code used in this work are openly available at https://github.com/elmurod1202/BERTbek. Also, the BERTbek models have been uploaded to the HuggingFace Models Hub at https://huggingface.co/elmurod1202/bertbek-news-big-cased.
Editor version
Rights
Atribución-NoComercial 3.0 España
Related items
Showing items related by title, author, creator and subject.
-
A influencia das linguas dominantes no desenvolvemento de políticas de linguaxe non-sexista en linguas subordinadas: análise do cado do galés
Baxter, Robert Neal (Universidade da Coruña, 2009)[Resumo] Partindo dunha análise do caso concreto das propostas para a elaboración dunha política oficial de linguaxe non-sexista para o galés, este artigo, por unha parte, explora como a condición de lingua subordinada ... -
Hacia una didáctica de la lengua minorizada en contextos de asimilación lingüística: algunas aportaciones desde la teoría
Campos, Iris Orosia (Universidade da Coruña, 2018)[Resumen] Actualmente, alrededor de 120 lenguas europeas son denominadas como regionales o minoritarias y solo el 10% de los idiomas del mundo están fuera de riesgo de desaparición. Uno de los principales campos para la ... -
O neofalantismo no ensino secundario: un espazo sen lexitimar
López Alonso, Luís (Universidade da Coruña - Servizo de Publicacións, 2020-12-31)[Resumo] Após de corenta anos de oficialización da lingua galega, os datos estatísticos evidencian que tanto o modelo educativo como os outros mecanismos de normalización lingüística fracasaron. A substitución lingüística ...