Parapar, JavierLosada, David E.Presedo-Quindimil, Manuel-AntonioBarreiro, Álvaro2019-02-132019-02-132019-01-11Parapar, J., Losada, D.E., Presedo-Quindimil, M.A. and Barreiro, A. (2020), 'Using Score Distributions to Compare Statistical Significance Tests for Information Retrieval Evaluation', Journal of the Association for Information Science and Technology, 71: 98-113, https://doi.org/10.1002/asi.24203.2330-1643http://hdl.handle.net/2183/21729This is the peer reviewed version of the following article: Parapar, J., Losada, D.E., Presedo-Quindimil, M.A. and Barreiro, A. (2020), 'Using Score Distributions to Compare Statistical Significance Tests for Information Retrieval Evaluation', Journal of the Association for Information Science and Technology, 71: 98-113, which has been published in final form at https://doi.org/10.1002/asi.24203. This article may be used for non-commercial purposes in accordance with Wiley Terms and Conditions for Use of Self-Archived Versions. This article may not be enhanced, enriched or otherwise transformed into a derivative work, without express permission from Wiley or by statutory rights under applicable legislation. Copyright notices must not be removed, obscured or modified. The article must be linked to Wiley’s version of record on Wiley Online Library and any embedding, framing or otherwise making available the article or pages thereof by third parties from platforms, services and websites other than Wiley Online Library must be prohibited.[Abstract] Statistical significance tests can provide evidence that the observed difference in performance between two methods is not due to chance. In Information Retrieval, some studies have examined the validity and suitability of such tests for comparing search systems.We argue here that current methods for assessing the reliability of statistical tests suffer from some methodological weaknesses, and we propose a novel way to study significance tests for retrieval evaluation. Using Score Distributions, we model the output of multiple search systems, produce simulated search results from such models, and compare them using various significance tests. A key strength of this approach is that we assess statistical tests under perfect knowledge about the truth or falseness of the null hypothesis. This new method for studying the power of significance tests in Information Retrieval evaluation is formal and innovative. Following this type of analysis, we found that both the sign test and Wilcoxon signed test have more power than the permutation test and the t-test. The sign test and Wilcoxon signed test also have a good behavior in terms of type I errors. The bootstrap test shows few type I errors, but it has less power than the other methods tested.eng© 2019 ASIS&T. Este artículo puede utilizarse con fines no comerciales conforme a los Términos y Condiciones de Wiley para el Uso de Versiones Autoarchivadas. Este artículo no puede ser mejorado, enriquecido ni transformado de otro modo en una obra derivada, sin el permiso expreso de Wiley o por derechos legales bajo la legislación aplicable. Los avisos de derechos de autor no deben ser eliminados, ocultos ni modificados. El artículo debe estar vinculado a la versión registrada de Wiley en Wiley Online Library y cualquier incrustación, encuadre o puesta a disposición del artículo o páginas del mismo por terceros de plataformas, servicios y sitios web distintos a Wiley Online Library debe estar prohibido.Information retrievalStatistical testSignificance testingWilcoxonPermutationSignBootstrapT-ttestjournal articleopen access10.1002/asi.24203