Otero, David2025-03-122025-03-122019-07Otero, D. (2019). Exploiting Pooling Methods for Building Datasets for Novel Tasks. In CEUR Workshop Proceedings, Vol. 2537, art. 16, pp. 96-102, Proceedings of the 9th PhD Symposium on Future Directions in Information Access co-located with 12th European Summer School in Information Retrieval (ESSIR 2019), Milan, Italy, July 17th to 18th, 2019.1613-0073http://hdl.handle.net/2183/41363Proceedings of the 9th PhD Symposium on Future Directions in Information Access (FDIA 2019) co-located with 12th European Summer School in Information Retrieval (ESSIR 2019) Milan, Italy, July 17th to 18th, 2019, published at http://ceur-ws.org.[Abstract]: Information Retrieval is not any more exclusively about document ranking. Continuously new tasks are proposed on this and sibling fields. With this proliferation of tasks, it becomes crucial to have a cheap way of constructing test collections to evaluate the new developments. Building test collections is time and resource consuming: it requires time to obtain the documents, to define the user needs and it requires assessors to judge a lot of documents. To reduce the latest, pooling strategies aim to decrease the assessment effort by presenting to the assessors a sample of documents in the corpus with the maximum number of relevant documents in it. The quality of these collections is also crucial, as the value of any evaluation depends on it. In this article, we propose the design of a system for building test collections easily and cheaply by implementing state-of-the-art pooling strategies and simulating competition participants with different retrieval models and query variants. We aim to achieve flexibility in terms of adding new retrieval models and pooling strategies to the system. We want the platform also to be useful to evaluate the obtained collections.engAtribución 4.0 InternacionalCopyright © 2019 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).http://creativecommons.org/licenses/by/3.0/es/Information retrievalTest collectionsPoolingconference outputopen access