| Conference and Labs of the Evaluation Forum | |
| Abbreviation: | CLEF |
| Discipline: | information retrieval |
| Publisher: | Springer and CEUR Workshop Proceedings |
| History: | (formerly known as Cross-Language Evaluation Forum from 2000 to 2009) |
| Frequency: | annual |
The Conference and Labs of the Evaluation Forum (formerly Cross-Language Evaluation Forum), or CLEF, is an organization promoting research in multilingual information access (currently focusing on European languages). Its specific functions are to maintain an underlying framework for testing information retrieval systems and to create repositories of data for researchers to use in developing comparable standards.[1] The organization holds a conference every September in Europe since a first constituting workshop in 2000. From 1997 to 1999, TREC, the similar evaluation conference organised annually in the US, included a track for the evaluation of Cross-Language IR for European languages. This track was coordinated jointly by NIST and by a group of European volunteers that grew over the years. At the end of 1999, a decision by some of the participants was made to transfer the activity to Europe and set it up independently. The aim was to expand coverage to a larger number of languages and to focus on a wider range of issues, including monolingual system evaluation for languages other than English. Over the years, CLEF has been supported by a number of various EU funded projects and initiatives.[2]
CLEF 2019 marked the 20th anniversary of the conference and it was celebrated by publishing a book[3] on the lessons learned in 20 years of evaluation activities.
Before 2010,[4] CLEF was organised as a workshop co-located with the European Conference on Digital Libraries, consisting of a number of evaluation labs or tracks, similarly to TREC. In 2010, CLEF moved to become a self-sufficiently organised conference with evaluation labs, laboratory workshops, and a main conference track. In 2012, INEX, a workshop on retrieval and access to structured text, previously organised annually at Schloß Dagstuhl, merged with CLEF to become one of its evaluation labs.
Prior to each CLEF conference, participants in evaluation labs receive a set of challenge tasks. The tasks are designed to test various aspects of information retrieval systems and encourage their development. Groups of researchers propose and organize campaigns to satisfy those tasks and the results are used as benchmarks for the state of the art in the specific areas.,[5] [6]
In the beginning, CLEF focussed mainly on fairly typical information retrieval tasks, but has moved to more specific tasks. For example, the 2005 interactive image search task worked with illustrating non-fiction texts using images from Flickr[7] and the 2010 medical retrieval task focused on retrieval of computed tomography, MRI, and radiographic images.[8] In 2017, CLEF accommodated a number of tasks e.g. on identifying biological species from photographs or video clips, on stylistic analysis of authorship, and on health related information access.
| Year | Location | Note | Proceedings | Editors | |
|---|---|---|---|---|---|
| 2000 | Lisbon | Constituting workshop | Cross-Language Information Retrieval and Evaluation: Workshop of Cross-Language Evaluation Forum (CLEF 2000) | Carol Ann Peters | |
| 2001 | Darmstadt | Evaluation of Cross-Language Information Retrieval Systems: Second Workshop of the Cross—Language Evaluation Forum (CLEF 2001) Revised Papers | Carol Ann Peters and Martin Braschler and Julio Gonzalo and Martin Kluck | ||
| 2002 | Rome | Advances in Cross-Language Information Retrieval: Third Workshop of the Cross—Language Evaluation Forum (CLEF 2002) Revised Papers | Carol Ann Peters and Martin Braschler and Julio Gonzalo and Martin Kluck | ||
| 2003 | Trondheim | Comparative Evaluation of Multilingual Information Access Systems: Fourth Workshop of the Cross—Language Evaluation Forum (CLEF 2003) Revised Selected Papers | Carol Ann Peters and Martin Braschler and Julio Gonzalo and Martin Kluck | ||
| 2004 | Bath | Multilingual Information Access for Text, Speech and Images: Fifth Workshop of the Cross—Language Evaluation Forum (CLEF 2004) Revised Selected Papers | Carol Ann Peters and Paul Clough and Julio Gonzalo and Gareth J F Jones and Martin Kluck and Bernardo Magnini | ||
| 2005 | Vienna | Accessing Multilingual Information Repositories: Sixth Workshop of the Cross—Language Evaluation Forum (CLEF 2005). Revised Selected Papers | Carol Ann Peters and Fred C Gey and Julio Gonzalo and Gareth J F Jones and Martin Kluck and Bernardo Magnini and Henning Müller and Maarten de Rijke | ||
| 2006 | Alicante | Evaluation of Multilingual and Multi-modal Information Retrieval : Seventh Workshop of the Cross—Language Evaluation Forum (CLEF 2006). Revised Selected Papers | Carol Ann Peters and Clough, P. and Fred C Gey and Jussi Karlgren and Bernardo Magnini and Doug Oard and Maarten de Rijke and M Stempfhuber | ||
| 2007 | Budapest | Advances in Multilingual and Multimodal Information Retrieval: Eighth Workshop of the Cross—Language Evaluation Forum (CLEF 2007). Revised Selected Papers | Carol Ann Peters and Jijkoun, V. and Mandl, T. and Henning Müller and Oard, D. W. and Peñas, A. and Petras, V. and Santos, D. | ||
| 2008 | Århus | Evaluating Systems for Multilingual and Multimodal Information Access: Ninth Workshop of the Cross—Language Evaluation Forum (CLEF 2008). Revised Selected Papers | Carol Ann Peters and Deselaers, T. and Nicola Ferro and Julio Gonzalo and Gareth J F Jones and Kurimo, M. and Mandl, T. and Peñas, A. | ||
| 2009 | Corfu | Multilingual Information Access Evaluation Vol. I Text Retrieval Experiments—Tenth Workshop of the Cross—Language Evaluation Forum (CLEF 2009). Revised Selected Papers | Carol Ann Peters and Di Nunzio, G. M. and Kurimo, M. and Mandl, T. and Mostefa, D. and Peñas, A. and Roda, G. | ||
| 2010 | Padua | First conference | Multilingual and Multimodal Information Access Evaluation. Proceedings of the International Conference of the Cross-Language Evaluation Forum (CLEF 2010 | Maristella Agosti and Nicola Ferro and Carol Ann Peters and Maarten de Rijke and Alan Smeaton | |
| 2011 | Amsterdam | Multilingual and Multimodal Information Access Evaluation. Proceedings of the Second International Conference of the Cross-Language Evaluation Forum (CLEF 2011) | Forner, P. and Julio Gonzalo and Kekäläinen, J. and Lalmas, M. and Maarten de Rijke | ||
| 2012 | Rome | Information Access Evaluation. Multilinguality, Multimodality, and Visual Analytics. Proceedings of the Third International Conference of the CLEF Initiative (CLEF 2012) | Catarci, T. and Forner, P. and Hiemstra, D. and Peñas, A. and Santucci, G. | ||
| 2013 | Valencia | Information Access Evaluation meets Multilinguality, Multimodality, and Visualization. Proceedings of the Fourth International Conference of the CLEF Initiative (CLEF 2013) | Forner, P. and Henning Müller and Paredes, R. and Rosso, P. and Stein, B. | ||
| 2014 | Sheffield | Information Access Evaluation—Multilinguality, Multimodality, and Interaction. Proceedings of the Fifth International Conference of the CLEF Initiative (CLEF 2014) | Kanoulas, E. and Lupu, M. and Clough, P. and Sanderson, M. and Hall, M. and Hanbury, A. and Toms, E. | ||
| 2015 | Toulouse | Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Sixth International Conference of the CLEF Association (CLEF 2015) | Mothe, J. and Savoy, J. and Jaap Kamps and Pinel-Sauvagnat, K. and Gareth J F Jones and SanJuan, E. and Cappellato, L. and Nicola Ferro | ||
| 2016 | Evora | Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Seventh International Conference of the CLEF Association (CLEF 2016) | Fuhr, N. and Quaresma, P. and Goncalves, T. and Larsen, B. and Balog, K. and Macdonald, C. and Cappellato, L. and Nicola Ferro | ||
| 2017 | Dublin | Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Eighth International Conference of the CLEF Association (CLEF 2017) | Gareth J F Jones and Lawless, S. and Julio Gonzalo and Kelly, L. and Lorraine Goeuriot and Thomas Mandl and Cappellato, L. and Nicola Ferro | ||
| 2018 | Avignone | Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Ninth International Conference of the CLEF Association (CLEF 2018) | Patrice Bellot, Chiraz Trabelsi, Josiane Mothe, Fionn Murtagh, Jian-Yun Nie, Laure Soulier, Eric SanJuan, Linda Cappellato, and Nicola Ferro | ||
| 2019 | Lugano | Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Tenth International Conference of the CLEF Association (CLEF 2019) | Fabio Crestani, Martin Braschler, Jacques Savoy, Andreas Rauber, Henning Müller, David Losada, Gundula Heinatz, Linda Cappellato, and Nicola Ferro | ||
| 2020 | Thessaloniki (virtual due to COVID-19) | Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Eleventh International Conference of the CLEF Association (CLEF 2020) | Avi Arampatzis, Evangelos Kanoulas, Theodora Tsikrika, Stefanos Vrochidis, Hideo Joho, Christina Lioma, Carsten Eickhoff, Aurélie Névéol, A., Linda Cappellato, and Nicola Ferro | ||
| 2021 | Bucharest (virtual due to COVID-19) | Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Twelfth International Conference of the CLEF Association (CLEF 2021) | K. Selçuk Candan, Bogdan Ionescu, Lorraine Goeuriot, Birger Larsen, Henning Müller, Alexis Joly, Maria Maistro, Florina Piroi, Guglielmo Faggioli, and Nicola Ferro |