Conference and Labs of the Evaluation Forum explained

Conference and Labs of the Evaluation Forum
Abbreviation:CLEF
Discipline:information retrieval
Publisher:Springer and CEUR Workshop Proceedings
History: (formerly known as Cross-Language Evaluation Forum from 2000 to 2009)
Frequency:annual

The Conference and Labs of the Evaluation Forum (formerly Cross-Language Evaluation Forum), or CLEF, is an organization promoting research in multilingual information access (currently focusing on European languages). Its specific functions are to maintain an underlying framework for testing information retrieval systems and to create repositories of data for researchers to use in developing comparable standards.[1] The organization holds a conference every September in Europe since a first constituting workshop in 2000. From 1997 to 1999, TREC, the similar evaluation conference organised annually in the US, included a track for the evaluation of Cross-Language IR for European languages. This track was coordinated jointly by NIST and by a group of European volunteers that grew over the years. At the end of 1999, a decision by some of the participants was made to transfer the activity to Europe and set it up independently. The aim was to expand coverage to a larger number of languages and to focus on a wider range of issues, including monolingual system evaluation for languages other than English. Over the years, CLEF has been supported by a number of various EU funded projects and initiatives.[2]

CLEF 2019 marked the 20th anniversary of the conference and it was celebrated by publishing a book[3] on the lessons learned in 20 years of evaluation activities.

Structure of CLEF

Before 2010,[4] CLEF was organised as a workshop co-located with the European Conference on Digital Libraries, consisting of a number of evaluation labs or tracks, similarly to TREC. In 2010, CLEF moved to become a self-sufficiently organised conference with evaluation labs, laboratory workshops, and a main conference track. In 2012, INEX, a workshop on retrieval and access to structured text, previously organised annually at Schloß Dagstuhl, merged with CLEF to become one of its evaluation labs.

Prior to each CLEF conference, participants in evaluation labs receive a set of challenge tasks. The tasks are designed to test various aspects of information retrieval systems and encourage their development. Groups of researchers propose and organize campaigns to satisfy those tasks and the results are used as benchmarks for the state of the art in the specific areas.,[5] [6]

In the beginning, CLEF focussed mainly on fairly typical information retrieval tasks, but has moved to more specific tasks. For example, the 2005 interactive image search task worked with illustrating non-fiction texts using images from Flickr[7] and the 2010 medical retrieval task focused on retrieval of computed tomography, MRI, and radiographic images.[8] In 2017, CLEF accommodated a number of tasks e.g. on identifying biological species from photographs or video clips, on stylistic analysis of authorship, and on health related information access.

List of CLEF workshops and conferences

YearLocationNoteProceedingsEditors
2000LisbonConstituting workshopCross-Language Information Retrieval and Evaluation: Workshop of Cross-Language Evaluation Forum (CLEF 2000)Carol Ann Peters
2001DarmstadtEvaluation of Cross-Language Information Retrieval Systems: Second Workshop of the Cross—Language Evaluation Forum (CLEF 2001) Revised PapersCarol Ann Peters and Martin Braschler and Julio Gonzalo and Martin Kluck
2002RomeAdvances in Cross-Language Information Retrieval: Third Workshop of the Cross—Language Evaluation Forum (CLEF 2002) Revised PapersCarol Ann Peters and Martin Braschler and Julio Gonzalo and Martin Kluck
2003TrondheimComparative Evaluation of Multilingual Information Access Systems: Fourth Workshop of the Cross—Language Evaluation Forum (CLEF 2003) Revised Selected PapersCarol Ann Peters and Martin Braschler and Julio Gonzalo and Martin Kluck
2004BathMultilingual Information Access for Text, Speech and Images: Fifth Workshop of the Cross—Language Evaluation Forum (CLEF 2004) Revised Selected PapersCarol Ann Peters and Paul Clough and Julio Gonzalo and Gareth J F Jones and Martin Kluck and Bernardo Magnini
2005ViennaAccessing Multilingual Information Repositories: Sixth Workshop of the Cross—Language Evaluation Forum (CLEF 2005). Revised Selected PapersCarol Ann Peters and Fred C Gey and Julio Gonzalo and Gareth J F Jones and Martin Kluck and Bernardo Magnini and Henning Müller and Maarten de Rijke
2006AlicanteEvaluation of Multilingual and Multi-modal Information Retrieval : Seventh Workshop of the Cross—Language Evaluation Forum (CLEF 2006). Revised Selected PapersCarol Ann Peters and Clough, P. and Fred C Gey and Jussi Karlgren and Bernardo Magnini and Doug Oard and Maarten de Rijke and M Stempfhuber
2007BudapestAdvances in Multilingual and Multimodal Information Retrieval: Eighth Workshop of the Cross—Language Evaluation Forum (CLEF 2007). Revised Selected PapersCarol Ann Peters and Jijkoun, V. and Mandl, T. and Henning Müller and Oard, D. W. and Peñas, A. and Petras, V. and Santos, D.
2008ÅrhusEvaluating Systems for Multilingual and Multimodal Information Access: Ninth Workshop of the Cross—Language Evaluation Forum (CLEF 2008). Revised Selected PapersCarol Ann Peters and Deselaers, T. and Nicola Ferro and Julio Gonzalo and Gareth J F Jones and Kurimo, M. and Mandl, T. and Peñas, A.
2009CorfuMultilingual Information Access Evaluation Vol. I Text Retrieval Experiments—Tenth Workshop of the Cross—Language Evaluation Forum (CLEF 2009). Revised Selected PapersCarol Ann Peters and Di Nunzio, G. M. and Kurimo, M. and Mandl, T. and Mostefa, D. and Peñas, A. and Roda, G.
2010PaduaFirst conferenceMultilingual and Multimodal Information Access Evaluation. Proceedings of the International Conference of the Cross-Language Evaluation Forum (CLEF 2010Maristella Agosti and Nicola Ferro and Carol Ann Peters and Maarten de Rijke and Alan Smeaton
2011AmsterdamMultilingual and Multimodal Information Access Evaluation. Proceedings of the Second International Conference of the Cross-Language Evaluation Forum (CLEF 2011)Forner, P. and Julio Gonzalo and Kekäläinen, J. and Lalmas, M. and Maarten de Rijke
2012RomeInformation Access Evaluation. Multilinguality, Multimodality, and Visual Analytics. Proceedings of the Third International Conference of the CLEF Initiative (CLEF 2012)Catarci, T. and Forner, P. and Hiemstra, D. and Peñas, A. and Santucci, G.
2013ValenciaInformation Access Evaluation meets Multilinguality, Multimodality, and Visualization. Proceedings of the Fourth International Conference of the CLEF Initiative (CLEF 2013)Forner, P. and Henning Müller and Paredes, R. and Rosso, P. and Stein, B.
2014SheffieldInformation Access Evaluation—Multilinguality, Multimodality, and Interaction. Proceedings of the Fifth International Conference of the CLEF Initiative (CLEF 2014)Kanoulas, E. and Lupu, M. and Clough, P. and Sanderson, M. and Hall, M. and Hanbury, A. and Toms, E.
2015ToulouseExperimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Sixth International Conference of the CLEF Association (CLEF 2015)Mothe, J. and Savoy, J. and Jaap Kamps and Pinel-Sauvagnat, K. and Gareth J F Jones and SanJuan, E. and Cappellato, L. and Nicola Ferro
2016EvoraExperimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Seventh International Conference of the CLEF Association (CLEF 2016)Fuhr, N. and Quaresma, P. and Goncalves, T. and Larsen, B. and Balog, K. and Macdonald, C. and Cappellato, L. and Nicola Ferro
2017DublinExperimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Eighth International Conference of the CLEF Association (CLEF 2017)Gareth J F Jones and Lawless, S. and Julio Gonzalo and Kelly, L. and Lorraine Goeuriot and Thomas Mandl and Cappellato, L. and Nicola Ferro
2018AvignoneExperimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Ninth International Conference of the CLEF Association (CLEF 2018)Patrice Bellot, Chiraz Trabelsi, Josiane Mothe, Fionn Murtagh, Jian-Yun Nie, Laure Soulier, Eric SanJuan, Linda Cappellato, and Nicola Ferro
2019LuganoExperimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Tenth International Conference of the CLEF Association (CLEF 2019)Fabio Crestani, Martin Braschler, Jacques Savoy, Andreas Rauber, Henning Müller, David Losada, Gundula Heinatz, Linda Cappellato, and Nicola Ferro
2020Thessaloniki (virtual due to COVID-19)Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Eleventh International Conference of the CLEF Association (CLEF 2020)Avi Arampatzis, Evangelos Kanoulas, Theodora Tsikrika, Stefanos Vrochidis, Hideo Joho, Christina Lioma, Carsten Eickhoff, Aurélie Névéol, A., Linda Cappellato, and Nicola Ferro
2021Bucharest (virtual due to COVID-19)Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Twelfth International Conference of the CLEF Association (CLEF 2021)K. Selçuk Candan, Bogdan Ionescu, Lorraine Goeuriot, Birger Larsen, Henning Müller, Alexis Joly, Maria Maistro, Florina Piroi, Guglielmo Faggioli, and Nicola Ferro

External links

Notes and References

  1. Carol . Peters. Martin . Braschler . Khalid . Choukri . Julio . Gonzalo . Michael . Kluck . The Future of Evaluation for Cross-Language Information Retrieval Systems . Second Workshop of the Cross-Language Evaluation Forum, CLEF 2001 . 10.1.1.109.7647 .
  2. Peters, Carol, and Martin Braschler. "European research letter: Cross‐language system evaluation: The CLEF campaigns." Journal of the Association for Information Science and Technology 52.12 (2001): 1067-1072.
  3. Book: Nicola . Ferro. Carol . Peters . Information Retrieval Evaluation in a Changing World - Lessons Learned from 20 Years of CLEF . 2019 . Springer International Publishing, Germany . volume 41 of The Information Retrieval Series . 10.1007/978-3-030-22948-1.
  4. http://clef.isti.cnr.it/ CLEF history before 2010
  5. Special Issue on CLEF . Information Retrieval . 7 . 1–2 . 2004 . 10.1023/B:INRT.0000009472.59294.f6 .
  6. Fredric C. Gey, Noriko Kando, and Carol Peters "Cross-Language Information Retrieval: the way ahead" in Information Processing & Managementvol. 41, no. 3, p.415-431 May 2005,
  7. Jussi. Karlgren. Jussi Karlgren. Julio. Gonzalo. Paul. Clough. iclef 2006 overview: Searching the flickr www photo-sharing repository. . Seventh Workshop of the Cross-Language Evaluation Forum CLEF 2006. 2006. .
  8. Web site: Mueller. Henning. Medical Retrieval Task. ImageCLEF - Cross-language image retrieval evaluations . 20 May 2010. 27 May 2010 .