Evaluating Systems for Multilingual and Multimodal Information Access

The ninth campaign of the Cross-Language Evaluation Forum (CLEF) for European languages was held from January to September 2008. There were seven main eval- tion tracks in CLEF 2008 plus two pilot tasks. The aim, as usual, was to test the p- formance of a wide range of multilingual information access (MLIA) systems or s- tem components. This year, 100 groups, mainly but not only from academia, parti- pated in the campaign. Most of the groups were from Europe but there was also a good contingent from North America and Asia plus a few participants from South America and Africa. Full details regarding the design of the tracks, the methodologies used for evaluation, and the results obtained by the participants can be found in the different sections of these proceedings. The results of the CLEF 2008 campaign were presented at a two-and-a-half day workshop held in Aarhus, Denmark, September 17–19, and attended by 150 resear- ers and system developers. The annual workshop, held in conjunction with the European Conference on Digital Libraries, plays an important role by providing the opportunity for all the groups that have participated in the evaluation campaign to get together comparing approaches and exchanging ideas. The schedule of the workshop was divided between plenary track overviews, and parallel, poster and breakout sessions presenting this year’s experiments and discu- ing ideas for the future. There were several invited talks.

What Happened in CLEF 2008.- What Happened in CLEF 2008.- I: Multilingual Textual Document Retrieval (Ad Hoc).- CLEF 2008: Ad Hoc Track Overview.- TEL@CLEF.- Logistic Regression for Metadata: Cheshire Takes on Adhoc-TEL.- Query Expansion via Library Classification System.- Experiments on a Multinomial Language Model versus Lucene’s Off-the-Shelf Ranking Scheme and Rocchio Query Expansion (TEL@CLEF Monolingual Task).- WikiTranslate: Query Translation for Cross-Lingual Information Retrieval Using Only Wikipedia.- UFRGS@CLEF2008: Using Association Rules for Cross-Language Information Retrieval.- CLEF 2008 Ad-Hoc Track: Comparing and Combining Different IR Approaches.- Multi-language Models and Meta-dictionary Adaptation for Accessing Multilingual Digital Libraries.- Persian@CLEF.- Improving Persian Information Retrieval Systems Using Stemming and Part of Speech Tagging.- Fusion of Retrieval Models at CLEF 2008 Ad Hoc Persian Track.- Cross Language Experiments at Persian@CLEF 2008.- Robust-WSD.- Evaluating Word Sense Disambiguation Tools for Information Retrieval Task.- IXA at CLEF 2008 Robust-WSD Task: Using Word Sense Disambiguation for (Cross Lingual) Information Retrieval.- SENSE: SEmantic N-levels Search Engine at CLEF2008 Ad Hoc Robust-WSD Track.- IR-n in the CLEF Robust WSD Task 2008.- Query Clauses and Term Independence.- Analysis of Word Sense Disambiguation-Based Information Retrieval.- Crosslanguage Retrieval Based on Wikipedia Statistics.- Ad Hoc Mixed: TEL and Persian.- Sampling Precision to Depth 10000 at CLEF 2008.- JHU Ad Hoc Experiments at CLEF 2008.- UniNE at CLEF 2008: TEL, and Persian IR.- II: Mono- and Cross-Language Scientific Data Retrieval (Domain-Specific).- The Domain-Specific Track at CLEF 2008.- UniNE at Domain-Specific IR - CLEF 2008.- Back to Basics – Again – for Domain-Specific Retrieval.- Concept Models for Domain-Specific Search.- The Xtrieval Framework at CLEF 2008: Domain-Specific Track.- Using Wikipedia and Wiktionary in Domain-Specific Information Retrieval.- III: Interactive Cross-Language Retrieval (iCLEF).- Overview of iCLEF 2008: Search Log Analysis for Multilingual Image Retrieval.- Log Analysis of Multilingual Image Searches in Flickr.- Cross-Lingual Image Retrieval Interactions Based on a Game Competition.- A Study of Users’ Image Seeking Behaviour in FlickLing.- SICS at iCLEF 2008: User Confidence and Satisfaction Tentatively Inferred from iCLEF Logs.- IV: Multiple Language Question Answering (QA@CLEF).- Overview of the Clef 2008 Multilingual Question Answering Track.- Overview of the Answer Validation Exercise 2008.- Overview of QAST 2008.- Mono and Bilingual QA.- Assessing the Impact of Thesaurus-Based Expansion Techniques in QA-Centric IR.- Using AliQAn in Monolingual QA@CLEF 2008.- Priberam’s Question Answering System in QA@CLEF 2008.- IdSay: Question Answering for Portuguese.- Dublin City University at QA@CLEF 2008.- Using Answer Retrieval Patterns to Answer Portuguese Questions.- Ihardetsi: A Basque Question Answering System at QA@CLEF 2008.- Question Interpretation in QA@L2F.- UAIC Participation at QA@CLEF2008.- RACAI’s QA System at the Romanian–Romanian QA@CLEF2008 Main Task.- Combining Logic and Machine Learning for Answering Questions.- The MIRACLE Team at the CLEF 2008 Multilingual Question Answering Track.- Efficient Question Answering with Question Decomposition and Multiple Answer Streams.- DFKI-LT at QA@CLEF 2008.- Integrating Logic Forms and Anaphora Resolution in the AliQAn System.- Some Experiments in Question Answering with a Disambiguated Document Collection.- Answer Validation Exercise (AVE).- Answer Validation on English and Romanian Languages.- The Answer Validation System ProdicosAV Dedicated to French.- Studying the Influence of Semantic Constraints in AVE.- RAVE: A Fast Logic-Based Answer Validator.- Information Synthesis for Answer Validation.- Analyzing the Use of Non-overlap Features for Supervised Answer Validation.- Question Answering on Script Transcription (QAST).- The LIMSI Multilingual, Multitask QAst System.- IBQAst: A Question Answering System for Text Transcriptions.- Robust Question Answering for Speech Transcripts: UPC Experience in QAst 2008.- V: Cross-Language Retrieval in Image Collections (ImageCLEF).- Overview of the ImageCLEFphoto 2008 Photographic Retrieval Task.- Overview of the ImageCLEFmed 2008 Medical Image Retrieval Task.- Medical Image Annotation in ImageCLEF 2008.- The Visual Concept Detection Task in ImageCLEF 2008.- Overview of the WikipediaMM Task at ImageCLEF 2008.- ImageCLEFphoto.- Meiji University at ImageCLEF2008 Photo Retrieval Task: Evaluation of Image Retrieval Methods Integrating Different Media.- Building a Diversity Featured Search System by Fusing Existing Tools.- Some Results Using Different Approaches to Merge Visual and Text-Based Features in CLEF’08 Photo Collection.- MIRACLE-GSI at ImageCLEFphoto 2008: Different Strategies for Automatic Topic Expansion.- Using Visual Concepts and Fast Visual Diversity to Improve Image Retrieval.- A Comparative Study of Diversity Methods for Hybrid Text and Image Retrieval Approaches.- University of Jaén at ImagePhoto 2008: Filtering the Results with the Cluster Term.- Combining TEXT-MESS Systems at ImageCLEF 2008.- Image Retrieval by Inter-media Fusion and Pseudo-relevance Feedback.- Increasing Precision and Diversity in Photo Retrieval by Result Fusion.- Diversity in Image Retrieval: DCU at ImageCLEFPhoto 2008.- Visual Affinity Propagation Improves Sub-topics Diversity without Loss of Precision in Web Photo Retrieval.- Exploiting Term Co-occurrence for Enhancing Automated Image Annotation.- Enhancing Visual Concept Detection by a Novel Matrix Modular Scheme on SVM.- SZTAKI @ ImageCLEF 2008: Visual Feature Analysis in Segmented Images.- THESEUS Meets ImageCLEF: Combining Evaluation Strategies for a New Visual Concept Detection Task 2009.- Query Types and Visual Concept-Based Post-retrieval Clustering.- Annotation-Based Expansion and Late Fusion of Mixed Methods for Multimedia Image Retrieval.- Evaluation of Diversity-Focused Strategies for Multimedia Retrieval.- Clustering for Photo Retrieval at Image CLEF 2008.- ImageCLEFmed.- Methods for Combining Content-Based and Textual-Based Approaches in Medical Image Retrieval.- An SVM Confidence-Based Approach to Medical Image Annotation.- LIG at ImageCLEF 2008.- The MedGIFT Group at ImageCLEF 2008.- MIRACLE at ImageCLEFmed 2008: Semantic vs. Statistical Strategies for Topic Expansion.- Experiments in Calibration and Validation for Medical Content-Based Images Retrieval.- MIRACLE at ImageCLEFannot 2008: Nearest Neighbour Classification of Image Feature Vectors for Medical Image Annotation.- Query Expansion on Medical Image Retrieval: MeSH vs. UMLS.- Query and Document Expansion with Medical Subject Headings Terms at Medical Imageclef 2008.- Multimodal Medical Image Retrieval OHSU at ImageCLEF 2008.- Baseline Results for the ImageCLEF 2008 Medical Automatic Annotation Task in Comparison over the Years.- ImageCLEFWiki.- Evaluating the Impact of Image Names in Context-Based Image Retrieval.- Large-Scale Cross-Media Retrieval of WikipediaMM Images with Textual and Visual Query Expansion.- Conceptual Image Retrieval over a Large Scale Database.- UJM at ImageCLEFwiki 2008.- VI: Multilingual Web Track (WebCLEF).- Overview of WebCLEF 2008.- On the Evaluation of Snippet Selection for WebCLEF.- UNED at WebCLEF 2008: Applying High Restrictive Summarization, Low Restrictive Information Retrieval and Multilingual Techniques.- Retrieval of Snippets of Web Pages Converted to Plain Text. More Questions Than Answers.- VII: Cross-Language Geographical Retrieval (GeoCLEF).- GeoCLEF 2008: The CLEF 2008 Cross-Language Geographic Information Retrieval Track Overview.- GIR with Language Modeling and DFR Using Terrier.- Cheshire at GeoCLEF 2008: Text and Fusion Approaches for GIR.- Geographic and Textual Data Fusion in Forostar.- Query Expansion for Effective Geographic Information Retrieval.- Integrating Methods from IR and QA for Geographic Information Retrieval.- Using Query Reformulation and Keywords in the Geographic Information Retrieval Task.- Using GeoWordNet for Geographical Information Retrieval.- GeoTextMESS: Result Fusion with Fuzzy Borda Ranking in Geographical Information Retrieval.- A Ranking Approach Based on Example Texts for Geographic Information Retrieval.- Ontology-Based Query Construction for GeoCLEF.- Experiments with Geographic Evidence Extracted from Documents.- GikiP at GeoCLEF 2008: Joining GIR and QA Forces for Querying Wikipedia.- VIII: Cross-Language Video Retrieval (VideoCLEF).- Overview of VideoCLEF 2008: Automatic Generation of Topic-Based Feeds for Dual Language Audio-Visual Content.- MIRACLE at VideoCLEF 2008: Topic Identification and Keyframe Extraction in Dual Language Videos.- DCU at VideoClef 2008.- Using an Information Retrieval System for Video Classification.- VideoCLEF 2008: ASR Classification with Wikipedia Categories.- Metadata and Multilinguality in Video Classification.- IX: Multilingual Information Filtering (INFILE@CLEF).- Overview of CLEF 2008 INFILE Pilot Track.- Online Document Filtering Using Adaptive k-NN.- X: Morpho Challenge at CLEF 2008.- Overview of Morpho Challenge 2008.- ParaMor and Morpho Challenge 2008.- Allomorfessor: Towards Unsupervised Morpheme Analysis.- Using Unsupervised Paradigm Acquisition for Prefixes.- Morpho Challenge Evaluation by Information Retrieval Experiments.