Evaluating Information Retrieval and Access Tasks

This open access book summarizes the first two decades of the NII Testbeds and Community for Information access Research (NTCIR). NTCIR is a series of evaluation forums run by a global team of researchers and hosted by the National Institute of Informatics (NII), Japan. The book is unique in that it discusses not just what was done at NTCIR, but also how it was done and the impact it has achieved. For example, in some chapters the reader sees the early seeds of what eventually grew to be the search engines that provide access to content on the World Wide Web, today’s smartphones that can tailor what they show to the needs of their owners, and the smart speakers that enrich our lives at home and on the move. We also get glimpses into how new search engines can be built for mathematical formulae, or for the digital record of a lived human life.

Key to the success of the NTCIR endeavor was early recognition that information access research is an empirical discipline and that evaluation therefore lay at the core of the enterprise. Evaluation is thus at the heart of each chapter in this book. They show, for example, how the recognition that some documents are more important than others has shaped thinking about evaluation design. The thirty-three contributors to this volume speak for the many hundreds of researchers from dozens of countries around the world who together shaped NTCIR as organizers and participants.

This book is suitable for researchers, practitioners, and students—anyone who wants to learn about past and present evaluation efforts in information retrieval, information access, and natural language processing, as well as those who want to participate in an evaluation task or even to design and organize one.

Tetsuya Sakai is a professor in the Department of Computer Science and Engineering, Waseda University, Japan. He is also a visiting professor at the National Institute of Informatics. He joined Toshiba in 1993 and obtained a Ph.D. from Waseda in 2000. In 2000 and 2001, he was supervised by the late Karen Sparck Jones at the Computer Laboratory, University of Cambridge, as a visiting researcher. In 2007, he joined NewsWatch, Inc. In 2009, he joined Microsoft Research Asia and the Waseda faculty in 2013. He was an associate dean (IT Strategies Division) there from 2015 to 2017, and department head from 2017 to 2019. He is the editor-in-chief of the Information Retrieval Journal (Springer) and an associate editor of ACM Transactions on Information Systems (TOIS). He is a distinguished member of the ACM, and the current vice chair of the ACM SIGIR (July 2019 – June 2022). 

Douglas W. Oard is a professor in the College of Information Studies (Maryland’s iSchool) and the University of Maryland Institute for Advanced Computer Studies (UMIACS). He holds affiliate appointments in Maryland’s Computer Science Department and Maryland’s Applied Mathematics, Statistics, and Scientific Computing program. He earned his Ph.D. in electrical engineering from the University of Maryland, College Park, and his B.A. and M.E.E. degrees in electrical engineering from Rice University. He is the current associate dean for faculty there. Previously, he was the associate dean for research in the College of Information Studies, and the director of the UMIACS Computational Linguistics and Information Processing (CLIP) laboratory.

Noriko Kando is a professor in the Information-Society Research Division of the National Institute of Informatics (NII), Tokyo, Japan, and also has been appointed as a professor in the Department of Informatics at the Graduate University of Advanced Studies (SOKENDAI), Japan. She initiated NTCIR in late 1997, an evaluation of information-access technologies such as information retrieval, summarization, question answering, and text mining, using various types of documents in East Asian languages and English, and has been a designer of various tasks and general co-chair of NTCIR. She received her Ph.D. from Keio University in library and information science in 1995 and has been teaching and conducting research at NII since 1994.