Skip to content

Toggle service links

Bias in System Evaluation
Julio Gonzalo Full Professor

This event took place on 24th October 2019 at 2:00pm (13:00 GMT)
Knowledge Media Institute, Berrill Building, The Open University, Milton Keynes, United Kingdom, MK7 6AA

"Bias" is a trending topic in the context of Artificial Intelligence and Data Science, and for a good reason: more and more decision making processes in our lives (such as getting a loan or being interviewed by a job) are mediated by Machine Learning systems; and both the research community and the society at large are increasingly aware that Machine Learning happens to be as prone to bias as human cognition. Most research on system bias currently focus on biases introduced by the algorithms and/or the data used by the algorithms to learn. But state-of-the-art systems are usually the result of a "natural selection" process where iterative evaluation, both inside and outside the lab, plays a key role. Consequently, biases in our evaluation methodologies may have a substantial impact on systems. In the talk I will discuss the many sources of bias in current evaluation practices, how they may impact research in the fields of Information Retrieval, Natural Language Processing and Recommender Systems, and what are the challenges to eliminate them.

The webcast was open to 300 users

(64 minutes)

Creative Commons Licence KMi logo