"Bias" is a trending topic in the context of Artificial Intelligence and Data Science, and for a good reason: more and more decision making processes in our lives (such as getting a loan or being interviewed by a job) are mediated by Machine Learning systems; and both the research community and the society at large are increasingly aware that Machine Learning happens to be as prone to bias as human cognition. Most research on system bias currently focus on biases introduced by the algorithms and/or the data used by the algorithms to learn. But state-of-the-art systems are usually the result of a "natural selection" process where iterative evaluation, both inside and outside the lab, plays a key role. Consequently, biases in our evaluation methodologies may have a substantial impact on systems. In the talk I will discuss the many sources of bias in current evaluation practices, how they may impact research in the fields of Information Retrieval, Natural Language Processing and Recommender Systems, and what are the challenges to eliminate them.