Special Requirements for Comparative Evaluation of  Web Search Engines
Wissam Tawileh, Thomas Mandl, Joachim Griesbaum
International Journal of Web Applications, Vol. 3, No. 2, 2011.
DOI: 
Link: http://www.dline.info/ijwa/fulltext/v3n2/1.pdf
Abstract:
Performance evaluation of classical information retrieval systems usually aims to assess the ability of these systems to find documents considered relevant to a certain search query based on a specific evaluation criteria. This approach, however, is not suitable to adequately evaluate some information retrieval applications such as web search engines. The web special characteristics make information retrieval tasks and the evaluation of search engines on the web face multiple challenges. Different web-specific, user-specific and language-specific requirements should be considered when designing and performing evaluation tests on operational web search engines. This paper discusses the special requirements for comprehensive comparative evaluation of different web search engines and highlights some languagespecific considerations for evaluation in Arabic language.