User-centric content freshness metrics for search engines

ABSTRACT

In order to return relevant search results, a search engine must keep its local repository synchronized to the Web, but it is usually impossible to attain perfect freshness. Hence, it is vital for a production search engine continually to monitor and improve repository freshness. Most previous freshness metrics, formulated in the context of developing better synchronization policies, focused on the web crawler while ignoring other parts of a search engine. But, the freshness of documents in a web crawler does not necessarily translate directly into the freshness of search results as seen by users. We propose metrics for measuring freshness from a user's perspective, which take into account the latency between when documents are crawled and when they are viewed by users, as well as the variation in user click and view frequency among different documents. We also describe a practical implementation of these metrics that were used in a production search engine.

REFERENCES

• 1: Junghoo Cho , Hector Garcia-Molina, Synchronizing a database to improve freshness, ACM SIGMOD Record, v.29 n.2, p.117-128, June 2000
• 2: Jianchao Han , Nick Cercone , Xiaohua Hu, A Weighted Freshness Metric for Maintaining Search Engine Local Repository, Proceedings of the 2004 IEEE/WIC/ACM International Conference on Web Intelligence, p.677-680, September 20-24, 2004 [doi>10.1109/WI.2004.17]
• 3: Qingzhao Tan , Prasenjit Mitra , C. Lee Giles, Designing clustering-based web crawling policies for search engine crawlers, Proceedings of the sixteenth ACM conference on Conference on information and knowledge management, November 06-10, 2007, Lisbon, Portugal [doi>10.1145/1321440.1321516]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Search process

General Terms:
Experimentation, Measurement

Keywords:
crawling, document age, freshness, latency, metrics, monitoring, search engine