EasyAlbum: an interactive photo annotation system based on face clustering and re-ranking

ABSTRACT

Digital photo management is becoming indispensable for the explosively growing family photo albums due to the rapid popularization of digital cameras and mobile phone cameras. In an effective photo management system photo annotation is the most challenging task. In this paper, we develop several innovative interaction techniques for semi-automatic photo annotation. Compared with traditional annotation systems, our approach provides the following new features: "cluster annotation" puts similar faces or photos with similar scene together, and enables user label them in one operation; "contextual re-ranking" boosts the labeling productivity by guessing the user intention; "ad hoc annotation" allows user label photos while they are browsing or searching, and improves system performance progressively through learning propagation. Our results show that these technologies provide a more user friendly interface for the annotation of person name, location, and event, and thus substantially improve the annotation performance especially for a large photo album.

REFERENCES

• 1: iView Media Pro. http://www.iview-multimedia.com.
• 2: ACDSee. http://www.acdsee.com.
• 3: Microsoft Diginal Image Suite. http://www.microsoft.com/products/imaging.
• 4: Picasa. http://picasa.google.com.
• 5: Photoshop Elements. http://www.adobe.com/products/photoshopelwin.
• 6: Flickr. http://www.flickr.com.
• 7: ESP Game. http://espgame.org.
• 8: Google Image Labeler. http://images.google.com/imagelabeler.
• 9: Riya. http://www.riya.com.
• 10: T. Ahonen, A. Hadid, and M. Pietikainen. Face recognition with local binary patterns. In Proc. the 8th European Conference on Computer Vision, 2004.
• 11: F. Bach and M. Jordan. Learning spectral clustering. In Proc. of Neural Info. Processing Systems, 2003.
• 12: L. Chen, B. Hu, L. Zhang, M. Li, and H. Zhang. Face annotation for family photo album management. International Journal of Image and Graphics (IJIG), Special Issue on Multimedia Data Storage and Management, 3, 2003.
• 13: Andreas Girgensohn , John Adcock , Lynn Wilcox, Leveraging face recognition technology to find and organize photos, Proceedings of the 6th ACM SIGMM international workshop on Multimedia information retrieval, October 15-16, 2004, New York, NY, USA [doi>10.1145/1026711.1026728]
• 14: Adrian Graham , Hector Garcia-Molina , Andreas Paepcke , Terry Winograd, Time as essence for photo browsing through personal digital libraries, Proceedings of the 2nd ACM/IEEE-CS joint conference on Digital libraries, July 14-18, 2002, Portland, Oregon, USA [doi>10.1145/544220.544301]
• 15: Allan Kuchinsky , Celine Pering , Michael L. Creech , Dennis Freeze , Bill Serra , Jacek Gwizdka, FotoFile: a consumer multimedia organization and retrieval system, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.496-503, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303143]
• 16: J. Platt. Autoalbum: Clustering digital photographs using probabalistic model merging, 2000.
• 17: J. C. Platt, M. Czerwinski, and B. A. Field. Phototoc: Automatic clustering for browsing personal photographs, 2002.
• 18: Kerry Rodden , Kenneth R. Wood, How do people manage their digital photographs?, Proceedings of the SIGCHI conference on Human factors in computing systems, April 05-10, 2003, Ft. Lauderdale, Florida, USA [doi>10.1145/642611.642682]
• 19: Direct Annotation: A Drag-and-Drop Strategy for Labeling Photos, Proceedings of the International Conference on Information Visualisation, p.88, July 19-21, 2000
• 20: B. Suh and B. Bederson. Semi-automatic image annotation using event and torso identification. Technical report, Computer Science Department, University of Maryland, College Park, MD, 2004.
• 21: P. Viola and M. Jones. Rapid object detection using a boosted cascade of simple features. In Proc. of IEEE Conf. on Computer Vision and Pattern Recognition, volume 1, pages 511--518, 2001.
• 22: Luis von Ahn , Laura Dabbish, Labeling images with a computer game, Proceedings of the SIGCHI conference on Human factors in computing systems, p.319-326, April 24-29, 2004, Vienna, Austria [doi>10.1145/985692.985733]
• 23: Xiaogang Wang , Xiaoou Tang, A Unified Framework for Subspace Face Recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.26 n.9, p.1222-1228, September 2004 [doi>10.1109/TPAMI.2004.57]
• 24: Xiaogang Wang , Xiaoou Tang, Random Sampling for Subspace Face Recognition, International Journal of Computer Vision, v.70 n.1, p.91-104, October 2006 [doi>10.1007/s11263-006-8098-z]
• 25: Lei Zhang , Longbin Chen , Mingjing Li , Hongjiang Zhang, Automated annotation of human faces in family albums, Proceedings of the eleventh ACM international conference on Multimedia, November 02-08, 2003, Berkeley, CA, USA [doi>10.1145/957013.957090]
• 26: Leizhong Zhang , Qiong Yang , Ta Bao , Dave Vronay , Xiaoou Tang, Imlooking: image-based face retrieval in online dating profile search, CHI '06 extended abstracts on Human factors in computing systems, April 22-27, 2006, Montréal, Québec, Canada [doi>10.1145/1125451.1125739]
• 27: W. Zhao , R. Chellappa , P. J. Phillips , A. Rosenfeld, Face recognition: A literature survey, ACM Computing Surveys (CSUR), v.35 n.4, p.399-458, December 2003 [doi>10.1145/954339.954342]
• 28: Y. Zhou, L. Gu, and H.-J. Zhang. Bayesian tangent shape model: Estimating shape and pose parameters via bayesian inference. In Proc. of IEEE Conf. on Computer Vision and Pattern Recognition, 2003.
• 29: Yi Zhou , Wei Zhang , Xiaoou Tang , Harry Shum, A Bayesian Mixture Model for Multi-View Face Alignment, Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Volume 2, p.741-746, June 20-26, 2005 [doi>10.1109/CVPR.2005.17]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Graphical user interfaces (GUI)

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Clustering; Information filtering
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Prototyping

General Terms:
Design, Human Factors

Keywords:
annotation, cluster annotation, face recognition, face tagging, photo tagging