Improving meeting summarization by focusing on user needs: a task-oriented evaluation

ABSTRACT

Advances in multimedia technologies have enabled the creation of huge archives of audio-video recordings of meetings, and there is burgeoning interest in developing meeting browsers to help users better leverage these archives. A recent study has shown that extractive summaries provide a more efficient way of navigating meeting content than simply reading through the transcript and using the audio-video record, or navigating via keyword search (Murray, 2007). The extractive summary technique identifies informative dialogue acts to generate general purpose summaries. These summaries can still be lengthy. Recently, we have developed a decision-focused summarization system that presents only 1-2% of the recordings related to decision making. In this paper, we describe a task-based evaluation in which we compare the decision-focused summaries to the general purpose summaries. Our results indicate that the more focused summaries help users perform the decision debriefing task more effectively and improve perceived efficiency. In addition, this study also investigates the effect of automatic summaries and transcription on task effectiveness, report quality, and users' perceptions of task success.

REFERENCES

• 1: S. Banerjee, C. Rose, and A. I. Rudnicky. The necessity of a meeting recording and playback system, and the benefit of topic-level annotations to meeting browsing. In Proceedings of the International Conference on Human-Computer Interaction, 2005.
• 2: Regina Barzilay , Kathleen R. Mckeown, Information fusion for multidocument summarization: paraphrasing and generation, Columbia University, New York, NY, 2003
• 3: J. Carletta et al. The AMI meeting corpus: A pre-announcement. In S. Renals and S. Bengio, editors, Springer-Verlag Lecture Notes in Computer Science, volume 3869. Springer-Verlag, 2006.
• 4: A. H. Cremers, B. Hilhorst, and A. P. Vermeeren. What was discussed by whom, how, when and where? personalized browsing of annotated multimedia meeting recordings. In Proceedings of HCI, pages 1--10, 2005.
• 5: H. P. Edmundson, New Methods in Automatic Extracting, Journal of the ACM (JACM), v.16 n.2, p.264-285, April 1969 [doi>10.1145/321510.321519]
• 6: T. Hain, J. Dines, G. Garau, M. Karafiat, D. Moore, V. Wan, R. Ordelman, and S. Renals. Transcription of conference room meetings: An investigation. In Proceedings of Interspeech, 2005.
• 7: P. Hsueh and J. D. Moore. What decisions have you made?: Automatic decision detection in meeting conversations. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT), pages 25--32, Rochester, New York, April 2007.
• 8: K. Koumpis, S. Renals, and M. Niranjan. Extractive summarization of voicemail using lexical and prosodic feature subset selection. In Proceedings of Eurospeech, pages 2377--2380, 2001.
• 9: Julian Kupiec , Jan Pedersen , Francine Chen, A trainable document summarizer, Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval, p.68-73, July 09-13, 1995, Seattle, Washington, United States [doi>10.1145/215206.215333]
• 10: C. Y. Lin. Rouge: A package for automatic evaluation of summaries. In Proceedings of the Workshop on Text Summarization Branches Out of ACL 2004, 2004.
• 11: A. Lisowska, A. Popescu-Belis, and S. Armstrong. User query analysis for the specification and evaluation of a dialogue processing and retrieval system. In Proceedings of LREC, pages 993--996, 2004.
• 12: Inderjeet Mani , Eric Bloedorn, Machine learning of generic and user-focused summarization, Proceedings of the fifteenth national/tenth conference on Artificial intelligence/Innovative applications of artificial intelligence, p.820-826, July 1998, Madison, Wisconsin, United States
• 13: S. Maskey and J. Hirschberg. Comparing lexical, acoustic/prosodic, structural and discourse features for speech summarization. In Proceedings of Eurospeech, 2005.
• 14: Margot Mieskes , Christoph Müller , Michael Strube, Improving extractive dialogue summarization by utilizing human feedback, Proceedings of the 25th conference on Proceedings of the 25th IASTED International Multi-Conference: artificial intelligence and applications, p.627-632, February 12-14, 2007, Innsbruck, Austria
• 15: G. Murray. Automatic summarization of meeting. PhD thesis, University of Edinburgh, Scotland, UK, 2007.
• 16: G. Murray, S. Renals, and J. Carletta. Extractive summarization of meeting recordings. In Proceedings of Interspeech, 2005.
• 17: V. Pallotta, J. Niekrasz, and M. Purver. Collaborative and argumentative models of meeting discussions. In Workshop on Computational Models of Natural Arguments (CMNA) at the IJCAI, 2005.
• 18: V. Pallotta, V. Seretan, and M. Ailomaa. User requirements analysis for meeting information retrieval based on query elicitation. In Proceedings of ACL, 2007.
• 19: W. M. Post, A. H. Cremers, and O. B. Henkemans. A research environment for meeting behavior. In Proceedings of the 3rd Workshop on Social Intelligence Design, 2004.
• 20: R. Rienks, D. Heylen, and E. van der Weijden. Argument diagramming of meeting conversations. In Multimodal Multiparty Meeting Processing Workshop at the ICMI, 2005.
• 21: N. Romano Jr , J. Nunamaker Jr, Meeting Analysis: Findings from Research and Practice, Proceedings of the 34th Annual Hawaii International Conference on System Sciences ( HICSS-34)-Volume 1, p.1072, January 03-06, 2001
• 22: Simone Teufel , Marc Moens, Summarizing scientific articles: experiments with relevance and rhetorical status, Computational Linguistics, v.28 n.4, p.409-445, December 2002 [doi>10.1162/089120102762671936]
• 23: P. Wellner, M. Flynn, and M. Guillemot. Browsing recorded meetings with ferret. In Proceedings of 1st conference on Machine Learning and Multi-Modal Interactions (MLMI'04). Springer-Verlag, 2004.
• 24: S. Whittaker, R. Laban, and S. Tucker. Analysing meeting records: An ethnographic study and technological implications. In Proceedings of MLMI, 2005.
• 25: Klaus Zechner, Automatic summarization of open-domain multiparty dialogues in diverse genres, Computational Linguistics, v.28 n.4, p.447-485, December 2002 [doi>10.1162/089120102762671945]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Evaluation/methodology

Additional Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.2 User Interfaces (D.2.2, H.1.2, I.3.6)
          Subjects: Natural language

General Terms:
Design, Human Factors, Languages, Performance

Keywords:
automatic summarization, meeting browser, multimedia information retrieval, task-oriented evaluation, user study.