Does context matter in quality evaluation of mobile television?

ABSTRACT

Subjective quality evaluation is used to optimize the produced audiovisual quality from fundamental signal processing algorithms to consumer services. These studies typically follow the basic principles of controlled psychoperceptual experiments. However, when compromising compression and transmission parameters for consumer services, the ecological validity of conventional quality evaluation methods can be questioned. To tackle this, we firstly present a novel user-oriented quality evaluation method for mobile television in its usage contexts. Secondly, we present the results of an experiment conducted with 30 participants comparing acceptability and satisfaction of quality as well as goals of viewing in three mobile contexts and under four different residual transmission error rates, when the participants also performed simultaneous assessment tasks. Finally, we compare the results with a previous laboratory experiment. The studied error rates impacted negatively on all measured tasks with some contextual differences. Moreover, the evaluations were more favorable and less discriminate in the mobile contexts compared to the laboratory.

REFERENCES

• 1: Christer Carlsson , Pirkko Walden, Mobile TV - To Live or Die by Content, Proceedings of the 40th Annual Hawaii International Conference on System Sciences, p.51, January 03-06, 2007 [doi>10.1109/HICSS.2007.382]
• 2: Casey, B., Casey, N., Calvert, B., French, L., Justin, L. Television Studies -- The Key Concepts. Routledge. London, 2002.
• 3: Sherry Y. Chen , Gheorghita Ghinea , Robert D. Macredie, A cognitive approach to user perception of multimedia quality: An empirical investigation, International Journal of Human-Computer Studies, v.64 n.12, p.1200-1213, December, 2006 [doi>10.1016/j.ijhcs.2006.08.010]
• 4: Coolican, H. Research methods and statistics in psychology, 4th ed, London: J. W. Arrowsmith Ltd, 2004.
• 5: DVB-H Global Mobile TV: Services, Trials & Pilots. http://www.dvb-h.com/services.htm
• 6: ETSI, Digital Video Broadcasting (DVB): DVB specification for data broadcasting, ETSI standard, EN 301 192 V1.4.1, 2004.
• 7: ETSI, Digital Video Broadcasting (DVB): Transmission systems for handheld terminals, ETSI standard, EN 302 304 V1.1.1, 2004.
• 8: ETSI, Digital Video Broadcasting (DVB); Specification for the use of video and audio coding in DVB services delivered directly over IP, ETSI standard, ETSI TS 102 005 V1.2.0, 2005.
• 9: Finnpanel. www.finnpanel.fi, visited 20.5.2006.
• 10: Ghinea, G & Chen, S. Y. The impact of cognitive styles on perceptual distributed multimedia quality. British Journal of Educational Technology. Vol 34, 4 (2003). 393--406.
• 11: G. Ghinea , J. P. Thomas, QoS impact on user perception and understanding of multimedia video clips, Proceedings of the sixth ACM international conference on Multimedia, p.49-54, September 13-16, 1998, Bristol, United Kingdom [doi>10.1145/290747.290754]
• 12: Goldstein, E. B. Sensation and Perception. United States of America: Wadsworth, 2002.
• 13: Gulliver, G., Ghinea, G. Stars in Their Eyes, What Eye-Tracking Reveals About Multimedia Perceptual Quality. IEEE Transactions on Systems, Man, and Cybernetics, Part A: Systems and Humans. Vol. 34, 4(2004) 472--482
• 14: Stephen R. Gulliver , Gheorghita Ghinea, Defining user perception of distributed multimedia quality, ACM Transactions on Multimedia Computing, Communications, and Applications (TOMCCAP), v.2 n.4, p.241-257, November 2006 [doi>10.1145/1201730.1201731]
• 15: Hands, D. "Multimodal Quality Perception: The Effects of Attending to Content on Subjective Quality Ratings". Proceedings of IEEE 3rd Workshop on Multimedia Signal Processing, 1999 pp. 503--508, Copenhagen, Denmark 1999.
• 16: Hannuksela, M. M., Malamal Vadakital, V. K., Jumisko-Pyykkö, S. Comparison of Error Protection Methods for Audio-Video Broadcast over DVB-H, EURASIP Journal on Advances in Signal Processing, vol. 2007, 2007. doi:10.1155/2007/71801
• 17: ISO/IEC 14496--10:2003, Coding of Audiovisual Objects---Part 10: Advanced Video Coding," 2003, also ITU-T Recommendation H.264. Advanced video coding for generic audiovisual services. 2003.
• 18: ISO/IEC 14496-3: Information technology -- Coding of audio-visual objects - Part 3: Audio, including amendment 1: "Bandwidth Extension" and amendment 2: "Parametric Coding for High Quality Audio". 2003.
• 19: ISO 13407 Human-centred design processes for interactive systems. International Standard, the International Organization for Standardization. 1999.
• 20: ITU-R BT.500-11 Methodology for the subjective assessment of the quality of television pictures, International Telecommunications Union -- Radiocommunication sector, 2002.
• 21: ITU-T P.911 Recommendation P.911, Subjective audiovisual quality assessment methods for multimedia application, International Telecommunication Union -- Telecommunication sector, 1998.
• 22: Jumisko, S., Ilvonen, V., Väänänen-Vainio-Mattila, K. The Effect of TV Content in Subjective Assessment of Video Quality on Mobile Devices. Proc. Multimedia on Mobile Devices, IS&T/SPIE Symposium on Electronic Imaging, (2005), 243--254.
• 23: Jumisko-Pyykkö, S. "I would like to see the subtitles and the face or at least hear the voice": Effects of Picture ratio and Audio-video Bitrate Ratio on Perception of Quality in Mobile Television. Personalized and Mobile Digital TV Applications in Springer Multimedia Tools and Applications Series. 2007.
• 24: Satu Jumisko-Pyykkö , Jukka Häkkinen, Evaluation of subjective video quality of mobile devices, Proceedings of the 13th annual ACM international conference on Multimedia, November 06-11, 2005, Hilton, Singapore [doi>10.1145/1101149.1101270]
• 25: Jumisko-Pyykkö, S. Häkkinen, J., Nyman, G. Experienced Quality Factors - Qualitative Evaluation Approach to Audiovisual Quality. Proc IST/SPIE conference Electronic Imaging, Multimedia on Mobile Devices 2007
• 26: Jumisko-Pyykkö, S., Vadakital, V., Liinasuo, M., Hannuksela M. M., 2006. Acceptance of Audiovisual Quality in Erroneous Television Sequences over a DVB-H Channel. Proc. VPQM, USA, January 2006.
• 27: Satu Jumisko-Pyykkö , Vinod Kumar M. V. , Jari Korhonen, Unacceptability of instantaneous errors in mobile television: from annoying audio to video, Proceedings of the 8th conference on Human-computer interaction with mobile devices and services, September 12-15, 2006, Helsinki, Finland [doi>10.1145/1152215.1152217]
• 28: Kaikkonen, A., Kekäläinen, A., Cankar, M., Kallio, T., Kankainen, A. Usability Testing of Mobile Applications: A Comparison between Laboratory and Field Testing. Journal of Usability Studies, Vol. 1 (1), (2005)4--17
• 29: Kjeldskov, J., Stage, J., New techniques for usability evaluation of mobile systems, International Journal of Human-Computer Studies, Vol 60, 5--6 (2004) 599--620.
• 30: Knoche, H. O., McCarthy, J. D.: Good news for mobile TV, Wireless world research forum, Proc. WWRF 14, 2005.
• 31: Hendrik Knoche , John D. McCarthy , M. Angela Sasse, Can small be beautiful?: assessing image resolution requirements for mobile TV, Proceedings of the 13th annual ACM international conference on Multimedia, November 06-11, 2005, Hilton, Singapore [doi>10.1145/1101149.1101331]
• 32: Knoche, H., Sasse, M. A. Breaking the news on mobile TV: user requirements of a popular mobile content. Proc IS&T/SPIE Symposium on Electronic Imaging, 2006
• 33: Hendrik Knoche , John D. Mccarthy , M. Angela Sasse, How low can you go? The effect of low resolutions on shot types in mobile TV, Multimedia Tools and Applications, v.36 n.1-2, p.145-166, January 2008 [doi>10.1007/s11042-006-0076-5]
• 34: Lang, A. The limited capacity model of mediated message processing. Journal of Communication, (2000) 50, 46--70.
• 35: John D. McCarthy , M. Angela Sasse , Dimitrios Miras, Sharp or smooth?: comparing the effects of quantization vs. frame rate for streamed video, Proceedings of the SIGCHI conference on Human factors in computing systems, p.535-542, April 24-29, 2004, Vienna, Austria [doi>10.1145/985692.985760]
• 36: Kenton O'Hara , April Slayden Mitchell , Alex Vorbau, Consuming video on mobile devices, Proceedings of the SIGCHI conference on Human factors in computing systems, April 28-May 03, 2007, San Jose, California, USA [doi>10.1145/1240624.1240754]
• 37: Oksman, V., Noppari,., Tammela, A., Mäkinen, M., Ollikainen, V., News in mobiles. Comparing text, audio and video. VTT 2007. http://www.vtt.fi/inf/pdf/tiedotteet/2007/T2375.pdf
• 38: Reiter, U. & Jumisko-Pyykköö S. Watch, Press and Catch - Impact of Divided Attention on Requirements of Audiovisual Quality. Proc. 12th HCI Int 2007.
• 39: Repo, P. Hyvonen, K. Pantzar, M. Timonen, P. Inventing Use for a Novel Mobile Service. International Journal of Technology and Human Interaction, vol. 2, 2 (2006) 49--62.
• 40: Rogers E. M. Diffusion of Innovations, 5th ed., New York: Free Press, 2003.
• 41: Södergård C. (ed.). Mobile television -- technology and user experiences, Report on the Mobile --TV Project. Espoo: VTT Publications 506, 2003.
• 42: Sakari Tamminen , Antti Oulasvirta , Kalle Toiskallio , Anu Kankainen, Understanding mobile contexts, Personal and Ubiquitous Computing, v.8 n.2, p.135-143, May 2004 [doi>10.1007/s00779-004-0263-1]
• 43: Anne Watson , M. Angela Sasse, Measuring perceived quality of speech and video in multimedia conferencing applications, Proceedings of the sixth ACM international conference on Multimedia, p.55-60, September 13-16, 1998, Bristol, United Kingdom [doi>10.1145/290747.290755]
• 44: Winkler, S., Faller, C. Perceived audiovisual quality of low-bitrate multimedia content. IEEE Transactions on Multimedia, vol. 8, no. 5, (2006) 973--980.
• 45: Wynekoop, J. L., Russo, N., L., Studying system development methodologies: an examination of research methods. Information Systems Journal 7, 1(1997)47--65.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems

General Terms:
Experimentation, Human Factors

Keywords:
context, ecological validity, evaluation, mobile TV, mobile television, subjective quality, transmission quality