Interaction and reflection via 3D path shape qualities in a mediated constructive learning environment

ABSTRACT

The mixed-reality environment, or hybrid physical-digital space, is an emerging human-computer interaction paradigm with great potential to support constructive learning in everyday settings as a complement to traditional classroom methods. Key advantages over screen-based media and immersive (virtual-reality) environments include a) dynamic, multimodal feedback, which engages diverse learning styles through multiple modes of representation; b) affordance of unencumbered, full-body movement, which enables interactions to be physically embodied; and c) physical continuity with the classroom, which fosters informal collaborative and social interactions. For this potential to be realized, however, we must address significant challenges in interaction design. We must develop modes of interaction which are implicitly learnable, which afford full-body movement, and which are cognitively well adapted to large physical spaces. Furthermore, we must create a mechanism by which students can reflect on what they have "constructed" through interacting with the space, so that implicit learning can be leveraged in the interest of explicit understanding. To these ends, we have developed a novel interaction paradigm based around 3D path shape qualities: straight, curved, random, and stop, which describe motion of an illuminated object (glowball) guided by the participant through the space. We infer these qualities in real-time (online) and also offline using a robust Bayesian framework operating on a low-cost, non-intrusive video sensing apparatus. Online inference drives the interaction and offline segmentation the post-interaction display, our mechanism for reflection where segmentation results are mapped onto a physical trace of the participant.s motion. An informal study a) validates the implicit learnability of straight, curved, and stop mappings based on shape quality controls, and b) highlights the comparative advantage of the postinteraction display for all mappings, when subjects are asked to identify the actions responsible for specific target outcomes.

REFERENCES

• 1: S. Asseo and R. Ardila. Sensor independent target state estimator design and evaluation. In Proceedings of the National Aerospace and Electronics Conference (NAECON), 1982.
• 2: R. Azuma. A survey of augmented reality. Presence: Teleoperators and Virtual Environments, 6(4):355--385, 1997.
• 3: S. Benford et al. Hybrid physical-digital artifacts. Technical Report CID-185, Centre for User Oriented IT Design, Kungl Tekniska Hogskolan, Stockholm, Sweden, 2001.
• 4: David Birchfield , Thomas Ciufo , Gary Minyard, SMALLab: a mediated platform for education, ACM SIGGRAPH 2006 Educators program, July 30-August 03, 2006, Boston, Massachusetts [doi>10.1145/1179295.1179329]
• 5: Tina Blaine, The convergence of alternate controllers and musical interfaces in interactive entertainment, Proceedings of the 2005 conference on New interfaces for musical expression, May 26-28, 2005, Vancouver, Canada
• 6: J. Bransford, A. Brown, and R. Cocking. How People Learn: Brain, Mind, Experience, and School. National Academies Press, Washington DC, 2000.
• 7: Christoph Bregler, Learning and Recognizing Human Dynamics in Video Sequences, Proceedings of the 1997 Conference on Computer Vision and Pattern Recognition (CVPR '97), p.568, June 17-19, 1997
• 8: G. Burt. Media effectiveness, essentiality, and amount of study: A mathematical model. British Journal of Educational Technology, 37, 2006.
• 9: Carolina Cruz-Neira , Daniel J. Sandin , Thomas A. DeFanti , Robert V. Kenyon , John C. Hart, The CAVE: audio visual experience automatic virtual environment, Communications of the ACM, v.35 n.6, p.64-72, June 1992 [doi>10.1145/129888.129892]
• 10: D. Cunningham. Beyond educational psychology: Steps toward an educational semiotic. Educational Psychology Review, 4:165--194, 1992.
• 11: D. Cunningham, T. Duffy, and R. Knuth. Textbook of the Future in Hypertext: A Psychological Perspective. Ellis Horwood, C. McKnight, Ed. London, 1993.
• 12: P. Dourish. Where the Action Is: The Foundations of Embodied Cognition. MIT Press, Cambridge, MA, 2000.
• 13: H. Gardner. Frames of Mind: The Theory of Multiple Intelligences. Basic Books, New York, NY, 1983.
• 14: Zoubin Ghahramani , Sam T. Roweis, Learning nonlinear dynamical systems using an EM algorithm, Proceedings of the 1998 conference on Advances in neural information processing systems II, p.431-437, July 1999
• 15: C. Hu et al. Adaptive Kalman filtering for vehicle navigation. Journal of Global Positioning Systems, 2(1):42--47, 2003.
• 16: Jodi James , Todd Ingalls , Gang Qian , Loren Olsen , Daniel Whiteley , Siew Wong , Thanassis Rikakis, Movement-based interactive dance performance, Proceedings of the 14th annual ACM international conference on Multimedia, October 23-27, 2006, Santa Barbara, CA, USA [doi>10.1145/1180639.1180733]
• 17: H. Jin, G. Qian, and S. Rajko. Real-time multi-view 3d object tracking in cluttered scenes. In Proceedings of International Symposium on Visual Computing, Lake Tahoe, NV, 2006.
• 18: D. Jonassen. Computers and Mindtools for Schools: Engaging Critical Thinking. Prentice-Hall, Engelwood Cliffs, NJ, 1999.
• 19: S. Julier and J. Uhlmann. A new extension of the Kalman filter to nonlinear systems. In Proceedings of the 11th International Symposium on Aerospace/Defense Sensing, Simulation, and Controls, Orlando, FL, 1997.
• 20: T. Kailath, A. H. Sayed, and B. Hassibi. Linear Estimation. Prentice Hall, Upper Saddle River, New Jersey, 2000.
• 21: X. Li and V. P-Jilkov. Survey of maneuvering target tracking. part I: Dynamic models. IEEE Transactions on Aerospace and Electronic System, 39(4):1333--1364, 2003.
• 22: Pattie Maes , Trevor Darrell , Bruce Blumberg , Alex Pentland, The ALIVE system: wireless, full-body interaction with autonomous agents, Multimedia Systems, v.5 n.2, p.105-112, March 1997 [doi>10.1007/s005300050046]
• 23: K. Murphy. Filtering, smoothing, and the junction tree algorithm. http://citeseer.nj.nec.com/361819.html, 1998.
• 24: N. Pares et al. MEDIATE: An interactive multisensory environment for children with severe autism and no verbal communication. In Proceedings of International Workshop on Virtual Rehabilitation, Lausanne, Switzerland, 2004.
• 25: H. E. Rauch, F. Tung, and C. T. Striebel. Maximum likelihood estimates of linear dynamic systems. J. Amer. Inst. Aeronaut. Astronauts, 3(8):1445--50, 1965.
• 26: A. Siegel and S. White. The development of spatial representations of large-scale environments. In H. Reese, editor, Advances in Child Development and Behavior, pages 10--45. Academic Press, New York, NY, 1975.
• 27: H. Sundaram. Participating in our multi-sensory world. Technical Report AME-TR-2005-09, Arts, Media, and Engineering, Arizona State University, 2005.
• 28: Yinpeng Chen , Weiwei Xu , Richard Isaac Wallis , Hari Sundaram , Thanassis Rikakis , Todd Ingalls , Loren Olson , Jiping He, A real-time, multimodal biofeedback system for stroke patient rehabilitation, Proceedings of the 14th annual ACM international conference on Multimedia, October 23-27, 2006, Santa Barbara, CA, USA [doi>10.1145/1180639.1180745]
• 29: Harvey Thornburg , Julius O. Smith, III, Detection and modeling of transient audio signals with prior information, Stanford University, Stanford, CA, 2005
• 30: K. Tu, H. Thornburg, M. Fulmer, and A. Spanias. Tracking the path shape qualities of human motion. In 2007 International Conference on Acoustics, Speech, and Signal Processing (ICASSP--07), Honololu, HI, 2007.
• 31: L. Vygotsky. Mind in Society: Development of Higher Psychological Processes. Harvard University Press, Cambridge MA, 1978.
• 32: E. Wan, R. van der Merwe, and A. Nelson. Dual estimation and the unscented transformation. In S. Solla, T. Leen, and K.-R. Mueller, editors, Advances in Neural and Information Processing Systems 12, pages 666--672. MIT Press, 2000.
• 33: F. Wilcoxon. Individual comparisons by ranking methods. Biometrics Bulletin, 1(6):80--83, 1945.
• 34: O. Zoeter, A. Ypma, and T. Heskes. Improved unscented Kalman smoothing for stock volatility estimation. In Proceedings of 2004 IEEE Workshop on Machine Learning for Signal Processing, pages 143--152, Sao Luis, Brazil, 2004.

INDEX TERMS

Primary Classification:
  K. Computing Milieux
  K.3 COMPUTERS AND EDUCATION
      K.3.1 Computer Uses in Education
          Subjects: Computer-assisted instruction (CAI)

General Terms:
Algorithms, Design, Human Factors

Keywords:
hybrid physical-digital space, mediated constructive learning, motion quality analysis