Prakash: lighting aware motion capture using photosensing markers and multiplexed illuminators

ABSTRACT

In this paper, we present a high speed optical motion capture method that can measure three dimensional motion, orientation, and incident illumination at tagged points in a scene. We use tracking tags that work in natural lighting conditions and can be imperceptibly embedded in attire or other objects. Our system supports an unlimited number of tags in a scene, with each tag uniquely identified to eliminate marker reacquisition issues. Our tags also provide incident illumination data which can be used to match scene lighting when inserting synthetic elements. The technique is therefore ideal for on-set motion capture or real-time broadcasting of virtual sets.

Unlike previous methods that employ high speed cameras or scanning lasers, we capture the scene appearance using the simplest possible optical devices - a light-emitting diode (LED) with a passive binary mask used as the transmitter and a photosensor used as the receiver. We strategically place a set of optical transmitters to spatio-temporally encode the volume of interest. Photosensors attached to scene points demultiplex the coded optical signals from multiple transmitters, allowing us to compute not only receiver location and orientation but also their incident illumination and the reflectance of the surfaces to which the photosensors are attached. We use our untethered tag system, called Prakash, to demonstrate methods of adding special effects to captured videos that cannot be accomplished using pure vision techniques that rely on camera images.

REFERENCES

• 1: Azizoglu, M. Y., Salehi, J. A., and Li, Y. 1992. Optical CDMA via Temporal Codes. IEEE Transactions on Communications 40, 7, 1162--1170.
• 2: Basri, R., and Jacobs, D. 2001. Photometric Stereo with General, Unknown Lighting. IEEE CVPR 02, 374--379.
• 3: Bayindir, M., Sorin, F., Abouraddy, A. F., Viens, J., Hart, S. D., Joannopoulos, J. D., and Fink, Y. 2004. Metal-insulator-semiconductor Optoelectronic Fibres. Nature 431, 826--829.
• 4: Kiran S. Bhat , Christopher D. Twigg , Jessica K. Hodgins , Pradeep K. Khosla , Zoran Popović , Steven M. Seitz, Estimating cloth simulation parameters from video, Proceedings of the 2003 ACM SIGGRAPH/Eurographics symposium on Computer animation, July 26-27, 2003, San Diego, California
• 5: Bolas, M. T., McDowall, I. E., Hoberman, P., and Fisher, S. S., 2004. Snared Illumination.
• 6: Codamotion, 2007. Charnwood Dynamics Ltd. http://www.charndyn.com/.
• 7: Daniel Cotting , Martin Naef , Markus Gross , Henry Fuchs, Embedding Imperceptible Patterns into Projected Images for Simultaneous Acquisition and Display, Proceedings of the 3rd IEEE/ACM International Symposium on Mixed and Augmented Reality, p.100-109, November 02-05, 2004 [doi>10.1109/ISMAR.2004.30]
• 8: Hamamatsu Photonics, 2005. Si PIN Photodiode S6560.
• 9: Jeffrey Hightower , Gaetano Borriello, Location Systems for Ubiquitous Computing, Computer, v.34 n.8, p.57-66, August 2001 [doi>10.1109/2.940014]
• 10: Iltanen, M., Kosola, H., Palovuori, K., and Vanhala, J. 1998. Optical Positioning and Tracking System for a Head Mounted Display Based on Spread Spectrum Technology. In 2nd International Conference on Machine Automation (ICMA), 597--608.
• 11: Kang, S., and Tesar, D. 2004. Indoor GPS Metrology System with 3D Probe for Precision Applications. In Proceedings of ASME IMECE 2004 International Mechanical Engineering Congress and RD&D Expo.
• 12: Johnny C. Lee , Scott E. Hudson , Jay W. Summet , Paul H. Dietz, Moveable interactive projected displays using projector based tracking, Proceedings of the 18th annual ACM symposium on User interface software and technology, October 23-26, 2005, Seattle, WA, USA [doi>10.1145/1095034.1095045]
• 13: LightPointe Communications, 2007. Free Space Optics. http://www.freespaceoptics.org/.
• 14: Hongshen Ma , Joseph A. Paradiso, The FindIT Flashlight: Responsive Tagging Based on Optically Triggered Microprocessor Wakeup, Proceedings of the 4th international conference on Ubiquitous Computing, p.160-167, September 29-October 01, 2002, Göteborg, Sweden
• 15: Nobuyuki Matsushita , Daisuke Hihara , Teruyuki Ushiro , Shinichi Yoshimura , Jun Rekimoto , Yoshikazu Yamamoto, ID CAM: A Smart Camera for Scene Capturing and ID Recognition, Proceedings of the 2nd IEEE/ACM International Symposium on Mixed and Augmented Reality, p.227, October 07-10, 2003
• 16: Motion Analysis Corporation, 2006. Hawk-I Digital System.
• 17: Hideaki NII , Maki SUGIMOTO , Masahiko INAMI, Smart Light-Ultra High Speed Projector for Spatial Multiplexing Optical Transmission, Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Workshops, p.95, June 20-26, 2005 [doi>10.1109/CVPR.2005.528]
• 18: Hideaki Nii , Jay Summet , Yong Zhao , Jonathan Westhues , Paul Dietz , Shree Nayar , John Barnwell , Michael Noland , Vlad Branzoi , Erich Bruns , Masahiko Inami , Ramesh Raskar, Instant replay using high speed motion capture and projected overlay, ACM SIGGRAPH 2006 Sketches, July 30-August 03, 2006, Boston, Massachusetts [doi>10.1145/1179849.1179988]
• 19: Optotrak, 2007. NDI Optotrak Certus Spatial Measurement. http://www.ndigital.com/certus.php.
• 20: Karri T. Palovuori , Jukka J. Vanhala , Markku A. Kivikoski, Shadowtrack: A Novel Tracking System Based on Spreadspectrum Spatio-Temporal Illumination, Presence: Teleoperators and Virtual Environments, v.9 n.6, p.581-592, December 2000 [doi>10.1162/105474600300040394]
• 21: Patel, S. N., and Abowd, G. D. 2003. A 2-Way Laser-Assisted Selection Scheme for Handhelds in a Physical Environment. In Ubicomp, 200--207.
• 22: Phase Space Inc, 2007. Impulse Camera. http://www.phasespace.com.
• 23: PTI INC, 2006. VisualEyez VZ 4000.
• 24: Ramesh Raskar , Paul Beardsley , Jeroen van Baar , Yao Wang , Paul Dietz , Johnny Lee , Darren Leigh , Thomas Willwacher, RFIG lamps: interacting with a self-describing world via photosensing wireless tags and projectors, ACM Transactions on Graphics (TOG), v.23 n.3, August 2004
• 25: Ringwald, M. 2002. Spontaneous Interaction with Everyday Devices Using a PDA Workshop on Supporting Spontaneous Interaction in Ubiquitous Computing Settings. In UbiComp.
• 26: Robertson, B. 2006. Big moves. Computer Graphics World 29, 11 (Nov).
• 27: Silicon Light Machines, 2006. Gated Light Valve. http://www.siliconlight.com.
• 28: Smith, J., Sample, A., Powledge, P., Roy, S., and Mamishev, A. 2006. A wirelessly-powered platform for sensing and computation. In Ubicomp.
• 29: Sorensen, B. R., Donath, M., Yang, G.-B., and Starr, R. C. 1989. The Minnesota Scanner: A Prototype Sensor for Three-dimensional Tracking of Moving Body Segments. IEEE Transactions on Robotics and Automation 45, 4, 499--509.
• 30: ViconPeak, 2006. Camera MX 40. http://www.vicon.com/products/mx40.html.
• 31: Greg Welch , Gary Bishop, SCAAT: incremental tracking with incomplete information, Proceedings of the 24th annual conference on Computer graphics and interactive techniques, p.333-344, August 1997 [doi>10.1145/258734.258876]
• 32: Greg Welch , Eric Foxlin, Motion Tracking: No Silver Bullet, but a Respectable Arsenal, IEEE Computer Graphics and Applications, v.22 n.6, p.24-38, November 2002 [doi>10.1109/MCG.2002.1046626]

INDEX TERMS

Primary Classification:
  I. Computing Methodologies
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.8 Scene Analysis
          Subjects: Motion

Additional Classification:
  I. Computing Methodologies
  I.2 ARTIFICIAL INTELLIGENCE
        I.2.10 Vision and Scene Understanding
            Subjects: Motion; Intensity, color, photometry, and thresholding
  I.4 IMAGE PROCESSING AND COMPUTER VISION
      I.4.1 Digitization and Image Capture
          Subjects: Reflectance
      I.4.8 Scene Analysis
          Subjects: Tracking

General Terms:
Algorithms, Design