M/ORIS: a medical/operating room interaction system

ABSTRACT

We propose an architecture for a real-time multimodal system, which provides non-contact, adaptive user interfacing for Computer-Assisted Surgery (CAS). The system, called M/ORIS (for Medical/Operating Room Interaction System) combines gesture interpretation as an explicit interaction modality with continuous, real-time monitoring of the surgical activity in order to automatically address the surgeon's needs. Such a system will help reduce a surgeon's workload and operation time. This paper focuses on the proposed activity monitoring aspect of M/ORIS. We analyze the issues of Human-Computer Interaction in an OR based on real-world case studies. We then describe how we intend to address these issues by combining a surgical procedure description with parameters gathered from vision-based surgeon tracking and other OR sensors (e.g. tool trackers). We called this approach Scenario-based Activity Monitoring (SAM). We finally present preliminary results, including a non-contact mouse interface for surgical navigation systems.

REFERENCES

• 1: M. Bichsel and A. P. Pentland, "Automatic Interpretation of Human Head Movements", JICAI Workshop on Looking At People, Chambery France, August 30, 1993.
• 2: M. Billinghurst, J. Savage, P. Oppenheimer, C. Edmond, "The Expert Surgical Assistant: An Intelligent Virtual Environment with Multimodal Input", Medicine Meets Virtual Reality IV, Amsterdam, 1996.
• 3: Aaron F. Bobick , James W. Davis, The Recognition of Human Movement Using Temporal Templates, IEEE Transactions on Pattern Analysis and Machine Intelligence, v.23 n.3, p.257-267, March 2001 [doi>10.1109/34.910878]
• 4: R. Bodor, B. Jackson and N. Papanikolopoulos, "Vision-Based Human Tracking and Activity Recognition," Proc. of the 11th Mediterranean Conf. on Control and Automation, June 18--20, 2003.
• 5: M. Caversaccio, R. Baechler, K. Laedrach, G. Schroth, L.-P. Nolte and R. Haeusler, "The Bernese Frameless Optical Computer Aided Surgery System", Computer Aided Surgery, no. 4, pp. 328--334, 1999.
• 6: R. Chellappa, N. Vaswani, and A. K. R. Chowdhury, "Activity Modeling and Recognition using Shape Theory", Behavior Representation in Modeling and Simulation, 2003.
• 7: K. Cleary, "Workshop Report", Workshop on Technical Requirements for Image-Guided Spine Procedures, April 17-20, Georgetown, USA, 1999.
• 8: Collins, R. T., Lipton, A. J., Fujiyoshi, H., Kanade, T., "Algorithms for Cooperative Multisensor Surveillance", Proc. of IEEE, Vol.89, pp1456--1477, Oct. 2001.
• 9: James W. Davis , Ambrish Tyagi, A Reliable-Inference Framework for Recognition of Human Actions, Proceedings of the IEEE Conference on Advanced Video and Signal Based Surveillance, p.169, July 21-22, 2003
• 10: J. Ellsmere, J. Stoll, D. Rattner, D. Brooks, R. Kane, W. Wells, R. Kikinis and K. Vosburgh, "A Navigation System for Augmenting Laparoscopic Ultrasound", R.E. Ellis and T.M. Peters (Eds.) MICCAI 2003, LNCS 2879, pp. 184--191, 2003.
• 11: Tom Fawcett , Foster Provost, Activity monitoring: noticing interesting changes in behavior, Proceedings of the fifth ACM SIGKDD international conference on Knowledge discovery and data mining, p.53-62, August 15-18, 1999, San Diego, California, United States [doi>10.1145/312129.312195]
• 12: D. M. Gavrila, The visual analysis of human movement: a survey, Computer Vision and Image Understanding, v.73 n.1, p.82-98, Jan. 1999 [doi>10.1006/cviu.1998.0716]
• 13: C. Graetzel, T. Fong, S. Grange, and C. Baur, "A Non-Contact Mouse for Surgeon-Computer Interaction", Technology and Health Care, IOS Press, (in press) 2004.
• 14: S. Grange, E. Casanova, T. Fong, and C. Baur, "Vision-based Sensor Fusion for Human-Computer Interaction", International Conference on Intelligent Robots and Systems, IEEE/RSJ, Lausanne, Switzerland, October 2002.
• 15: M. A. Grasso, "The Long-Term Adoption of Speech Recognition in Medical Applications", Proc. of the 16th IEEE Symposium on Computer-Based Medical Systems (CBMS 2003), pp: 257--262, 2003.
• 16: Rosco Hill , James Begole, Activity rhythm detection and modeling, CHI '03 extended abstracts on Human factors in computing systems, April 05-10, 2003, Ft. Lauderdale, Florida, USA [doi>10.1145/765891.765989]
• 17: Nebojsa Jojic , Thomas Huang , Barry Brumitt , Brian Meyers , Steve Harris, Detection and Estimation of Pointing Gestures in Dense Disparity Maps, Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition 2000, p.468, March 26-30, 2000
• 18: Victor Kaptelinin , Bonnie A. Nardi, Activity theory: basic concepts and applications, CHI '97 extended abstracts on Human factors in computing systems: looking to the future, March 22-27, 1997, Atlanta, Georgia [doi>10.1145/1120212.1120321]
• 19: Rajesh Kumar , Gregory D. Hager , Aaron C. Barnes , Patrick Jensen , Russell H. Taylor, An Augmentation System for Fine Manipulation, Proceedings of the Third International Conference on Medical Image Computing and Computer-Assisted Intervention, p.956-965, October 11-14, 2000
• 20: S. Marcel, "Gestures for Multi-Modal Interfaces: A Review", IDIAP Technical Report IDIAP-RR 02-34, September 2002.
• 21: S. Martelli, S. Bignozzi, M. Bontempi, S. Zaffagnini and L. Garcia, "Comparison of an Optical and a Mechanical Navigation System", R.E. Ellis and T.M. Peters (Eds.): MICCAI 2003, LNCS 2879, pp. 303--310, 2003.
• 22: I. Mikic , K. Huang , M. Trivedi, Activity monitoring and summarization for an intelligent meeting room, Proceedings of the Workshop on Human Motion (HUMO'00), p.107, December 07-08, 2000
• 23: Thomas B. Moeslund , Erik Granum, A survey of computer vision-based human motion capture, Computer Vision and Image Understanding, v.81 n.3, p.231-268, March 2001 [doi>10.1006/cviu.2000.0897]
• 24: A. B. Mor, J. E. Moody, D. Davidson, R. S. Labarca, B. Jaramaz, and A. M. Digioia III, "A Framework for Determining Component and Overall Accuracy for Computer Assisted Surgery Systems", R.E. Ellis and T.M. Peters (Eds.), MICCAI 2003, LNCS 2879, pp. 985--986, 2003.
• 25: Bonnie A. Nardi, Context and consciousness: activity theory and human-computer interaction, Massachusetts Institute of Technology, Cambridge, MA, 1995
• 26: A. Nishikawa, S. Asano, R. Fujita, S. Yamaguchi, T. Yohda, F. Miyazaki, M. Sekimoto, M. Yasui, Y. Miyake, S. Takiguchi and M. Monden, "Selective Use of Face Gesture Interface and Instrument Tracking System for Control of a Robotic Laparoscope Positioner", R.E. Ellis and T.M. Peters (Eds.) MICCAI 2003, LNCS 2879, pp. 973--974, 2003.
• 27: A. Nishikawa, T. Hosoi, K. Koara, D. Negoro, A. Hikita, S. Asano, H. Kakutani, F. Miyazaki, M. Sekimoto, M. Yasui, Y. Miyake, S. Takiguchi, and M. Monden, "FAce MOUSe: A Novel Human-Machine Interface for Controlling the Position of a Laparoscope", IEEE Transactions on Robotics and Automation (Special Issue on Medical Robotics), 2003.
• 28: Nuria Oliver , Barbara Rosario , Alex Pentland, A Bayesian Computer Vision System for Modeling Human Interaction, Proceedings of the First International Conference on Computer Vision Systems, p.255-272, January 13-15, 1999
• 29: P. Peixoto, J. Batista, and H. Araujo, "Real-Time Human Activity Monitoring Exploring Multiple Vision Sensors", Robotics and Autonomous Systems, 2000.
• 30: M. Porta, "Vision-based user interfaces: methods and applications", International Journal of Human-Computer Studies, Vol. 57, pp. 27--73, Elsevier Science - Academic Press, 2002.
• 31: D. H. Rice, S. D. Schaefer, "Endoscopic Paranasal Sinus Surgery", pp. 159--186, Raven Press, New York, 1993.
• 32: J. Rickel and W. L. Johnson, "Animated Agents for Procedural Training in Virtual Reality, Perception, Cognition, and Motor Control", Applied Artificial Intelligence, Volume 13, pp. 343--382, 1999.
• 33: D. Sawyer, K. J. Aziz, C. L. Backinger, E. T. Beers, A. Lowery and S. M. Sykes, "An Introduction to Human Factors in Medical Devices", US Department of Health and Human Services, Public Health Service, Food and Drug Administration, Center for Devices and Radiological Health; 1996.
• 34: M. Turk, "Perceptive Media: Machine Perception and Human Computer Interaction", Chinese Journal of Computers, Vol. 23, No. 12, pp. 1235--1244, 2000.
• 35: F. Vogt, S. Krüger, H. Niemann and C. Schick, "A System for Real-Time Endoscopic Image Enhancement", R.E. Ellis and T.M. Peters (Eds.), MICCAI 2003, LNCS 2879, pp. 356--363, 2003.
• 36: E. Wahlstrom, O. Masoud, and N. Papanikolopoulos, "Vision-based Methods for Driver Monitoring", Proc. IEEE 6th International Conference on Intelligent Transportation Systems, pp. 903-908, Shanghai, China, Oct. 2003.
• 37: M. E. Wicklund, "Making Medical Device Interfaces More User-Friendly", Medical Device and Diagnostic Industry, 1998.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES
      H.1.2 User/Machine Systems
          Subjects: Human information processing

General Terms:
Algorithms, Design, Experimentation, Human Factors, Theory

Keywords:
CAS, HCI, medical user interfaces, multimodal interaction