The user in experimental computer systems research

ABSTRACT

Experimental computer systems research typically ignores the end-user, modeling him, if at all, in overly simple ways. We argue that this (1) results in inadequate performance evaluation of the systems, and (2) ignores opportunities. We summarize our experiences with (a) directly evaluating user satisfaction and (b) incorporating user feedback in different areas of client/server computing, and use our experiences to motivate principles for that domain. Specifically, we report on user studies to measure user satisfaction with resource borrowing and with different clock frequencies in desktop computing, the development and evaluation of user interfaces to integrate user feedback into scheduling and clock frequency decisions in this context, and results in predicting user action and system response in a remote display system. We also present initial results on extending our work to user control of scheduling and mapping of virtual machines in a virtualization-based distributed computing environment. We then generalize (a) and (b) as recommendations for incorporating the user into experimental computer systems research.

REFERENCES

• 1: Thomas E. Anderson , David E. Culler , David A. Patterson , and the NOW team, A Case for NOW (Networks of Workstations), IEEE Micro, v.15 n.1, p.54-64, February 1995 [doi>10.1109/40.342018]
• 2: Ricardo A. Baratto , Leonard N. Kim , Jason Nieh, THINC: a virtual display architecture for thin-client computing, Proceedings of the twentieth ACM symposium on Operating systems principles, October 23-26, 2005, Brighton, United Kingdom
• 3: Sumeer Bhola , Mustaque Ahamad, Workload modeling for highly interactive applications, Proceedings of the 1999 ACM SIGMETRICS international conference on Measurement and modeling of computer systems, p.210-211, May 01-04, 1999, Atlanta, Georgia, United States
• 4: Lawrence S. Brakmo , Sean W. O'Malley , Larry L. Peterson, TCP Vegas: new techniques for congestion detection and avoidance, Proceedings of the conference on Communications architectures, protocols and applications, p.24-35, August 31-September 02, 1994, London, United Kingdom
• 5: Stuart K. Card , Allen Newell , Thomas P. Moran, The Psychology of Human-Computer Interaction, Lawrence Erlbaum Associates, Inc., Mahwah, NJ, 2000
• 6: Ramesh Chandra , Nickolai Zeldovich , Constantine Sapuntzakis , Monica S. Lam, The collective: a cache-based system management architecture, Proceedings of the 2nd conference on Symposium on Networked Systems Design & Implementation, p.259-272, May 02-04, 2005
• 7: Chapman, M. rdesktop: A remote desktop protocol client for accessing windows nt terminal server. http://www.rdesktop.org.
• 8: Andrew Chien , Brad Calder , Stephen Elbert , Karan Bhatia, Entropia: architecture and performance of an enterprise desktop grid system, Journal of Parallel and Distributed Computing, v.63 n.5, p.597-610, May 2003 [doi>10.1016/S0743-7315(03)00006-6]
• 9: Curtin, M., and Dolske, J. A brute force search of DES keyspace. ;login: (May 1998).
• 10: Dinda, P. A. The statistical properties of host load. Scientific Programming 7, 3, 4 (1999). A version of this paper is also available as CMU Technical Report CMU-CS-TR-98-175. A much earlier version appears in LCR '98 and as CMU-CS-TR-98-143.
• 11: Dourish, P. Evolution in the adoption and use of collaborative technologies.
• 12: David W. Embley , George Nagy, Behavioral Aspects of Text Editors, ACM Computing Surveys (CSUR), v.13 n.1, p.33-70, March 1981 [doi>10.1145/356835.356838]
• 13: Yasuhiro Endo , Zheng Wang , J. Bradley Chen , Margo Seltzer, Using latency to evaluate interactive system performance, Proceedings of the second USENIX symposium on Operating systems design and implementation, p.185-199, October 29-November 01, 1996, Seattle, Washington, United States
• 14: Kevin Fall , Sally Floyd, Simulation-based comparisons of Tahoe, Reno and SACK TCP, ACM SIGCOMM Computer Communication Review, v.26 n.3, p.5-21, July 1996 [doi>10.1145/235160.235162]
• 15: Krisztián Flautner , Trevor Mudge, Vertigo: automatic performance-setting for Linux, ACM SIGOPS Operating Systems Review, v.36 n.SI, Winter 2002 [doi>10.1145/844128.844139]
• 16: James Frey , Todd Tannenbaum , Miron Livny , Ian Foster , Steven Tuecke, Condor-G: A Computation Management Agent for Multi-Institutional Grids, Proceedings of the 10th IEEE International Symposium on High Performance Distributed Computing, p.55, August 07-09, 2001
• 17: Alexandros V. Gerbessiotis , Leslie G. Valiant, Direct bulk-synchronous parallel algorithms, Journal of Parallel and Distributed Computing, v.22 n.2, p.251-267, Aug. 1994 [doi>10.1006/jpdc.1994.1085]
• 18: Google Corporation. Google compute. http://toolbar.google.com/dc/.
• 19: Gupta, A., Lin, B., and Dinda, P. A framework and toolkit for understanding user comfort with resource borrowing. Tech. Rep. NWU-CS-04-28, Department of Computer Science, Northwestern University, February 2004.
• 20: Ashish Gupta , Bin Lin , Peter A. Dinda, Measuring and Understanding User Comfort With Resource Borrowing, Proceedings of the 13th IEEE International Symposium on High Performance Distributed Computing, p.214-224, June 04-06, 2004 [doi>10.1109/HPDC.2004.21]
• 21: Huber, P. Robust Statistics. Wiley and Sons, 2003.
• 22: Klein, J. T. Computer response to user frustration. Master's thesis, Massachusetts Institute of Technology, 1999.
• 23: Akinori Komatsubara, Psychological Upper and Lower Limits of System Response Time and User's Preference on Skill Level, Proceedings of the Seventh International Conference on Human-Computer Interaction-Volume 1, p.829-832, August 24-29, 1997
• 24: Mike Kuniavsky, Observing the User Experience: A Practitioner's Guide to User Research (Morgan Kaufmann Series in Interactive Technologies) (The Morgan Kaufmann Series in Interactive Technologies), Morgan Kaufmann Publishers Inc., San Francisco, CA, 2003
• 25: Albert Lai , Jason Nieh, Limits of wide-area thin-client computing, Proceedings of the 2002 ACM SIGMETRICS international conference on Measurement and modeling of computer systems, June 15-19, 2002, Marina Del Rey, California
• 26: Larson, S. M., Snow, C. D., Shirts, M., and Pande, V. S. Folding@home and genome@home: Using distributed computing to tackle previously intractable problems in computational biology. In Computational Genomics, R. Grant, Ed. Horizon Press, 2002.
• 27: Lin, B. Human-directed adaptation. Thesis Proposal, Department of Electrical Engineering and Computer Science, Northwestern University, October 2005.
• 28: Bin Lin , Peter A. Dinda , Dong Lu, User-Driven Scheduling of Interactive Virtual Machines, Proceedings of the 5th IEEE/ACM International Workshop on Grid Computing, p.380-387, November 08-08, 2004 [doi>10.1109/GRID.2004.64]
• 29: Bin Lin , Peter A. Dinda, VSched: Mixing Batch And Interactive Virtual Machines Using Periodic Real-time Scheduling, Proceedings of the 2005 ACM/IEEE conference on Supercomputing, p.8, November 12-18, 2005 [doi>10.1109/SC.2005.80]
• 30: Lin, B., and Dinda, P. Putting the user in direct control of cpu scheduling. Tech. Rep. NWU-EECS-06-07, Department of Electrical Engineering and Computer Science, Northwestern University, July 2006.
• 31: Bin Lin , Peter A. Dinda, Towards Scheduling Virtual Machines Based On Direct User Input, Proceedings of the 2nd International Workshop on Virtualization Technology in Distributed Computing, p.6, November 17-17, 2006 [doi>10.1109/VTDC.2006.15]
• 32: Bin Lin , Arindam Mallik , Peter A. Dinda , Gokhan Memik , Robert P. Dick, Power reduction through measurement and modeling of users and CPUs: summary, Proceedings of the 2007 ACM SIGMETRICS international conference on Measurement and modeling of computer systems, June 12-16, 2007, San Diego, California, USA
• 33: Litzkow, M., Livny, M., and Mutka, M. W. Condor -- a hunter of idle workstations. In Proceedings of the 8th International Conference of Distributed Computing Systems (ICDCS '88) (June 1988), pp. 104--111.
• 34: C. L. Liu , James W. Layland, Scheduling Algorithms for Multiprogramming in a Hard-Real-Time Environment, Journal of the ACM (JACM), v.20 n.1, p.46-61, Jan. 1973 [doi>10.1145/321738.321743]
• 35: Liu, J. Real-time Systems. Prentice Hall, 2000.
• 36: Lorch, J. R., and Smith, A. J. Using User Interface Event Information in Dynamic Voltage Scaling Algorithms. In Technical Report UCB/CSD-02-1190, Computer Science Division, EECS, University of California at Berkeley, August (2002). citeseer. ist.psu.edu/lorch03using.html.
• 37: Allan MacLean , Kathleen Carter , Lennart Lövstrand , Thomas Moran, User-tailorable systems: pressing the issues with buttons, Proceedings of the SIGCHI conference on Human factors in computing systems: Empowering people, p.175-182, April 01-05, 1990, Seattle, Washington, United States [doi>10.1145/97243.97271]
• 38: Mallik, A., Lin, B., Dinda, P., Memik, G., and Dick, R. Process and user driven dynamic voltage and frequency scaling. Tech. Rep. NWU-EECS-06-11, Department of Electrical Engineering and Computer Science, Northwestern University, August 2006.
• 39: Arindam Mallik , Bin Lin , Gokhan Memik , Peter Dinda , Robert P. Dick, User-Driven Frequency Scaling, IEEE Computer Architecture Letters, v.5 n.2, p.16, July 2006 [doi>10.1109/L-CA.2006.16]
• 40: Microsoft. Remote desktop protocol (rdp) features and performance. Tech. rep., 2000.
• 41: Matt W. Mutka , Miron Livny, The available capacity of a privately owned workstation environment, Performance Evaluation, v.12 n.4, p.269-284, July 1991 [doi>10.1016/0166-5316(91)90005-N]
• 42: Proctor, R., and Van Zandt, T. Human Factors in Simple and Complex Systems. Allyn and Bacon, 1993.
• 43: Reynolds, C. J. The sensing and measurement of frustration with computers. Master's thesis, Massachusetts Institute of Technology Media Laboratory, 2001. http://www.media.mit.edu/~carsonr/pdf/sm_thesis.pdf.
• 44: Tristan Richardson , Quentin Stafford-Fraser , Kenneth R. Wood , Andy Hopper, Virtual Network Computing, IEEE Internet Computing, v.2 n.1, p.33-38, January 1998 [doi>10.1109/4236.656066]
• 45: Romano, P. Itu-t recommendation t.128 (application sharing). Tech. rep., ITU, March 1997.
• 46: Rossoff, S., and Dinda, P. Prospects for speculative remote display. Tech. Rep. NWU-EECS-06-08, Department of Electrical Engineering and Computer Science, Northwestern University, August 2006.
• 47: Ruth, P., McGachey, P., Jiang, X., and Xu, D. Viocluster: Virtualization for dynamic computational domains. In Proceedings of the IEEE International Conference on Cluster Computing (Cluster) (September 2005).
• 48: Satyanarayanan, M., Kozuch, M., Helfrich, C., and O'Hallaron, D. Towards seamless mobility on pervasive hardware. Pervasive and Mobile Computing 1, 2 (June 2005), 157--189.
• 49: Shoykhet, A., Lange, J., and Dinda, P. Virtuoso: A system for virtual machine marketplaces. Tech. Rep. NWU-CS-04-39, Department of Computer Science, Northwestern University, July 2004.
• 50: Sousa, J., Balan, R., Poladian, V., Garlan, D., and Satyanarayanan, M. Giving users the steering wheel for guiding resource-adaptive systems. Tech. Rep. CMU-CS-05-198, Department of Computer Science, Carnegie Mellon University, December 2005.
• 51: W. Stevens, TCP Slow Start, Congestion Avoidance, Fast Retransmit, and Fast Recovery Algorithms, RFC Editor, 1997
• 52: Stricker, L. J. The true deceiver. Psychological Bulletin, 68 (1967), 13--20.
• 53: Sullivan, W. T., Werthimer, D., Bowyer, S., Cobb, J., Gedye, D., and Anderson, D. A new major seti project based on project serendip data and 100,000 personal computers. In Proceedings of the Fifth International Conference on Bioastronomy (1997), C. Cosmovici, S. Bowyer, and D. Werthimer, Eds., no. 161 in IAU Colloquim, Editrice Compositori, Bologna, Italy.
• 54: Ananth Inamti Sundararaj , Peter A. Dinda, Automatic, run-time and dynamic adaptation of distributed applications executing in virtual environments, Northwestern University, Evanston, IL, 2006
• 55: A. I. Sundararaj , Ashish Gupta , P. A. Dinda, Increasing application performance in virtual environments through run-time inference and adaptation, Proceedings of the High Performance Distributed Computing, 2005. HPDC-14. Proceedings. 14th IEEE International Symposium, p.47-58, July 24-27, 2005 [doi>10.1109/HPDC.2005.1520935]
• 56: Sundararaj, A., Sanghi, M., Lange, J., and Dinda, P. An optimization problem in adaptive virtual environments. In Proceedings of the Seventh Workshop on Mathematical Performance Modeling and Analysis (MAMA) (June 2005).
• 57: Sundararaj, A., Sanghi, M., Lange, J., and Dinda, P. Hardness of approximation and greedy algorithms for the adaptation problem in virtual environments. In Proceedings of the 3rd IEEE International Conference on Autonomic Computing (ICAC) (2006).
• 58: Theocharous, G., Mannor, S., Shah, N., Gandhi, P., Kveton, B., Siddiqi, S., and Yu, C.-H. Machine learning for adaptive power management. Intel Technology Journal 10, 4 (Nov. 2006)
• 59: Zheng Wang , Jon Crowcroft, Eliminating periodic packet losses in the 4.3-Tahoe BSD TCP congestion control algorithm, ACM SIGCOMM Computer Communication Review, v.22 n.2, p.9-16, April 1992 [doi>10.1145/141800.141801]
• 60: Le Yan , Lin Zhong , Niraj K. Jha, User-perceived latency driven voltage scaling for interactive applications, Proceedings of the 42nd annual conference on Design automation, June 13-17, 2005, Anaheim, California, USA [doi>10.1145/1065579.1065743]

INDEX TERMS

Primary Classification:
  D. Software
  D.4 OPERATING SYSTEMS

Additional Classification:
  C. Computer Systems Organization
  C.4 PERFORMANCE OF SYSTEMS

  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.1 Multimedia Information Systems

General Terms:
Design, Experimentation, Human Factors, Measurement, Performance

Keywords:
autonomic systems, human directed adaptation, speculative remote display, user comfort with resource borrowing, user-driven power management, user-driven scheduling