Distributed versus face-to-face meetings for architecture evalution: a controlled experiment

ABSTRACT

Scenario-based methods for evaluating software architecture require a large number of stakeholders to be collocated for evaluation sessions. Collocating stakeholders is often an expensive exercise. To reduce expense, we have proposed a framework for supporting software architecture evaluation process using groupware systems. This paper presents a controlled experiment that we conducted to assess the effectiveness of scenario profile construction using distributed meetings. We used a cross-over experiment involving 32 teams of three 3rd and 4th year undergraduate students. We found that the quality of scenarios produced by distributed teams using a groupware tool were significantly better than the quality of scenarios produced by face-to-face teams (p<0.001). However, questionnaires indicated that most participants preferred the face-toface arrangement (82%) and 60% thought the distributed meetings were less efficient. We conclude that distributed meetings are extremely effective but that tool support must be of a high standard or participants will not find distributed meetings acceptable.

REFERENCES

• 1: ZWiki System, Last accessed on 8th June 2004.
• 2: Muhammed Ali Babar , Barbara Kitchenham , Ian Gorton, Towards a distributed software architecture evaluation process: a preliminary assessment, Proceeding of the 28th international conference on Software engineering, May 20-28, 2006, Shanghai, China [doi>10.1145/1134285.1134430]
• 3: Muhammad Ali Babar , Barbara Kitchenham , Liming Zhu , Ian Gorton , Ross Jeffery, An empirical study of groupware support for distributed software architecture evaluation process, Journal of Systems and Software, v.79 n.7, p.912-925, July 2006 [doi>10.1016/j.jss.2005.06.043]
• 4: Muhammad Ali Babar , Liming Zhu , Ross Jeffery, A Framework for Classifying and Comparing Software Architecture Evaluation Methods, Proceedings of the 2004 Australian Software Engineering Conference (ASWEC'04), p.309, April 13-16, 2004
• 5: V R Basili , R W Selby , D H Hutchens, Experimentation in software engineering, IEEE Transactions on Software Engineering, v.12 n.7, p.733-743, July 1986
• 6: Len Bass , Paul Clements , Rick Kazman, Software architecture in practice, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 1998
• 7: Bengtsson, P. Architecture-Level Modifiability Analysis Department of Software Engineering and Computer Science, Blekinge Institute of Technology, Sweden, 2002.
• 8: PerOlof Bengtsson , Jan Bosch, An experiment on creating scenario profiles for software change, Annals of Software Engineering, v.9 n.1-4, p.59-78, 2000 [doi>10.1023/A:1018968506874]
• 9: Robert P. Biuk-Aghai , Igor T. Hawryszkiewycz, Analysis of Virtual Workspaces, Proceedings of the 1999 International Symposium on Database Applications in Non-Traditional Environments, p.325, November 28-30, 1999
• 10: Barry Boehm , Paul Grünbacher , Robert O. Briggs, Developing Groupware for Requirements Negotiation: Lessons Learned, IEEE Software, v.18 n.3, p.46-55, May 2001 [doi>10.1109/52.922725]
• 11: Evaluating software architectures: methods and case studies, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 2001
• 12: Daniela E. Herlea Damian , Armin Eberlein , Mildred L. G. Shaw , Brian R. Gaines, Using Different Communication Media in Requirements Negotiation, IEEE Software, v.17 n.3, p.28-36, May 2000 [doi>10.1109/52.896247]
• 13: Norman Fenton , Shari Lawrence Pfleeger , Robert L. Glass, Science and Substance: A Challenge to Software Engineers, IEEE Software, v.11 n.4, p.86-95, July 1994 [doi>10.1109/52.300094]
• 14: Jerry Fjermestad, An analysis of communication mode in group support systems research, Decision Support Systems, v.37 n.2, p.239-263, May 2004 [doi>10.1016/S0167-9236(03)00021-6]
• 15: Fjermestad, J. and Hiltz, S.R. Group Support Systems: A Descriptive Evaluation of Case and Field Studies. Journal of Management Information Systems, 17 (3), 2000-2001, pp. 115--159.
• 16: Jerry Fjermestad , Starr Roxanne Hiltz, An assessment of group support systems experimental research: methodology and results, Journal of Management Information Systems, v.15 n.3, p.7-149, December 1998
• 17: Michiel Van Genuchten , Wieger Cornelissen , Cor Van Dijk, Supporting inspections with an electronic meeting system, Journal of Management Information Systems, v.14 n.3, p.165-178, December 1997
• 18: Michiel van Genuchten , Cor van Dijk , Henk Scholten , Doug Vogel, Using Group Support Systems for Software Inspections, IEEE Software, v.18 n.3, p.60-65, May 2001 [doi>10.1109/52.922727]
• 19: James D. Herbsleb , Deependra Moitra, Guest Editors' Introduction: Global Software Development, IEEE Software, v.18 n.2, p.16-20, March 2001 [doi>10.1109/52.914732]
• 20: Starr Roxanne Hiltz , Murray Turoff, The network nation: human communication via computer, MIT Press, Cambridge, MA, 1993
• 21: Martin Höst , Björn Regnell , Claes Wohlin, Using Students as Subjects—A Comparative Study ofStudents and Professionals in Lead-Time Impact Assessment, Empirical Software Engineering, v.5 n.3, p.201-214, November 2000 [doi>10.1023/A:1026586415054]
• 22: Sirkka L. Jarvenpaa , V. Srinivasan Rao , George P. Huber, “Computer support for meetings of groups working on unstructured problems: a field experiment", Management Information Systems Quarterly, v.12 n.4, p.645-665, December 1988 [doi>10.2307/249137]
• 23: Rick Kazman , Mario Barbacci , Mark Klein , S. Jeromy Carrière , Steven G. Woods, Experience with performing architecture tradeoff analysis, Proceedings of the 21st international conference on Software engineering, p.54-63, May 16-22, 1999, Los Angeles, California, United States [doi>10.1145/302405.302452]
• 24: Rick Kazman , Len Bass, Making Architecture Reviews Work in the Real World, IEEE Software, v.19 n.1, p.67-73, January 2002 [doi>10.1109/52.976943]
• 25: Kiesler, S., Siegel, J. and McGuire, T.W. Social Psychological Aspects of Computer-Mediated Communication. American Psychologist, 9 (10), 1984, pp. 1123--1134.
• 26: Barbara Kitchenham , John Fry , Stephen Linkman, The Case Against Cross-Over Designs in Software Engineering, Proceedings of the Eleventh Annual International Workshop on Software Technology and Engineering Practice (STEP'03), p.65-67, September 19-21, 2003
• 27: Barbara A. Kitchenham , Shari Lawrence Pfleeger , Lesley M. Pickard , Peter W. Jones , David C. Hoaglin , Khaled El Emam , Jarrett Rosenberg, Preliminary guidelines for empirical research in software engineering, IEEE Transactions on Software Engineering, v.28 n.8, p.721-734, August 2002 [doi>10.1109/TSE.2002.1027796]
• 28: Nico Lassing , Perlof Bengtsson , Hans van Vliet , Jan Bosch, Experiences with ALMA: architecture-level modifiability analysis, Journal of Systems and Software, v.61 n.1, p.47-57, March 2002 [doi>10.1016/S0164-1212(01)00113-3]
• 29: Joseph F. Maranzano , Sandra A. Rozsypal , Gus H. Zimmerman , Guy W. Warnken , Patricia E. Wirth , David M. Weiss, Architecture Reviews: Practice and Experience, IEEE Software, v.22 n.2, p.34-43, March 2005 [doi>10.1109/MS.2005.28]
• 30: Joseph Edward McGrath , Andrea B. Hollingshead, Groups Interacting with Technology: Ideas, Evidence, Issues and an Agenda, Sage Publications, Inc., Thousand Oaks, CA, 1993
• 31: Mennecke, B.E., Hoffer, J.A. and Wynne, B.E. The Implications of Group Development and History for Group Support System Theory and Practice. Small Group Research, 23 (4), pp. 524--572.
• 32: Nunamaker, J., Vogel, D., Heminger, A. and Martz, B. Experiences at IBM with Group Support Systems: A Field Study. Decision Support Systems, 5, 1989, pp. 183--196.
• 33: Jay F. Nunamaker, Jr. , Robert O. Briggs , Daniel D. Mittleman , Douglas R. Vogel , Pierre A. Balthazard, Lessons from a dozen years of group support systems research: a discussion of lab and field findings, Journal of Management Information Systems, v.13 n.3, p.163-207, December 1996
• 34: J. F. Nunamaker , Alan R. Dennis , Joseph S. Valacich , Douglas Vogel , Joey F. George, Electronic meeting systems, Communications of the ACM, v.34 n.7, p.40-61, July 1991 [doi>10.1145/105783.105793]
• 35: Paasivaara, M., Casper and Lassenius Collaboration Practices in Global Inter-organizational Software Development Projects. Software Process Improvement and Practice, 8 (4), 2003, pp. 183--199.
• 36: Dewayne E. Perry , Adam Porter , Michael W. Wade , Lawrence G. Votta , James Perpich, Reducing inspection interval in large-scale software development, IEEE Transactions on Software Engineering, v.28 n.7, p.695-705, July 2002 [doi>10.1109/TSE.2002.1019483]
• 37: Poole, M.S. and Desanctis, G. Understanding the Use of Group Decision Support Systems: The Theory of Adaptive Structuration. in Fulk, J. and Steinfield, C. eds. Organizations and Communication Techology, Sage Publications, Inc., Newbury, CA, USA, 1990, pp. 173--193.
• 38: Rosnow, R.L. and Rosenthal, R. People Studying People: Artifacts and Ethics in Behavioral Research. W.H. Freeman and Company, 1997.
• 39: Sakthivel, S. Virtual workgroups in offshore systems development. Information and Software Technology, 47 (5), 2005, pp. 305--318.
• 40: Senn, S. Cross-Over Trials in Clinical Research. John Wiley & Sons Ltd., 2002.
• 41: Toothaker, L.E. and Miller, L. Introductory Statistics for the Behavioral Science. Brooks/Cole Publishing Company, Pacific Grove, CA, USA, 1996.
• 42: Craig K. Tyran , Alan R. Dennis , Douglas R. Vogel , J. F. Nunamaker, Jr., The application of electronic meeting technology to support strategic management, MIS Quarterly, v.16 n.3, p.313-334, Sept. 1992 [doi>10.2307/249531]
• 43: Valacich, J., Dennis, A.R. and Nunamaker, J.F. Group Size and Anonymity Effects on Computer-Mediated Idea Generation. Small Group Research, 23 (1), 1992, pp. 49--73.
• 44: Joseph S. Valacich , Alan R. Dennis , J. F. Nunamaker, Jr., Electronic meeting support: the GroupSystems concept, International Journal of Man-Machine Studies, v.34 n.2, p.261-282, Feb. 1991 [doi>10.1016/0020-7373(91)90044-8]
• 45: Warkentin, M.E., Sayeed, L. and Hightower, R. Virtual Teams Versus Face-to-Face Teams: An Exploratory Study of a Web-based Conference System. Decision Sciences, 28 (4), 1997, pp. 975--996.
• 46: Claes Wohlin , Per Runeson , Martin Höst , Magnus C. Ohlsson , Bjöorn Regnell , Anders Wesslén, Experimentation in software engineering: an introduction, Kluwer Academic Publishers, Norwell, MA, 2000

INDEX TERMS

Primary Classification:
  K. Computing Milieux
  K.6 MANAGEMENT OF COMPUTING AND INFORMATION SYSTEMS
      K.6.3 Software Management
          Subjects: Software process

General Terms:
Experimentation, Management

Keywords:
architecture evaluation, controlled experiments, groupware support, process improvement, scneario development