MANET simulation studies: the incredibles

ABSTRACT

Simulation is the research tool of choice for a majority of the mobile ad hoc network (MANET) community. However, while the use of simulation has increased, the credibility of the simulation results has decreased. To determine the state of MANET simulation studies, we surveyed the 2000-2005 proceedings of the ACM International Symposium on Mobile Ad Hoc Networking and Computing (MobiHoc). From our survey, we found significant shortfalls. We present the results of our survey in this paper. We then summarize common simulation study pitfalls found in our survey. Finally, we discuss the tools available that aid the development of rigorous simulation studies. We offer these results to the community with the hope of improving the credibility of MANET simulation-based studies.

REFERENCES

• 1: R. Bagrodia and M. Takai. Position paper on validation of network simulation models. Computer Science Department of University of California, Los Angeles, 1999.
• 2: Osman Balci, Validation, verification, and testing techniques throughout the life cycle of a simulation study, Proceedings of the 26th conference on Winter simulation, p.215-220, December 11-14, 1994, Orlando, Florida, United States
• 3: Osman Balci, Verification validation and accreditation of simulation models, Proceedings of the 29th conference on Winter simulation, p.135-141, December 07-10, 1997, Atlanta, Georgia, United States [doi>10.1145/268437.268462]
• 4: Russell R. Barton, Design of experiments: designing simulation experiments, Proceedings of the 33nd conference on Winter simulation, December 09-12, 2001, Arlington, Virginia
• 5: Santashil PalChaudhuri , Jean-Yves Le Boudec , Milan Vojnovic, Perfect Simulations for Random Trip Mobility Models, Proceedings of the 38th annual Symposium on Simulation, p.72-79, April 04-06, 2005 [doi>10.1109/ANSS.2005.33]
• 6: Lawrence S. Brakmo , Larry L. Peterson, Experiences with network simulation, Proceedings of the 1996 ACM SIGMETRICS international conference on Measurement and modeling of computer systems, p.80-90, May 23-26, 1996, Philadelphia, Pennsylvania, United States
• 7: Linda Briesemeister , Günter Hommel, Role-based multicast in highly mobile but sparsely connected ad hoc networks, Proceedings of the 1st ACM international symposium on Mobile ad hoc networking & computing, November 20, 2000, Boston, Massachusetts
• 8: Thomas D. Dyer , Rajendra V. Boppana, A comparison of TCP performance over three routing protocols for mobile ad hoc networks, Proceedings of the 2nd ACM international symposium on Mobile ad hoc networking & computing, October 04-05, 2001, Long Beach, CA, USA [doi>10.1145/501422.501425]
• 9: Association for Computing Machinery. The ACM International Symposium on Mobile Ad Hoc Networking and Computing (MobiHoc). URL: http://www.sigmobile.org/mobihoc. Page accessed on April 23, 2005.
• 10: David Goldsman , Gamze Tokol, Output analysis: output analysis procedures for computer simulations, Proceedings of the 32nd conference on Winter simulation, December 10-13, 2000, Orlando, Florida
• 11: Akaroa Research Group. Akaroa2. URL: http://www.tkn.tu-berlin.de/research/ns2_akaroa-2/ns.html. Page accessed on April 21, 2005.
• 12: The VINT Group. Ns-users mailing list. http://mailman.isi.edu/mailman/listinfo/nsusers. Page accessed on April 23, 2005.
• 13: J. Heidemann, S. Kumar, and K. Mills. Expanding confidence in network simulations. IEEE Network, September/October:58--63, 2001.
• 14: Xiaoyan Hong , Mario Gerla , Guangyu Pei , Ching-Chuan Chiang, A group mobility model for ad hoc wireless networks, Proceedings of the 2nd ACM international workshop on Modeling, analysis and simulation of wireless and mobile systems, p.53-60, August 20-20, 1999, Seattle, Washington, United States [doi>10.1145/313237.313248]
• 15: Yih-Chun Hu , David B. Johnson, Implicit source routes for on-demand ad hoc network routing, Proceedings of the 2nd ACM international symposium on Mobile ad hoc networking & computing, October 04-05, 2001, Long Beach, CA, USA [doi>10.1145/501417.501418]
• 16: D. Johnson and D. Maltz. Dynamic source routing in ad hoc wireless networks. In Mobile Computing, pages 153--181. 1996.
• 17: Stuart Kurkowski , Tracy Camp , Neil Mushell , Michael Colagrosso, A Visualization and Analysis Tool for NS-2 Wireless Simulations: iNSpect, Proceedings of the 13th IEEE International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems, p.503-506, September 27-29, 2005 [doi>10.1109/MASCOT.2005.11]
• 18: Averill M. Law , W. David Kelton, Simulation Modeling and Analysis, McGraw-Hill Higher Education, 1997
• 19: Pierre L'Ecuyer , Richard Simard, On the performance of birthday spacings tests with certain families of random number generators, Mathematics and Computers in Simulation, v.55 n.1-3, p.131-137, Feb. 15, 2001 [doi>10.1016/S0378-4754(00)00253-6]
• 20: Seungjoon Lee , Chongkwon Kim, Neighbor supporting ad hoc multicast routing protocol, Proceedings of the 1st ACM international symposium on Mobile ad hoc networking & computing, November 20, 2000, Boston, Massachusetts
• 21: J. Liu, L. F. Perrone, Y. Yuan, and D. Nicol. The simulator for wireless ad hoc networks (SWAN). URL: http://www.eg.bucknell.edu/swan/. Page accessed on August 19, 2005.
• 22: Tommaso Melodia , Dario Pompili , Vehbi C. Gungor , Ian F. Akyildiz, A distributed coordination framework for wireless sensor and actor networks, Proceedings of the 6th ACM international symposium on Mobile ad hoc networking and computing, May 25-27, 2005, Urbana-Champaign, IL, USA [doi>10.1145/1062689.1062704]
• 23: S. Midkiff. Simulation: Methodology. Presentation to ECE590 class. URL: http://fiddle.visc.vt.edu/courses/ece5984net2/lectures/09_sim.pdf. Page accessed on April 15, 2005.
• 24: D. Montgomery. Design and Analysis of Experiments. Wiley, 2005.
• 25: E. Mota, A. Wolisz, and K. Pawlikowski. Experience with Akaroa: speeding up stochastic simulation using parallel replications. In Proceedings of the High Performance Computing and Advanced Simulation Technologies Conference, pages 296--301, 1998.
• 26: W. Navidi and T. Camp. Stationary distributions for the random waypoint mobility model. IEEE Transactions on Mobile Computing, pages 99--108, 2004.
• 27: The NS manual: formerly known as notes and documentation, 2003. URL: http://www.isi.edu/nsnam/ns/. Page accessed on April 21, 2005.
• 28: K. Pawlikowski. Do not trust all simulation studies of telecommunications networks. International Conference on Information Networking (ICOIN), pages 899--908, 2003.
• 29: K. Pawlikowski, J. Jeong, and R. Lee. On credibility of simulation studies of telecommunication networks. IEEE Communications Magazine, pages 132--139, 2001.
• 30: K. Pawlikowski, J. Jeong, and R. Lee. Letters to the editor. IEEE Communications Magazine, pages 132--139, 2002.
• 31: K. Pawlikowski and V. Yau. Akaroa-2 Manual, 1995. Department of Computer Science, University of Canterbury, ChristChurch New Zealand. URL: http://www.tkn.tu-berlin.de/research/ns2_akaroa-2. Page accessed on April 21, 2005.
• 32: G. Pei, M. Gerla, X. Hong, and C. Chiang. A wireless hierarchical routing protocol with group mobility. In Proceedings of the IEEE Wireless Communications and Networking Conference (WCNC), pages 1538--1542, 1999.
• 33: Luiz Felipe Perrone , Yougu Yuan , David M. Nicol, Simulation of large scale networks II: modeling and simulation best practices for wireless ad hoc networks, Proceedings of the 35th conference on Winter simulation: driving innovation, December 07-10, 2003, New Orleans, Louisiana
• 34: The CMU Monarch Project. The CMU monarch extensions to the ns simulator. URL: http://www.monarch.cs.cmu.edu/. Page accessed on April 25, 2005.
• 35: The SEACORN Project. SEACORN simulation tools. URL: http://seacorn.ptinovacao.pt/sim_tools.html. Page accessed on April 23, 2005.
• 36: E. Royer, P. M. Melliar-Smith, and L. Moser. An analysis of the optimum node density for ad hoc mobile networks. In Proceedings of the IEEE International Conference on Communications (ICC), pages 857--861, 2001.
• 37: Narayanan Sadagopan , Fan Bai , Bhaskar Krishnamachari , Ahmed Helmy, PATHS: analysis of PATH duration statistics and their impact on reactive MANET routing protocols, Proceedings of the 4th ACM international symposium on Mobile ad hoc networking & computing, June 01-03, 2003, Annapolis, Maryland, USA [doi>10.1145/778415.778444]
• 38: Susan M. Sanchez, Output modeling: abc's of output analysis, Proceedings of the 33nd conference on Winter simulation, December 09-12, 2001, Arlington, Virginia
• 39: Robert G. Sargent, Verification, validation, and accreditation: verification, validation, and accreditation of simulation models, Proceedings of the 32nd conference on Winter simulation, December 10-13, 2000, Orlando, Florida
• 40: L. Schruben. Detecting initialization bias in simulation output. Operations Research, pages 569--590, 1982.
• 41: Rong Zheng , Jennifer C. Hou , Lui Sha, Asynchronous wakeup for ad hoc networks, Proceedings of the 4th ACM international symposium on Mobile ad hoc networking & computing, June 01-03, 2003, Annapolis, Maryland, USA [doi>10.1145/778415.778420]

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.2 COMPUTER-COMMUNICATION NETWORKS
      C.2.1 Network Architecture and Design
          Subjects: Wireless communication

Additional Classification:
  C. Computer Systems Organization
  C.4 PERFORMANCE OF SYSTEMS
      Subjects: Modeling techniques

  I. Computing Methodologies
  I.6 SIMULATION AND MODELING
      I.6.7 Simulation Support Systems

General Terms:
Design, Experimentation, Measurement, Performance