Performance of wireless network simulators: a case study

ABSTRACT

Designing protocols for wireless networks is a challenging task. Combined with the fact that such networks are often deployed for critical missions like forest fire detection in the WSN scenario or have to function properly and efficiently for an extended period of time, it is desirable to thoroughly test, analyze and evaluate newly developed communication protocols before deployment. In order to do this, simulations are a good compromise between cost/complexity and accuracy of the results. Since there are many simulators for wireless networks, it is often difficult to decide which one to choose. Therefore, we present a case study in which four common wireless network simulators were used to evaluate a well-known topology control protocol (SPAN). Within the case study, we describe the strengths and weaknesses of the examined network simulators: First, we evaluate the usability of the simulators in terms of different parts of the protocol developer's work process. Moreover, we also focus on the simulator's support for reusability and maintainability of simulation models by measuring particular model properties. For this purpose, we have proposed a model of quality for network simulators. The model of quality defines which properties of models to measure and how to interpret them. As opposed to other simulator comparisons, we do not focus on the correlation of the individual simulation results. Through this paper, we aim at providing a basis for finding an adequate simulator for a particular task.

REFERENCES

• 1: L. Begg, W. Liu, K. Pawlikowski, S. Perera, and H. Sirisena. Survey of simulators of next generation networks for studying service availability and resilience. Technical Report TR-COSC 05/06, Department of Computer Science & Software Engineering, University of Canterbury, Christchurch, New Zealand, February 2006.
• 2: David Cavin , Yoav Sasson , André Schiper, On the accuracy of MANET simulators, Proceedings of the second ACM international workshop on Principles of mobile computing, October 30-31, 2002, Toulouse, France [doi>10.1145/584490.584499]
• 3: Benjie Chen , Kyle Jamieson , Hari Balakrishnan , Robert Morris, Span: an energy-efficient coordination algorithm for topology maintenance in ad hoc wireless networks, Wireless Networks, v.8 n.5, p.481-494, September 2002 [doi>10.1023/A:1016542229220]
• 4: D. Curren. A survey of simulation in sensor networks. Student project, www.cs.binghamton.edu/~kang/teaching/cs580s/david.pdf, 2007.
• 5: DARPA/NSF. The network simulator - ns-2. http://www.isi.edu/nsnam/ns/.
• 6: T. J.-S. developers. J-sim. http://www.j-sim.org.
• 7: T. S. developers. Shox - a scalable ad hoc network simulator. http://shox.sourceforge.net.
• 8: S. Duflos, G. L. Grand, A. A. Diallo, C. Chaudet, A. Hecker, C. Balducelli, F. Flentge, C. Schwaegerl, and O. Seifert. Deliverable d 1.3.2: List of available and suitable simulation components. Technical report, École Nationale Supérieure des Télécommunications (ENST), September 2006.
• 9: V. Efthimia. Free tools for network simulation. Master's thesis, University of Macedonia, Thessaloniki, 2006.
• 10: E. Egea-Lopez, J. Vales-Alonso, A. Martinez-Sala, P. Pavon-Mario, and J. Garcia-Haro. c. IEEE Communications Magazine, 44(7):64--73, July 2006.
• 11: L. Hogie, P. Bouvry, and F. Guinand. An overview of manets simulation. In Electronic Notes in Theoretical Computer Science, Proc. of 1st International Workshop on Methods and Tools for Coordinating Concurrent, Distributed and Mobile Systems (MTCoord 2005), LNCS, pages 81--101, Namur, Belgium, April 2005. Elsevier.
• 12: M. Karl. A comparison of the architecture of network simulators ns-2 and tossim. In Proceedings of Performance Simulation of Algorithms and Protocols Seminar. Institut fr Parallele und Verteilte Systeme, Abteilung Verteilte Systeme, Universität Stuttgart, 2005.
• 13: Brad Karp , H. T. Kung, GPSR: greedy perimeter stateless routing for wireless networks, Proceedings of the 6th annual international conference on Mobile computing and networking, p.243-254, August 06-11, 2000, Boston, Massachusetts, United States [doi>10.1145/345910.345953]
• 14: A. Lemke and A. Sarkohi. Werkzeuge zur netzwerksimulation. In G. Wittenburg, editor, Proceedings of Seminar Technische Informatik. Freie Universität Berlin, June 2006.
• 15: G. F. Lucio, M. Paredes-Farrera, E. Jammeh, M. Fleury, and M. J. Reed. Opnet modeler and ns-2 - comparing the accuracy of network simulators for packet-level analysis using a network testbed. WSEAS Transactions on Computers, 2(3):700--707, July 2003.
• 16: D. Nicol. Comparison of network simulators revisited. http://www.ssfnet.org/Exchange/gallery/dumbbell/dumbbell-performance-May02.pdf, May 2002.
• 17: P. Novák. Simulation of network structures. Master's thesis, Department of Software Engineering, Charles University in Prague, August 2006.
• 18: I. Opnet Technologies. Opnet. http://www.opnet.com.
• 19: R. Repp. Vergleich der verfahren simulation und emulation fr die evaluation von protokollen. Master's thesis, Institut fr Parallele und Verteilte Systeme (IPVS), Universitäts Stuttgart, December 2003.
• 20: B. Schilling. Qualitative comparison of network simulation tools. Technical report, Institute of Parallel and Distributed Systems (IPVS), University of Stuttgart, January 2005.
• 21: A. Vargas. Omnet++ - discrete event simulation system. http://www.omnetpp.org.

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.2 COMPUTER-COMMUNICATION NETWORKS
      C.2.1 Network Architecture and Design
          Subjects: Wireless communication

Additional Classification:
  C. Computer Systems Organization
  C.2 COMPUTER-COMMUNICATION NETWORKS
      C.2.1 Network Architecture and Design
          Subjects: Network topology

General Terms:
Design, Performance

Keywords:
J-Sim, OMNeT++, SPAN, ShoX, case study, maintainability, model of quality, model properties, network simulators, ns-2, reusability, wireless network