Testing iptables

ABSTRACT

iptables is the most recent entry in a series of Linux firewall services. Because iptables is a security product in widespread use, correctness and performance are important issues. We present a methodology for iptables regression testing. Typically, correctness test suites generate test inputs and then log the observed output to a file. Log files from the first execution are manually checked for correctness and then compared to the results from subsequent test runs with a file differencing utility. The log files tend to be large; checking their correctness is tedious and error-prone. In contrast, we generate test traffic and verify the correctness of the iptables behaviour from the same program, eliminating the need for log files. We also present performance test results, focusing on throughput and delay as a function of the size of the iptables rule base. The measurements were conducted over Ethernet links running at 10, 100, and 1,000 Mbps.

REFERENCES

• 1: Sergio Antoy , Dick Hamlet, Automatically Checking an Implementation against Its Formal Specification, IEEE Transactions on Software Engineering, v.26 n.1, p.55-69, January 2000 [doi>10.1109/32.825766]
• 2: Boris Beizer, Software testing techniques (2nd ed.), Van Nostrand Reinhold Co., New York, NY, 1990
• 3: Gilles Bernot , Marie Claude Gaudel , Bruno Marre, Software testing based on formal specifications: a theory and a tool, Software Engineering Journal, v.6 n.6, p.387-405, Nov. 1991
• 4: {4} D. A. Carrington, I. D. MacColl, J. Mc-Donald, L. Murray, and P. A. Strooper. From Object-Z specifications to Class-Bench test suites. Journal on Software Testing, Verification and Reliability, 10(2):111-137, 2000.
• 5: Jeremy Dick , Alain Faivre, Automating the Generation and Sequencing of Test Cases from Model-Based Specifications, Proceedings of the First International Symposium of Formal Methods Europe on Industrial-Strength Formal Methods, p.268-284, April 19-23, 1993
• 6: Roong-Ko Doong , Phyllis G. Frankl, The ASTOOT approach to testing object-oriented programs, ACM Transactions on Software Engineering and Methodology (TOSEM), v.3 n.2, p.101-130, April 1994 [doi>10.1145/192218.192221]
• 7: {7} A. Grote, R. Funke, and H.-U. Heiss. Performance evaluation of firewalls in gigabit-networks. In Proc. Symposium on Performance Evaluation of Computer and Telecommunication Systems. Society for Computer Simulation, 1999.
• 8: {8} R. M. Hierons. Testing from a Z specification. Software Testing, Verification and Reliability, 7(1):19-33, 1997.
• 9: {9} D.M. Hoffman, P.A. Strooper, and L. White. Boundary values and automated component testing. Journal of Software Testing, Verification, and Review, 9(1):3- 26, 1999.
• 10: Hans-Martin Hörcher, Improving Software Tests Using Z Specifications, Proceedings of the 9th International Conference of Z Usres on The Z Formal Specification Notation, p.152-166, September 07-09, 1995
• 11: {11} W.E. Howden. Reliability of the path analysis testing strategy. IEEE Transaction on Software Engineering, SE- 2(3):208-215, September 1976.
• 12: {12} W.E. Howden. Functional program testing. IEEE Transaction on Software Engineering , SE-6(3):162-169, 1980.
• 13: Jason McDonald , Paul Strooper, Translating Object-Z Specifications to Passive Test Oracles, Proceedings of the Second IEEE International Conference on Formal Engineering Methods, p.165, December 09-11, 1998
• 14: Glenford J. Myers, Art of Software Testing, John Wiley & Sons, Inc., New York, NY, 1979
• 15: Erich Mikk, Compilation of Z Specifications into C for Automatic Test Result Evaluation, Proceedings of the 9th International Conference of Z Usres on The Z Formal Specification Notation, p.167-180, September 07-09, 1995
• 16: Sandra Rapps , Elaine J. Weyuker, Selecting software test data using data flow information, IEEE Transactions on Software Engineering, v.11 n.4, p.367-375, April 1985 [doi>10.1109/TSE.1985.232226]
• 17: {17} D.J. Richardson and L.A. Clarke. Partition analysis: a method combining testing and verification. IEEE Transaction on Software Engineering, SE-11(12):1477- 1490, 1985.
• 18: Mike Schiffman, Building Open Source Network Security Tools: Components and Techniques, John Wiley & Sons, Inc., New York, NY, 2002
• 19: William Stallings, High Speed Networks and Internets: Performance and Quality of Service, Prentice Hall PTR, Upper Saddle River, NJ, 2001
• 20: Susan Stepney, Testing as Abstraction, Proceedings of the 9th International Conference of Z Usres on The Z Formal Specification Notation, p.137-151, September 07-09, 1995
• 21: Phil Stocks , David Carrington, A Framework for Specification-Based Testing, IEEE Transactions on Software Engineering, v.22 n.11, p.777-793, November 1996 [doi>10.1109/32.553698]
• 22: Andrew S. Tanenbaum, Computer Networks, Prentice Hall PTR, Upper Saddle River, NJ, 1985
• 23: {23} E.J. Weyuker and T.J. Ostrand. Theories of program testing and the application of revealing subdomains. IEEE Transaction on Software Engineering, SE- 6(3):236-246, 1980.
• 24: {24} L.J. White and E.I. Cohen. A domain strategy for computer program testing. IEEE Transaction on Software Engineering , SE-6(3):247-257, 1980.

INDEX TERMS

Primary Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING
      D.2.5 Testing and Debugging
          Subjects: Testing tools (e.g., data generators, coverage testing)

Additional Classification:
  C. Computer Systems Organization
  C.2 COMPUTER-COMMUNICATION NETWORKS
      C.2.1 Network Architecture and Design
          Subjects: Network communications
      C.2.2 Network Protocols
          Subjects: Routing protocols
          Nouns: IP

General Terms:
Algorithms, Design, Experimentation, Measurement, Performance