An empirical study of the reliability of UNIX utilities

Subscribe (Full Service)


	Search: The ACM Digital Library The Guide

	Feedback

An empirical study of the reliability of UNIX utilities

Full text

Pdf (2.38 MB)

Source

Communications of the ACM archive
Volume 33 , Issue 12 (December 1990) table of contents

Pages: 32 - 44

Year of Publication: 1990

ISSN:0001-0782

Authors

Barton P. Miller	Univ. of Wisconsin, Madison
Louis Fredriksen	Univ. of Wisconsin, Madison
Bryan So	Univ. of Wisconsin, Madison

Publisher

ACM New York, NY, USA

Bibliometrics

Downloads (6 Weeks): 14, Downloads (12 Months): 86, Citation Count: 57

Additional Information:

abstract references cited by index terms review collaborative colleagues

Tools and Actions:	Request Permissions Review this Article Save this Article to a Binder Display Formats: BibTeX EndNote ACM Ref

DOI Bookmark:	Use this link to bookmark this Article: http://doi.acm.org/10.1145/96267.96279 What is a DOI?

ABSTRACT

The following section describes the tools we built to test the utilities. These tools include the fuzz (random character) generator, ptyjig (to test interactive utilities), and scripts to automate the testing process. Next, we will describe the tests we performed, giving the types of input we presented to the utilities. Results from the tests will follow along with an analysis of the results, including identification and classification of the program bugs that caused the crashes. The final section presents concluding remarks, including suggestions for avoiding the types of problems detected by our study and some commentary on the bugs we found. We include an Appendix with the user manual pages for fuzz and ptyjig.

REFERENCES

Note: OCR errors may be found in this Reference List extracted from the full text article. ACM has opted to expose the complete List rather than only correct and linked references.

	1	Edsger W. Dijkstra, Letters to the editor: go to statement considered harmful, Communications of the ACM, v.11 n.3, p.147-148, March 1968 [doi>10.1145/362929.362947]
	2	Jon A. Rochlis , Mark W. Eichin, With microscope and tweezers: the worm from MIT's perspective, Communications of the ACM, v.32 n.6, p.689-698, June 1989 [doi>10.1145/63526.63528]
	3	E. H. Spafford, Crisis and aftermath, Communications of the ACM, v.32 n.6, p.678-687, June 1989 [doi>10.1145/63526.63527]
	4	David A. Wood , Garth A. Gibson , Randy H. Katz, Verifying a Multiprocessor Cache Controller Using Random Case, University of California at Berkeley, Berkeley, CA, 1989

CITED BY 57

	Jeffrey M. Voas, Certifying Off-the-Shelf Software Components, Computer, v.31 n.6, p.53-59, June 1998
	Andy Chou , Junfeng Yang , Benjamin Chelf , Seth Hallem , Dawson Engler, An empirical study of operating systems errors, ACM SIGOPS Operating Systems Review, v.35 n.5, Dec. 2001
	Todd M. Austin , Scott E. Breach , Gurindar S. Sohi, Efficient detection of all pointer and array access errors, ACM SIGPLAN Notices, v.29 n.6, p.290-301, June 1994
	Harold Thimbleby , Paul Cairns , Matt Jones, Usability analysis with Markov models, ACM Transactions on Computer-Human Interaction (TOCHI), v.8 n.2, p.99-132, June 2001
	Roland H. Untch, Mutation-based software testing using program schemata, Proceedings of the 30th annual Southeast regional conference, April 08-10, 1992, Raleigh, North Carolina
	Ralf Hildebrandt , Andreas Zeller, Simplifying failure-inducing input, ACM SIGSOFT Software Engineering Notes, v.25 n.5, p.135-145, Sept. 2000
	Andreas Zeller , Ralf Hildebrandt, Simplifying and Isolating Failure-Inducing Input, IEEE Transactions on Software Engineering, v.28 n.2, p.183-200, February 2002
	Jeffrey Voas, Developing a Usage-Based Software Certification Process, Computer, v.33 n.8, p.32-37, August 2000
	James A. Whittaker, Software's Invisible Users, IEEE Software, v.18 n.3, p.84-88, May 2001
	Larry Augustin , Dan Bressler , Guy Smith, Accelerating software development through collaboration, Proceedings of the 24th International Conference on Software Engineering, May 19-25, 2002, Orlando, Florida
	Ghani A. Kanawati , Nasser A. Kanawati , Jacob A. Abraham, FERRARI: A Flexible Software-Based Fault and Error Injection System, IEEE Transactions on Computers, v.44 n.2, p.248-260, February 1995
	Mihai Christodorescu , Somesh Jha, Testing malware detectors, ACM SIGSOFT Software Engineering Notes, v.29 n.4, July 2004
	Philip Koopman , John DeVale, The Exception Handling Effectiveness of POSIX Operating Systems, IEEE Transactions on Software Engineering, v.26 n.9, p.837-848, September 2000
	Arup Mukherjee , Daniel P. Siewiorek, Measuring Software Dependability by Robustness Benchmarking, IEEE Transactions on Software Engineering, v.23 n.6, p.366-378, June 1997
	Jedidiah R. Crandall , Zhendong Su , S. Felix Wu , Frederic T. Chong, On deriving unknown vulnerabilities from zero-day polymorphic and metamorphic worm exploits, Proceedings of the 12th ACM conference on Computer and communications security, November 07-11, 2005, Alexandria, VA, USA
	Concettina Del Grosso , Giuliano Antoniol , Massimiliano Di Penta , Philippe Galinier , Ettore Merlo, Improving network applications security: a new heuristic to generate stress testing data, Proceedings of the 2005 conference on Genetic and evolutionary computation, June 25-29, 2005, Washington DC, USA
	John Regehr, Random testing of interrupt-driven software, Proceedings of the 5th ACM international conference on Embedded software, September 18-22, 2005, Jersey City, NJ, USA
	A. Boulanger, Open-source versus proprietary software: is one more reliable and secure than the other?, IBM Systems Journal, v.44 n.2, p.239-248, January 2005
	Kent D. Wilken , Timothy Kong, Concurrent Detection of Software and Hardware Data-Access Faults, IEEE Transactions on Computers, v.46 n.4, p.412-424, April 1997
	Barton P. Miller , Gregory Cooksey , Fredrick Moore, An empirical study of the robustness of MacOS applications using random testing, Proceedings of the 1st international workshop on Random testing, July 20-20, 2006, Portland, Maine
	Rupak Majumdar , Ru-Gang Xu, Directed test generation using symbolic grammars, Proceedings of the twenty-second IEEE/ACM international conference on Automated software engineering, November 05-09, 2007, Atlanta, Georgia, USA
	Cristian Cadar , Vijay Ganesh , Peter M. Pawlowski , David L. Dill , Dawson R. Engler, EXE: automatically generating inputs of death, Proceedings of the 13th ACM conference on Computer and communications security, October 30-November 03, 2006, Alexandria, Virginia, USA
	James H. Andrews , Susmita Haldar , Yong Lei , Felix Chun Hang Li, Tool support for randomized unit testing, Proceedings of the 1st international workshop on Random testing, July 20-20, 2006, Portland, Maine
	T. Y. Chen , F.-C. Kuo, Is adaptive random testing really better than random testing, Proceedings of the 1st international workshop on Random testing, July 20-20, 2006, Portland, Maine
	Jeffrey Putnam , Kaleb Albee, Breaking browsers: a survey, Journal of Computing Sciences in Colleges, v.22 n.2, p.59-65, December 2006
	Jeffrey C. Mogul, Emergent (mis)behavior vs. complex software systems, ACM SIGOPS Operating Systems Review, v.40 n.4, October 2006
	Barton P. Miller , Gregory Cooksey , Fredrick Moore, An empirical study of the robustness of MacOS applications using random testing, ACM SIGOPS Operating Systems Review, v.41 n.1, January 2007
	Trevor Jim , J. Greg Morrisett , Dan Grossman , Michael W. Hicks , James Cheney , Yanling Wang, Cyclone: A Safe Dialect of C, Proceedings of the General Track: 2002 USENIX Annual Technical Conference, p.275-288, June 10-15, 2002
	F. C. Kuo , T. Y. Chen , H. Liu , W. K. Chan, Enhancing adaptive random testing in high dimensional input domains, Proceedings of the 2007 ACM symposium on Applied computing, March 11-15, 2007, Seoul, Korea
	Crispin Cowan , Calton Pu , Dave Maier , Heather Hintony , Jonathan Walpole , Peat Bakke , Steve Beattie , Aaron Grier , Perry Wagle , Qian Zhang, StackGuard: automatic adaptive detection and prevention of buffer-overflow attacks, Proceedings of the 7th conference on USENIX Security Symposium, 1998, p.5-5, January 26-29, 1998, San Antonio, Texas
	Patrice Godefroid , Adam Kiezun , Michael Y. Levin, Grammar-based whitebox fuzzing, ACM SIGPLAN Notices, v.43 n.6, June 2008
	Will Drewry , Tavis Ormandy, Flayer: exposing application internals, Proceedings of the first conference on First USENIX Workshop on Offensive Technologies, p.1-1, August 06, 2007, Boston, MA
	Carlos Pacheco , Shuvendu K. Lahiri , Michael D. Ernst , Thomas Ball, Feedback-Directed Random Test Generation, Proceedings of the 29th International Conference on Software Engineering, p.75-84, May 20-26, 2007
	Alex Groce , Gerard Holzmann , Rajeev Joshi, Randomized Differential Testing as a Prelude to Formal Verification, Proceedings of the 29th International Conference on Software Engineering, p.621-631, May 20-26, 2007
	Andrea Lanzi , Lorenzo Martignoni , Mattia Monga , Roberto Paleari, A Smart Fuzzer for x86 Executables, Proceedings of the Third International Workshop on Software Engineering for Secure Systems, p.7, May 20-26, 2007
	Raghavendra Rao Loka, Software engineering: quality assurance, ACM SIGSOFT Software Engineering Notes, v.17 n.3, p.34-38, July 1992
	Joseph Tucek , Shan Lu , Chengdu Huang , Spiros Xanthos , Yuanyuan Zhou, Triage: diagnosing production run failures at the user's site, ACM SIGOPS Operating Systems Review, v.41 n.6, December 2007
	Cherry Keahey Owen, Tracking system bugs: why are buffer overruns still around?, Proceedings of the 35th annual ACM SIGUCCS conference on User services, p.280-283, October 07-10, 2007, Orlando, Florida, USA
	James H. Andrews , Felix C. H. Li , Tim Menzies, Nighthawk: a two-level genetic-random unit test data generator, Proceedings of the twenty-second IEEE/ACM international conference on Automated software engineering, November 05-09, 2007, Atlanta, Georgia, USA
	Stephan Neuhaus , Thomas Zimmermann , Christian Holler , Andreas Zeller, Predicting vulnerable software components, Proceedings of the 14th ACM conference on Computer and communications security, October 28-31, 2007, Alexandria, Virginia, USA
	Jong P. Yoon, Presto Authorization: A Bitmap Indexing Scheme for High-Speed Access Control to XML Documents, IEEE Transactions on Knowledge and Data Engineering, v.18 n.7, p.971-987, July 2006
	Carlos Pacheco , Shuvendu K. Lahiri , Thomas Ball, Finding errors in .net with feedback-directed random testing, Proceedings of the 2008 international symposium on Software testing and analysis, July 20-24, 2008, Seattle, WA, USA
	Joachim Viide , Aki Helin , Marko Laakso , Pekka Pietikäinen , Mika Seppänen , Kimmo Halunen , Rauli Puuperä , Juha Röning, Experiences with model inference assisted fuzzing, Proceedings of the 2nd conference on USENIX Workshop on offensive technologies, p.1-6, July 28, 2008, San Jose, CA
	C. Del Grosso , G. Antoniol , E. Merlo , P. Galinier, Detecting buffer overflow via automatic test input data generation, Computers and Operations Research, v.35 n.10, p.3125-3143, October, 2008
	Daniel Bilar, Callgraph properties of executables, AI Communications, v.20 n.4, p.231-243, December 2007
	Lorenzo Martignoni , Roberto Paleari , Giampaolo Fresi Roglia , Danilo Bruschi, Testing CPU emulators, Proceedings of the eighteenth international symposium on Software testing and analysis, July 19-23, 2009, Chicago, IL, USA
	Fei-Ching Kuo , Tsong Yueh Chen , Huai Liu , Wing Kwong Chan, Enhancing adaptive random testing for programs with high dimensional input domains or failure-unrelated parameters, Software Quality Control, v.16 n.3, p.303-327, September 2008
	Tsong Yueh Chen , Fei-Ching Kuo , Huai Liu, Distributing test cases more evenly in adaptive random testing, Journal of Systems and Software, v.81 n.12, p.2146-2162, December, 2008
	Jaeyeon Jung , Anmol Sheth , Ben Greenstein , David Wetherall , Gabriel Maganis , Tadayoshi Kohno, Privacy oracle: a system for finding application leaks with black box differential testing, Proceedings of the 15th ACM conference on Computer and communications security, October 27-31, 2008, Alexandria, Virginia, USA
	Qixu Liu , Yuqing Zhang, TFTP vulnerability finding technique based on fuzzing, Computer Communications, v.31 n.14, p.3420-3426, September, 2008
	Cristian Cadar , Vijay Ganesh , Peter M. Pawlowski , David L. Dill , Dawson R. Engler, EXE: Automatically Generating Inputs of Death, ACM Transactions on Information and System Security (TISSEC), v.12 n.2, p.1-38, December 2008
	Diomidis Spinellis , Georgios Gousios , Vassilios Karakoidas , Panagiotis Louridas , Paul J. Adams , Ioannis Samoladas , Ioannis Stamelos, Evaluating the Quality of Open Source Software, Electronic Notes in Theoretical Computer Science (ENTCS), 233, p.5-28, March, 2009
	Stelios Sidiroglou , Oren Laadan , Carlos Perez , Nicolas Viennot , Jason Nieh , Angelos D. Keromytis, ASSURE: automatic software self-healing using rescue points, ACM SIGPLAN Notices, v.44 n.3, March 2009
	Michael W. Bigrigg, Framework for exercising I/O exception handling code, International Journal of Information and Communication Technology, v.1 n.3/4, p.244-258, March 2008
	T. Y. Chen , De Hao Huang , F.-C. Kuo , R. G. Merkel , Johannes Mayer, Enhanced lattice-based adaptive random testing, Proceedings of the 2009 ACM symposium on Applied Computing, March 08-12, 2009, Honolulu, Hawaii
	Vijay Ganesh , Tim Leek , Martin Rinard, Taint-based directed whitebox fuzzing, Proceedings of the 2009 IEEE 31st International Conference on Software Engineering, p.474-484, May 16-24, 2009
	Westley Weimer , ThanhVu Nguyen , Claire Le Goues , Stephanie Forrest, Automatically finding patches using genetic programming, Proceedings of the 2009 IEEE 31st International Conference on Software Engineering, p.364-374, May 16-24, 2009

INDEX TERMS

Primary Classification:
D. Software
D.4 OPERATING SYSTEMS

General Terms:
Design, Reliability, Security

REVIEW

"Brett D. Fleisch : Reviewer"

This study examined the reliability of UNIX utilities using empirical testing procedures. The authors tested roughly 90 different utility programs on seven versions of UNIX, including a commercial version. The paper presents the tools used to more...

Collaborative Colleagues:

Barton P. Miller: colleagues

Louis Fredriksen: colleagues

Bryan So: colleagues

Adobe Acrobat

QuickTime

Windows Media Player

Real Player