Bias in computer systems

ABSTRACT

From an analysis of actual cases, three categories of bias in computer systems have been developed: preexisting, technical, and emergent. Preexisting bias has its roots in social institutions, practices, and attitudes. Technical bias arises from technical constraints of considerations. Emergent bias arises in a context of use. Although others have pointed to bias inparticular computer systems and have noted the general problem, we know of no comparable work that examines this phenomenon comprehensively and which offers a framework for understanding and remedying it. We conclude by suggesting that freedom from bias should by counted amoung the select set of criteria—including reliability, accuracy, and efficiency—according to which the quality of systems in use in society should be judged.

REFERENCES

• 1: BERLINS,M.AND HODGES, L. 1981. Nationality Bill sets out three new citizenship catego-ries. The London Times (Jan. 15), 1, 15.
• 2: CORBAT~ O, F. J., MERWIN-DAGGETT, M., AND DALEY, R. C. 1962. An experimental time-sharing system. In Proceedings of the Spring Joint Computer Conference. Spartan Books, 335-344.
• 3: FISHLOCK, T. 1981. Delhi press detect racism in Nationality Bill. The London Times (Jan. 20).
• 4: FOTOS, C. P. 1988. British Airways assails U.S. decision to void CRS agreement with American. Aviat. Week Space Tech. (Oct. 24), 78.
• 5: GAO. 1992. Patriot Missile defense: Software problem led to system failure at Dhahran, Saudi Arabia. GAO/IMTEC-92-26, U.S. General Accounting Office, Washington, D.C.
• 6: GRAETTINGER,J.S.AND PERANSON, E. 1981a. The matching program. New Engl. J. Med. 304, 1163-1165.
• 7: GRAETTINGER,J.S.AND PERANSON, E. 1981b. National resident matching program. New Engl. J. Med. 305, 526.
• 8: HUFF,C.AND COOPER, J. 1987. Sex bias in educational software: The effect of designers' stereotypes on the software they design. J. Appl. Soc. Psychol. 17, 519-532.
• 9: JOHNSON,D.G.AND MULVEY, J. M. 1993. Computer decisions: Ethical issues of responsibil-ity and bias. Statistics and Operations Res. Series SOR-93-11, Dept. of Civil Engineering and Operations Research, Princeton Univ., Princeton, N.J.
• 10: P. Leith, Fundamental errors in legal logic programming, The Computer Journal, v.29 n.6, p.545-552, 1986 [doi>10.1093/comjnl/29.6.545]
• 11: MOOR, J. 1985. What is computer ethics? Metaphilosophy 16, 266-275.
• 12: OTT, J. 1988. American Airlines settles CRS dispute with British Airways. Aviat. Week Space Tech. (July 18).
• 13: ROTH, A. E. 1984. The evolution of the labor market for medical interns and residents: A case study in game theory. J. Pol. Econ. 92, 6, 991-1016.
• 14: ROTH, A. E. 1990. New physicians: A natural experiment in market organization. Science 250, (Dec. 14), 1524-1528.
• 15: M. J. Sergot , F. Sadri , R. A. Kowalski , F. Kriwaczek , P. Hammond , H. T. Cory, The British Nationality Act as a logic program, Communications of the ACM, v.29 n.5, p.370-386, May 1986 [doi>10.1145/5689.5920]
• 16: SHIFRIN, C. A. 1985. Justice will weigh suit challenging airlines' computer reservations. Aviat. Week Space Tech. (Mar. 25), 105-111.
• 17: SUDARSHAN,A.AND ZISOOK, S. 1981. National resident matching program. New Engl. J. Med. 305, 525-526.
• 18: TAIB, I. M. 1990. Loophole allows bias in displays on computer reservations systems. Aviat. Week Space Tech. (Feb.), 137.
• 19: WILLIAMS, K. J., WERTH, V. P., AND WOLFF, J. A. 1981. An analysis of the resident match. New Engl. J. Med. 304, 19, 1165-1166.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.1 MODELS AND PRINCIPLES

Additional Classification:
  D. Software
  D.2 SOFTWARE ENGINEERING

  K. Computing Milieux
  K.4 COMPUTERS AND SOCIETY

General Terms:
Design, Human Factors

Keywords:
bias, computer ethics, computers and society, design methods, ethics, human values, social computing, social impact, standards, system design, universal design, values