Awareness, training and trust in interaction with adaptive spam filters

ABSTRACT

Even though adaptive (trainable) spam filters are a common example of systems that make (semi-)autonomous decisions on behalf of the user, trust in these filters has been underexplored. This paper reports a study of usage of spam filters in the daily workplace and user behaviour in training these filters (N=43). User observation, interview and survey techniques were applied to investigate attitudes towards two types of filters: a user-adaptive (trainable) and a rule-based filter. While many of our participants invested extensive effort in training their filters, training did not influence filter trust. Instead, the findings indicate that users' filter awareness and understanding seriously impacts attitudes and behaviour. Specific examples of difficulties related to awareness of filter activity and adaptivity are described showing concerns relevant to all adaptive and (semi-)autonomous systems that rely on explicit user feedback.

REFERENCES

• 1: Sherman R. Alpert , John Karat , Clare-Marie Karat , Carolyn Brodie , John G. Vergo, User Attitudes Regarding a User-Adaptive eCommerce Web Site, User Modeling and User-Adapted Interaction, v.13 n.4, p.373-396, November 2003 [doi>10.1023/A:1026201108015]
• 2: Henriette Cramer , Vanessa Evers , Satyan Ramlal , Maarten Someren , Lloyd Rutledge , Natalia Stash , Lora Aroyo , Bob Wielinga, The effects of transparency on trust in and acceptance of a content-based art recommender, User Modeling and User-Adapted Interaction, v.18 n.5, p.455-496, November 2008 [doi>10.1007/s11257-008-9051-3]
• 3: Mary T. Dzindolet , Scott A. Peterson , Regina A. Pomranky , Linda G. Pierce , Hall P. Beck, The role of trust in automation reliance, International Journal of Human-Computer Studies, v.58 n.6, p.697-718, June 2003 [doi>10.1016/S1071-5819(03)00038-7]
• 4: B. J. Fogg , Hsiang Tseng, The elements of computer credibility, Proceedings of the SIGCHI conference on Human factors in computing systems: the CHI is the limit, p.80-87, May 15-20, 1999, Pittsburgh, Pennsylvania, United States [doi>10.1145/302979.303001]
• 5: Jonathan L. Herlocker , Joseph A. Konstan , Loren G. Terveen , John T. Riedl, Evaluating collaborative filtering recommender systems, ACM Transactions on Information Systems (TOIS), v.22 n.1, p.5-53, January 2004 [doi>10.1145/963770.963772]
• 6: Lee J.D. and See K.A. Trust in automation: designing for appropriate reliance. Hum. Fact 42, 1 (2004), 50--80.
• 7: Anthony Jameson , Eric Schwarzkopf, Pros and Cons of Controllability: An Empirical Study, Proceedings of the Second International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, p.193-202, May 29-31, 2002
• 8: Jian, J.Y., Bisantz, A.M., and Drury, C.G. Foundations for an Empirically Determined Scale of Trust in Automated Systems. Int. J. Cog. Erg. 4, 1 (2000), 53--71.
• 9: Jøsang, A. and Lo Presti, S. Analysing the relationship between risk and trust. In Proc. Trust Man. (2004).
• 10: Pattie Maes, Agents that reduce work and information overload, Communications of the ACM, v.37 n.7, p.30-40, July 1994 [doi>10.1145/176789.176792]
• 11: Muir B.M. and Moray N. Trust in automation. Part II. Ergonomics 39, (1996), 429--460.
• 12: Rashmi Sinha , Kirsten Swearingen, The role of transparency in recommender systems, CHI '02 extended abstracts on Human factors in computing systems, April 20-25, 2002, Minneapolis, Minnesota, USA [doi>10.1145/506443.506619]
• 13: Joe Tullio , Anind K. Dey , Jason Chalecki , James Fogarty, How it works: a field study of non-technical users interacting with an intelligent system, Proceedings of the SIGCHI conference on Human factors in computing systems, April 28-May 03, 2007, San Jose, California, USA [doi>10.1145/1240624.1240630]
• 14: Venkatesh, V., Morris, M.G., Davis, G.B. and Davis, F.D. User Acceptance of Information Technology. MIS Quarterly 27, 3 (2003), 425--478.
• 15: Patricia Victor , Chris Cornelis , Martine De Cock , Paulo Pinheiro da Silva, Gradual trust and distrust in recommender systems, Fuzzy Sets and Systems, v.160 n.10, p.1367-1382, May, 2009 [doi>10.1016/j.fss.2008.11.014]
• 16: Annika Wærn, User Involvement in Automatic Filtering: An Experimental Study, User Modeling and User-Adapted Interaction, v.14 n.2-3, p.201-237, June 2004 [doi>10.1023/B:USER.0000028984.13876.9b]

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.5 INFORMATION INTERFACES AND PRESENTATION (I.7)
      H.5.m Miscellaneous

General Terms:
Human Factors

Keywords:
adaptivity, autonomy, filters, spam, trainable systems