Image browsing, processing, and clustering for participatory sensing: lessons from a DietSense prototype

ABSTRACT

Imagers are an increasingly significant source of sensory observations about human activity and the urban environment. ImageScape is a software tool for processing, clustering, and browsing large sets of images. Implemented as a set of web services with an Adobe Flash-based user interface, it supports clustering by both image features and context tags, as well as re-tagging of images in the user interface. Though expected to be useful in many applications, ImageScape was designed as an analysis component of DietSense, a software system under development at UCLA to support (1) the use of mobile devices for automatic multimedia documentation of dietary choices with just-in-time annotation, (2) efficient post facto review of captured media by participants and researchers, and (3) easy authoring and dissemination of the automatic data collection protocols. A pilot study, in which participants ran software that enabled their phones to autonomously capture images of their plates during mealtime, was conducted using an early prototype of the DietSense system, and the resulting image set used in the creation of ImageScape. ImageScape will support two kinds of users within the DietSense application: The participants in dietary studies will have the ability to easily audit their images, while the recipients of the images, health care professionals managing studies and performing analysis, will be able to rapidly browse and annotate large sets of images.

REFERENCES

• 1: J. Burke, D. Estrin, M. Hansen, A. Parker, N. Ramanathan, S. Reddy, and MB Srivastava. Participatory Sensing. WSW at Sensys, 2006.
• 2: M. Srivastava, M. Hansen, J. Burke, A. Parker, S. Reddy, G. Saurabh, M. Allman, V. Paxson, and D. Estrin. Wireless Urban Sensing Systems. Technical report, Report 65, CENS, UCLA, Apr. 2006., 2006.
• 3: A. Schatzkin, V. Kipnis, R. J. Carroll, D. Midthune, A. F. Subar, S. Bingham, D. A. Schoeller, R. P. Troiano, and L. S. Freedman. A comparison of a food frequency questionnaire with a 24-hour recall for use in an epidemiological cohort study. J of Epidemiology, 32:1054--1062, 2003.
• 4: KS Kubena. Accuracy in dietary assessment: on the road to good science. J Am Diet Assoc, 100(7):777--83, 2000.
• 5: AA Stone, S. Shiffman, JE Schwartz, JE Broderick, and MR Hufford. Patient compliance with paper and electronic diaries. Control Clin Trials, 24(2):182--99, 2003.
• 6: C. H. Kaczkowski, P. J. Jones, and J. Feng. Four day multimedia diet records underestimate energy needs in middle aged and elderly women as determined by doubly-labeled water. Journal of Nutrition, pages 802--805, 2000.
• 7: D. H. Wang, M. Kogashiwa, and S. Kira. Development of a New Instrument for Evaluating Individuals' Dietary Intakes. JADA, pages 1588--1593, 2006.
• 8: A. Farmer, O. Gibson, P. Hayton, K. Bryden, C. Dudley, A. Neil, and L. Tarassenko. A real-time, mobile phone-based telemedicine system to support young adults with type 1 diabetes. Informatics in Primary Care, 13(3):171--178, 2005.
• 9: H. S. P. D. Kim, N. C. P. D. Kim, and S. H. P. D. Ahn. Impact of a Nurse Short Message Service Intervention for Patients With Diabetes. Journal of Nursing Care Quality, 21(3):266--271, 2006.
• 10: DA Williamson, HR Allen, PD Martin, AJ Alfonso, B. Gerald, and A. Hunt. Comparison of digital photography to weighed and visual estimation of portion sizes. J Am Diet Assoc, 103(9):1139--45, 2003.
• 11: I. T. L. Lillegaard, N. C. Øverby, and L. F. Andersen. Can children and adolescents use photographs of food to estimate portion sizes? European Journal of Clinical Nutrition, 59:611--617, 2005.
• 12: S. Hodges, L. Williams, E. Berry, S. Izadi, J. Srinivasan, A. Butler, G. Smyth, N. Kapur, and K. Wood. SenseCam: a Retrospective Memory Aid. Ubicomp, 2006.
• 13: HP Laboratories. The Casual Capture Prototype. http://-hpl.hp.com/, 2004.
• 14: J. C. Platt, M. Czerwinski, and A. Brent. Phototoc: Automatic clustering for browsing personal photographs. IEEE PCM, 1:6--10, 2003.
• 15: Adrian Graham , Hector Garcia-Molina , Andreas Paepcke , Terry Winograd, Time as essence for photo browsing through personal digital libraries, Proceedings of the 2nd ACM/IEEE-CS joint conference on Digital libraries, July 14-18, 2002, Portland, Oregon, USA [doi>10.1145/544220.544301]
• 16: K. Chang, N. Yau, M. Hansen, and D. Estrin. Sensorbase-a centralized repository to slog sensor data. In DCOSS/EAWMS, June 17 2006.
• 17: J. C. Platt, M. Czerwinski, and B. A. Field. Photo-TOC: Automatic Clustering for Browsing Personal Photographs. IEEE PCM, 2003.
• 18: Kerry Rodden , Wojciech Basalaj , David Sinclair , Kenneth Wood, Does organisation by similarity assist image browsing?, Proceedings of the SIGCHI conference on Human factors in computing systems, p.190-197, March 2001, Seattle, Washington, United States [doi>10.1145/365024.365097]
• 19: D. Zhong, H. J. Zhang, and S. F. Chang. Clustering methods for video browsing and annotation. Proc. SPIE, 2670:239--246, 1996.
• 20: Yahoo Inc. FlickR. http://flickr.com/, 2007.
• 21: H. Kang and B. Shneiderman. Visualization Methods for Personal Photo Collections: Browsing and Searching in the PhotoFinder. Human-Computer Interaction Laboratory, University of Maryland, 2000.
• 22: BS Manjunath, J. R. Ohm, VV Vasudevan, and A. Yamada. Color and texture descriptors. Circuits and Systems for Video Technology, IEEE Transactions on, 11(6):703--715, 2001.
• 23: J. R. Parker, Algorithms for Image Processing and Computer Vision, John Wiley & Sons, Inc., New York, NY, 1996

INDEX TERMS

Primary Classification:
  C. Computer Systems Organization
  C.3 SPECIAL-PURPOSE AND APPLICATION-BASED SYSTEMS

General Terms:
Algorithms, Design, Experimentation

Keywords:
data auditing, mobile sensors, urban sensing