Augmented Reality

Performing a touchscreen gesture after listening to a soundWearable technologies that augment human senses have the potential to increase the accessibility of interactions in the physical world. Our work explores the use of augmented reality to support people with visual or cognitive impairments, and the deaf and hard of hearing. One major focus is the HandSight project, which investigates how wearable micro-cameras can be used to augment a blind or visually impaired user’s sense of touch with computer vision. The goal is to support an array of activities of daily living by sensing and feeding back non-tactile information (e.g., color, printed text, patterns) about an object as it is touched. We have also studied language and sound support on head-mounted displays, as well as social acceptability issues with such devices.

Funding

The HandSight research is funded by the Department of Defense (2014-2017). A Google Faculty Research Award (2014-2015) has supported some of the head-mounted display accessibility work.

Related Publications

Profita, H., Albaghli, R., Findlater, L., Jaeger, P., Kane, S. 2016. The AT Effect: How Disability Affects the Perceived Social Acceptability of Head-Mounted Display Use. Proc. SIGCHI Conference on Human Factors in Computing Systems (CHI 2016), 10 pages. To appear.

Findlater, L., Stearns, L., Du, R., Oh, U., Ross, D., Chellappa, R., Froehlich, J.E. 2015. Supporting everyday activities for persons with visual impairments through computer vision-augmented touch. Proceedings of ACM SIGACCESS Conference on Computers and Accessibility (ASSETS’14), 383-384.

Williams, K., Findlater, L., & Moffatt, K. 2015. Designing Conversation Cues on a Head-Mounted Display to Support Persons with Aphasia. Proceedings of CHI 2015, 231-240.

Jain, D., Findlater, L., Gilkeson, J., Holland, B., Duraiswami, R., Zotkin, D., Vogler, C., & Froehlich, J. 2015. Head-Mounted Display Visualizations to Support Sound Awareness for the Deaf and Hard of Hearing. Proceedings of CHI 2015, 241-250.

Stearns, L., Du, R., Oh, U., Wang, Y., Chellappa, R., Findlater, L., Froehlich, J.E. 2014. The Design and Preliminary Evaluation of a Finger-Mounted Camera and Feedback System to Enable Reading of Printed Text for the Blind. Proceedings of Computer Vision-ECCV 2014 Workshops, 615-631.