Personalized Serious Games for Cognitive Intervention with Lifelog Visual Analytics

ABSTRACT

This paper presents a novel serious game app and a method to cre- ate and integrate personalized game content based on lifelog visual analytics. The main objective is to extract personalized content from visual lifelogs, integrate it into mobile games, and evaluate the effect of personalization on user experience. First, a suite of visual analysis methods is proposed to extract semantic informa- tion from visual lifelogs and discover the association among the lifelog entities. The outcome is dataset that contains augmented and personal lifelog images. Next, a mobile game app is developed that makes use of the dataset as game content. Finally, an experiment is conducted to evaluate user gameplay behaviors in the wild over three months, where a mixture of generic and personalized game content is deployed. It is observed that user adherence is heightened by personalized game content as compared to generic content. Also observed is a higher enjoyment level in personalized than generic game content. The result provides the first empirical evidence of the effect of personalized games on user adherence and preference for cognitive intervention. This work paves the way for effective cognitive training with user-generated content.

References

1. J. A. Anguera, J. Boccanfuso, J. L. Rintoul, O. Al-Hashimi, F. Faraji, J. Janowich, E. Kong, Y. Larraburo, C. Rolle, E. Johnston, and A. Gazzaley. Video game training enhances cognitive control in older adults. Nature, 510(97--101), 2013.
2. J. A. Anguera and A. Gazzaley. Video games, cognitive exercises, and hte enhancement of cognitive abilities. Current Opinion in Behavioral Sciences, 4:160--165, 2015.
3. P. Belchior, M.Marsiske, S. Sisco, A. Yam, and W. Mann. Older adults' engagement with a video game training program. Act. Adapt. Aging, 36:269--279, 2012.
4. A. Bermingham, J. O'Rourke, C. Gurrin, R. Collins, K. Irving, and A. F. Smeaton. Automatically recommending multimedia content for use in group reminiscence therap. In MIIRH'13, pages 49--58, New York, NY, USA, 2013. ACM.
5. E. Berry, N. Kapur, L. Williams, S. Hodges, P. Watson, G. Smyth, J. Srinivasan, R. Smith, B. Wilson, and K. Wood. The use of a wearable camera, sensecam, as a pictorial diary to improve autobiographical memory in a patient with limbic encephalitis: A preliminary report. Neuropsychol Rehabil, 17(4--5):582--601, 2007.
6. K. A. Blocker, T. J. Wright, and W. R. Boot. Gaming preferences of aging generations. Gerontechnology, 12(3):174--184, 2014.
7. M. Bolanos, M. Dimiccoli, and P. Radeva. Toward storytelling from visual lifelogging: An overview. IEEE Trans. Human--Mach. Syst., 47:77--90, 2017.
8. W. R. Boot, D. Souders, N. Charness, K. Blocker, N. Roque, and T. Vitale. The gamification of cognitive training: Older adults' perceptions of and attitudes toward digital game-based interventions. In J. Zhou and G. Salvendy, editors, Lecture Notes in Computer Science, volume 9754 of Human Aspects of IT for the Aged Population. Design for Aging, ITAP 2016. Springer, Cham, 2016.
9. Y. Chen and G. J. Jones. Augmenting human memory using personal lifelogs. In AH'10. ACM, 2010.
10. M. Cotelli, R. Manenti, O. Zanetti, and C. Miniussi. Non-pharmacological intervention for memory decline. Frontiers in Human Neuroscience, 6:No. 46, 2012.
11. M. Csikszentmihalyi. Flow: The psychology of the optimal experience. New York: Harper & Row., 1990.
12. J. R. Finley, W. F. Brewer, and A. S. Benjamin. The effects of end-of-day picture review and a sensor-based picture capture procedure on autobiographical memory using sensecam. Memory, 19(7):796--807, 2011.
13. A. Garcia del Molino. First person view video summarization subject to the user needs. In MM '16, pages 1440--1444. ACM, 2016.
14. N. J. Gates and P. Sachdev. Is cognitive training an effective treatment for preclinical and early alzheimer's disease? Journal of Alzheimer's Disease, 42:S551--S559, 2014.
15. G. Gowans, J. Campbell, N. Alm, R. Dye, A. Astell, and M. Ellis. Designing a multimedia conversation aid for reminiscence therapy in dementia care environments. CHI EA '04, pages 825--836, New York, NY, USA, 2004. ACM.
16. K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR' 16, pages 770--778.
17. G. Kim, L. Sigal, and E. Xing. Joint summarization of large-scale collections of web images and videos for storyline reconstruction. In CVPR, 2014.
18. A. M. Kueider, J. M. Parisi, A. L. Gross, and G. W. Rebok. Computerized cognitive training with older adults: A systematic review. PLoS ONE, 7(7):e40588, 2012.
19. A. Lampit, H. Hallock, and M. Valenzuela. Computerized cognitive training in cognitively healthy older adults: A systematic review and meta-analysis of effect modifiers. PLOS Medicine, 11(11):e1001756, 2014.
20. M. Lee and K. Dey. Providing good memory cues for people with episodic memory impairment. In ASSETS'07, 2007.
21. T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollar, and L. Zitnick. Microsoft coco: Common objects in context. In ECCV'14.
22. Z. Lu and K. Grauman. Story-driven summarization for egocentric video. 2013.
23. H. W. Mahncke, B. B. Connor, J. Appelman, O. N. Ahsanuddin, J. L. Hardy, R. A. Wood, N. M. Joyce, T. Boniske, S. M. Atkins, and M. M. Merzenich. Memory enhancement in healthy older adults using a brain plasticity-based training program: A randomized, controlled study. PNAS, 103(33):12523--28, 2006.
24. V. Manera, G. Ben-Sadoun, and T. Aalbers. Recommendations for the use of serious games in neurodegenerative disorders: 2016 delphi panel. Frontiers in Psychology, 8:1243:1--10, 2017.
25. A. McLaughlin, M. Gandy, J. Allaire, and L. Whitlock. Putting fun into aging? overcoming usability and motivational issues in video games for older adults. Ergonomics in Design, 20(13--20), 2012.
26. A. Mora, C. González, and J. Arnedo-Moreno. Gamification of cognitive training: A crowdsourcing- inspired approach for older adults. In Interacción '16, page Article No. 5, 2016.
27. A. C. Oei and M. D. Patterson. Enhancing cognition with video games: A multiple game training study. PLOS ONE, 8(3):e58546, 2013.
28. G. Oliveira-Barra, M. Bola nos, E. Talavera, A. Due nas, O. Gelonch, and M. Garolera. Serious games application for memory training using egocentric images. Sept. 11--15 2017.
29. C. Peretz, A. D. Korczyn, E. Shatil, V. Aharonson, S. Birnboim, and N. Giladi. Computer-based, personalized cognitive training versus classical computer games: A randomized double-blind prospective trial of cognitive stimulation. Neruoepidemiology, 36:91--99., 2011.
30. V. Pieramico, R. Esposito, S. Cesinaro, V. Frazzini, and S. L. Sensi. Effects of non-pharmacological or pharmacological interventions on cognition and brain plasticity of aging individuals. Front in Syst Neurosci, 8:153: 1--10, 2014.
31. S. Ren, K. He, R. Girshick, and J. Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(6):1137 -- 1149, 2017.
32. J. P. Salmon, S. M. Dolan, R. S. Drake, G. C. Wilson, R. M. Klein, and G. A. Eskes. A survey of video game preferences in adults: Building better games for older adults. Entertainment Computing, 21:45--64, 2017.
33. V. Sarne-Fleischmann, N. Tractinsky, T. Dwolatzky, and I. Rief. Personalized reminiscence therapy for patients with alzheimer's disease using a computerized system. In PETRA '11, pages 48:1--48:4. ACM, 2011.
34. A. Sellen and S. Whittaker. Beyond total capture: A constructive critique of lifelogging. Communications of the ACM, 53(5):70--77, 2010.
35. E. Shatil. Does combined cognitive training and physical activity training enhance cognitive abilities more than either alone? a four-condition randomized controlled trial among healthy older adults. Front. Aging Neurosci., 5:8:1--12, 2013.
36. E. Shatil, A. Metzer, O. Horvitz, and A. Miller. Home-based personalized cognitive training in ms patients: A study of adherence and cognitive performance. NeuroRehabilitation, 26:143--153, 2010.
37. A. R. Silva, S. Pinho, L. M. Macedo, and C. J. Moulin. Benefits of sensecam review on neuropsychological test performance. Am J Prev Med, 44(3):302--307, 2013.
38. P. Siriaraya and C. S. Ang. Recreating living experiences from past memories through virtual worlds for people with dementia. In CHI'14, pages 3977--3986, 2014.
39. V. Subbaraju, Q. Xu, B. Mandal, L. Li, and J.-H. Lim. An empirical approach for automatic face clustering on personal lifelogging images. In ICSIP'17, pages 127--131.
40. C. Szegedy, S. Ioffe, V. Vanhoucke, and A. A. Alemi. Inception-v4, inception-resnet and the impact of residual connections on learning. In AAAI'17, pages 4278--4284.
41. P. Wang and A. F. Smeaton. Using visual lifelogs to automatically characterize everyday activities. Information Sciences, 230:147--161, 2013.
42. S. L. Willis, S. L. Tennstedt, M. Marsiske, K. Ball, J. Elias, K. M. Koepke, J. N. Morris, G. W. Rebok, F. W. Unverzagt, A. M. Stoddard, and E. Wright. Long-term effects of cognitive training on everyday functional outcomes in older adults. JAMA, 296(23):2805--2814, 2006.
43. Q. Xu, V. Subbaraju, A. G. del Molino, J. Lin, F. Fang, J. Lim, L. Li, and V. Chandrasekhar. Visualizing personal lifelog data for deeper insights at the ntcir-13 lifelog-2 task. In NTCIR17, pages 33--39, 2017.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)

    • HCI design and evaluation methods

  • Interaction design

    • Interaction design process and methods

• Software and its engineering

  • Software organization and properties

    • Contextual software domains

      • Virtual worlds software

        • Interactive games