CoCo: Collaboration Coach for Understanding Team Dynamics during Video Conferencing

Abstract

We present and discuss a fully-automated collaboration system, CoCo, that allows multiple participants to video chat and receive feedback through custom video conferencing software. After a conferencing session, a virtual feedback assistant provides insights on the conversation to participants. CoCo automatically pulls audial and visual data during conversations and analyzes the extracted streams for affective features, including smiles, engagement, attention, as well as speech overlap and turn-taking. We validated CoCo with 39 participants split into 10 groups. Participants played two back-to-back team-building games, Lost at Sea and Survival on the Moon, with the system providing feedback between the two. With feedback, we found a statistically significant change in balanced participation---that is, everyone spoke for an equal amount of time. There was also statistically significant improvement in participants' self-evaluations of conversational skills awareness, including how often they let others speak, as well as of teammates' conversational skills. The entire framework is available at https://github.com/ROC-HCI/CollaborationCoach_PostFeedback.

References

1. 1996. Lost at Sea. (1996). http://insight.typepad.co.uk/lost_at_sea.pdf
2. 2011. Siri. (2011). https://en.wikipedia.org/wiki/Siri
3. 2015. Video Conferencing Trends 2016. http://resources.idgenterprise.com/original/AST-0162982_Video_Conferencing_Trends_of_2016.pdf
4. 2016. Affectiva. http://www.affectiva.com/. (2016). Accessed: 2016-10-30.
5. Brandon Amos, Bartosz Ludwiczuk, and Mahadev Satyanarayanan. 2016. OpenFace: A general-purpose face recognition library with mobile applications. Technical Report. Technical report, CMU-CS-16-118, CMU School of Computer Science.
6. Patricia B. Barger and Alicia A. Grandey. 2006. Service with a Smile and Encounter Satisfaction: Emotional Contagion and Appraisal Mechanisms. The Academy of Management Journal 49, 6 (2006), 1229--1238. http://www.jstor.org/stable/20159829
7. Sigal G. Barsade. 2002. The Ripple Effect: Emotional Contagion and Its Influence on Group Behavior. Administrative Science Quarterly 47, 4 (2002), 644--675. http://www.jstor.org/stable/3094912
8. JÃľrÃľmy Bauchet, HÃľlÃÍne Pigot, Sylvain Giroux, Dany Lussier-Desrochers, Yves Lachapelle, and Mounir Mokhtari. [n. d.]. Designing judicious interactions for cognitive assistance: the acts of assistance approach. In Proceedings of the 11th international ACM SIGACCESS conference on Computers and accessibility. ACM, 11--18. https://doi.org/10.1145/1639642.1639647
9. Paul Boersma and David Weenink. 2016. Praat: doing phonetics by computer. http://www.praat.org/. (2016). Accessed: 2016-06-01.
10. LouAnne E. Boyd, Alejandro Rangel, Helen Tomimbang, Andrea Conejo-Toledo, Kanika Patel, Monica Tentori, and Gillian R. Hayes. 2016. SayWAT: Augmenting Face-to-Face Conversations for Adults with Autism. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI ‘16). ACM, New York, NY, USA, 4872--4883. https://doi.org/10.1145/2858036.2858215
11. Erin Bradner, Gloria Mark, and Tammie D. Hertel. 2005. Team size and technology fit: Participation, awareness, and rapport in distributed teams. IEEE Transactions on Professional Communication 48, 1 (2005), 68--77. https://doi.org/10.1109/TPC.2005.843299
12. T. Prevett. C. Wickens. 1995. Exploring the dimensions of egocentricity in aircraft navigation displays: influences on local guidance and global situation awareness. Journal of Experimental Psychology: Applied, Neuilly-Sur-Seine, France.
13. Dan Calacci, Oren Lederman, David Shrier, and Alex Sandy Pentland. 2016. Breakout: An Open Measurement and Intervention Tool for Distributed Peer Learning Groups. CoRR abs/1607.01443 (2016). http://arxiv.org/abs/1607.01443
14. Jean Carletta, Simone Ashby, Sebastien Bourban, Mike Flynn, Mael Guillemot, Thomas Hain, Jaroslav Kadlec, Vasilis Karaiskos, Wessel Kraaij, Melissa Kronenthal, Guillaume Lathoud, Mike Lincoln, Agnes Lisowska, Iain McCowan, Wilfried Post, Dennis Reidsma, and Pierre Wellner. 2006. The AMI Meeting Corpus: A Pre-announcement. In Proceedings of the Second International Conference on Machine Learning for Multimodal Interaction (MLMI‘05). Springer-Verlag, Berlin, Heidelberg, 28--39. https://doi.org/10.1007/11677482_3
15. Samuele Centorrino, Elodie Djemai, Astrid Hopfensitz, Manfred Milinski, and Paul Seabright. 2015. Honest signaling in trust interactions: smiles rated as genuine induce trust and signal higher earning opportunities. Evolution and Human Behavior 36, 1 (2015), 8--16. https://doi.org/10.1016/j.evolhumbehav.2014.08.001
16. K. M. Cohen. 1982. Speaker interaction: Video teleconference versus face-to-face meetings (Teleconferencing and Electronic Communications). ACM, University of Wisconsin.
17. RALPH D'AGOSTINO and Egon S Pearson. 1973. Tests for departure from normality. Empirical results for the distributions of b 2 andâĹŽ b. Biometrika 60, 3 (1973), 613--622.
18. Ralph B d'Agostino. 1971. An omnibus test of normality for moderate and large size samples. Biometrika 58, 2 (1971), 341--348.
19. Ionut Damian, Chiew Seng (Sean) Tan, Tobias Baur, Johannes SchÃűning, Kris Luyten, and Elisabeth AndrÃľ. [n. d.]. Augmenting Social Interactions: Realtime Behavioural Feedback using Social Signal Processing Techniques. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 565--574. https://doi.org/10.1145/2702123.2702314
20. Richard W. DeVaul, Alex “Sandy” Pentland, and Vicka R. Corey. 2003. The Memory Glasses: Subliminal vs. Overt Memory Support with Imperfect Information. In Proceedings of the 7th IEEE International Symposium on Wearable Computers (ISWC ‘03). IEEE Computer Society, Washington, DC, USA, 146--. http://dl.acm.org/citation.cfm?id=946249.946907
21. J. J. Diehl and R. Paul. 2009. The assessment and treatment of prosodic disorders and neurological theories of prosody. Int J Speech Lang Pathol 11, 4 (2009), 287--92. https://doi.org/10.1080/17549500902971887
22. Joan Morris DiMicco, Anna Pandolfo, and Walter Bender. 2004. Influencing group participation with a shared display. Proceedings of the 2004 ACM conference on Computer supported cooperative work - CSCW ‘04 (2004), 614--623. https://doi.org/10.1145/1031607.1031713
23. Norah E. Dunbar and Judee K. Burgoon. 2005. Perceptions of power and interactional dominance in interpersonal relationships. Journal of Social and Personal Relationships 22, 2 (2005), 207--233. https://doi.org/10.1177/0265407505050944 arXiv:http://dx.doi.org/10.1177/0265407505050944
24. Jay Hall and Wilfred Harvey Watson. 1970. Survival on the moon. Human relations (1970). https://www.psychologicalscience.org/observer/nasa-exercise
25. Laura A. Hambley, Thomas A. O'Neill, and Theresa J B Kline. 2007. Virtual team leadership: The effects of leadership style and communication medium on team interaction styles and outcomes. Organizational Behavior and Human Decision Processes 103, 1 (2007), 1--20. https://doi.org/10.1016/j.obhdp.2006.09.004
26. Crystal L. Hoyt and Jim Blascovich. 2003. Transformational and Transactional Leadership in Virtual and Physical Environments. Small Group Research 34, 6 (2003), 678--715. https://doi.org/10.1177/1046496403257527 arXiv:http://dx.doi.org/10.1177/1046496403257527
27. H. Hung, Yan Huang, G. Friedland, and D. Gatica-Perez. 2011. Estimating Dominance in Multi-Party Meetings Using Speaker Diarization. Trans. Audio, Speech and Lang. Proc. 19, 4 (May 2011), 847--860. https://doi.org/10.1109/TASL.2010.2066267
28. Inseok HWANG, Chungkuk YOO, Chanyou HWANG, Dongsun YIM, Youngki LEE, Singapore Management University, Chulhong MIN, Hjohn KIM, and Junehwa SONG. 2014. TalkBetter: Family-driven Mobile Intervention Care for Children with Language Delay. (2014), 1283. https://doi.org/10.1145/2531602.2531668
29. Jeroen Janssen, Gijsbert Erkens, and Gellof Kanselaar. 2007. Visualization of Agreement and Discussion Processes During Computer-supported Collaborative Learning. Comput. Hum. Behav. 23, 3 (May 2007), 1105--1125. https://doi.org/10.1016/j.chb.2006.10.005
30. Jeroen Janssen, Gijsbert Erkens, Gellof Kanselaar, and Jos Jaspers. 2007. Visualization of Participation: Does It Contribute to Successful Computer-supported Collaborative Learning? Comput. Educ. 49, 4 (Dec. 2007), 1037--1065. https://doi.org/10.1016/j.compedu.2006.01.004
31. Dinesh Babu Jayagopi, Hayley Hung, Chuohao Yeo, and Daniel Gatica-Perez. 2009. Modeling Dominance in Group Conversations Using Nonverbal Activity Cues. Trans. Audio, Speech and Lang. Proc. 17, 3 (March 2009), 501--513. https://doi.org/10.1109/TASL.2008.2008238
32. Vaiva Kalnikaité, Patrick Ehlen, and Steve Whittaker. 2012. Markup As You Talk: Establishing Effective Memory Cues While Still Contributing to a Meeting. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work (CSCW ‘12). ACM, New York, NY, USA, 349--358. https://doi.org/10.1145/2145204.2145260
33. Taemie Kim, Pamela Hinds, and Alex Pentland. 2012. Awareness As an Antidote to Distance: Making Distributed Groups Cooperative and Consistent. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work (CSCW ‘12). ACM, New York, NY, USA, 1237--1246. https://doi.org/10.1145/2145204.2145391
34. G. Ross. Lawford. 2003. Beyond success: Achieving synergy in teamwork. The Journal for Quality and Participation 26, 3 (Fall 2003), 23--27. https://search.proquest.com/docview/219147502?accountid=13567
35. Yale Child Jane L. Mcsweeny Ami Klin Donald J. Cohen Fred R. Volkmar Lawrence D. Shriberg, Rhea Paul. [n. d.]. Speech and prosody characteristics of adolescents and adults with high functioning autism and Asperger syndrome. In Journal of Speech, Language, and Hearing Research. http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.385.7116
36. Harold J. Leavitt. 1951. Some effects of certain communication patterns on group performance. Journal of abnormal psychology (1951), 38--50.
37. Gilly Leshed. 2009. Automated language-based feedback for teamwork behaviors. Ph.D. Dissertation. https://search.proquest.com/docview/304872230?accountid=13567 Copyright - Database copyright ProQuest LLC; ProQuest does not claim copyright in the individual underlying works; Last updated - 2016-06-03.
38. Gilly Leshed, Diego Perez, Jeffrey T. Hancock, Dan Cosley, Jeremy Birnholtz, Soyoung Lee, Poppy L. McLeod, and Geri Gay. 2009. Visualizing Real-time Language-based Feedback on Teamwork Behavior in Computer-mediated Groups. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (2009), 537--546. https://doi.org/10.1145/1518701.1518784
39. Bill Mark and R Perrault. 2005. Calo: Cognitive assistant that learns and organizes. (2005).
40. Poppy Lauretta McLeod, Jeffrey K. Liker, and Sharon A. Lobel. 1992. Process Feedback in Task Groups: An Application of Goal Setting. The Journal of Applied Behavioral Science 28, 1 (1992), 15--41. https://doi.org/10.1177/0021886392281003 arXiv:http://dx.doi.org/10.1177/0021886392281003
41. Janice Nadler. 2003. Rapport in negotiation and conflict resolution. Marq. L. Rev. 285, 1990 (2003), 875--882. https://doi.org/10.1525/sp.2007.54.1.23.
42. Eyal Ofek, Shamsi T. Iqbal, and Karin Strauss. 2013. Reducing Disruption from Subtle Information Delivery During a Conversation: Mode and Bandwidth Investigation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ‘13). ACM, New York, NY, USA, 3111--3120. https://doi.org/10.1145/2470654.2466425
43. Jeffery R. Ringer. 1985. Pardon Me, Can I Talk Now?: A Look at the Roles of Interruptions in Conversation. Annual Meeting of the Organization for the Study of Communication, Language, and Gneder (1985), 1--29.
44. Marcello Russo. 2012. Diversity in goal orientation, team performance, and internal team environment. Equality, Diversity and Inclusion: An International Journal 31, 2 (2012), 124--143. https://doi.org/10.1108/02610151211202781 arXiv:http://dx.doi.org/10.1108/02610151211202781
45. Harvey Sacks, Emanuel a Schegloff, and Gail Jefferson. 1974. A simplest systematics for the organization of turn taking for conversation. (1974), 696--735 pages. https://doi.org/10.2307/412243
46. M. Satyanarayanan. 2004. From the Editor in Chief: Augmenting Cognition. IEEE Pervasive Computing 3, 2 (2004), 4--5. https://doi.org/10.1109/MPRV.2004.1316809
47. Charles R. Scherl and Jay Haley. 2000. Computer Monitor Supervision: A Clinical Note. The American Journal of Family Therapy 28, 3 (2000), 275--282. https://doi.org/10.1080/01926180050081702 arXiv:http://dx.doi.org/10.1080/01926180050081702
48. Mark Schittekatte and Alain Van Hiel. 1996. Effects of Partially Shared Information and Awareness of Unshared Information on Information Sampling. Small Group Research 27, 3 (1996), 431--449. https://doi.org/10.1177/1046496496273006 arXiv:http://dx.doi.org/10.1177/1046496496273006
49. Abigail J. Sellen. 1992. Speech Patterns in Video-mediated Conversations. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ‘92). ACM, New York, NY, USA, 49--59. https://doi.org/10.1145/142750.142756
50. D. Tanen. 1995. The Power of Talk: Who Gets Heard and Why. Harvard Business Review (1995).
51. Deborah Tannen. 1997. Gender and Discourse. Discourse (1997), 548--567. https://doi.org/10.2307/2655317
52. M. Iftekhar Tanveer, Emy Lin, and Mohammed (Ehsan) Hoque. 2015. Rhema: A Real-Time In-Situ Intelligent Interface to Help People with Public Speaking. In Proceedings of the 20th International Conference on Intelligent User Interfaces (IUI ‘15). ACM, New York, NY, USA, 286--295. https://doi.org/10.1145/2678025.2701386
53. Yla R. Tausczik and James W. Pennebaker. 2013. Improving teamwork using real-time language feedback. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI ‘13 (2013), 459--468. https://doi.org/10.1145/2470654.2470720
54. Monica Tentori and Gillian R. Hayes. [n. d.]. Designing for interaction immediacy to enhance social skills of children with autism. In Proceedings of the 12th ACM international conference on Ubiquitous computing. ACM, 51--60. https://doi.org/10.1145/1864349.1864359
55. Linda Tickle-Degnen and Robert Rosenthal. 1990. The Nature of Rapport and Its Nonverbal Correlates. Psychological Inquiry 1, 4 (1990), 285--293. http://www.jstor.org/stable/1449345
56. Janet van der Linden, Rose Johnson, Jon Bird, Yvonne Rogers, and Erwin Schoonderwaldt. 2011. Buzzing to Play: Lessons Learned from an in the Wild Study of Real-time Vibrotactile Feedback. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ‘11). ACM, New York, NY, USA, 533--542. https://doi.org/10.1145/1978942.1979017
57. Joseph B. Walther and Lisa C. Tidwell. 1995. Nonverbal cues in computer-mediated communication, and the effect of chronemics on relational communication. J. Org. Computing 5, 4 (1995), 355--378. http://dblp.uni-trier.de/db/journals/jocec/jocec5.html
58. C. Wickens. 1996. Situation awareness: impact of automation and display technology. In NATO AGARD Aerospace Medical Panel Symposium on Situation Awareness.
59. Anita Williams Woolley, Christopher F. Chabris, Alex Pentland, Nada Hashmi, and Thomas W. Malone. 2010. Evidence for a Collective Intelligence Factor in the Performance of Human Groups. Science 330, 6004 (2010), 686--688. https://doi.org/10.1126/science.1193147 arXiv:http://science.sciencemag.org/content/330/6004/686.full.pdf
60. Qianli Xu, Michal Mukawa, Liyuan Li, Joo Hwee Lim, Cheston Tan, Shue Ching Chia, Tian Gan, and Bappaditya Mandal. [n. d.]. Exploring users' attitudes towards social interaction assistance on Google Glass. In Proceedings of the 6th Augmented Human International Conference. ACM, 9--12. https://doi.org/10.1145/2735711.2735831

Index Terms

• Human-centered computing

  • Collaborative and social computing

    • Collaborative and social computing design and evaluation methods