Article

Distributed pointing for multimodal collaboration over sketched diagrams

Authors:

Paulo Barthelmess,

David DemirdjianAuthors Info & Claims

ICMI '05: Proceedings of the 7th international conference on Multimodal interfaces

Pages 10 - 17

https://doi.org/10.1145/1088463.1088469

Published: 04 October 2005 Publication History

Abstract

A problem faced by groups that are not co-located but need to collaborate on a common task is the reduced access to the rich multimodal communicative context that they would have access to if they were collaborating face-to-face. Collaboration support tools aim to reduce the adverse effects of this restricted access to the fluid intermixing of speech, gesturing, writing and sketching by providing mechanisms to enhance the awareness of distributed participants of each others' actions.In this work we explore novel ways to leverage the capabilities of multimodal context-aware systems to bridge co-located and distributed collaboration contexts. We describe a system that allows participants at remote sites to collaborate in building a project schedule via sketching on multiple distributed whiteboards, and show how participants can be made aware of naturally occurring pointing gestures that reference diagram constituents as they are performed by remote participants.The system explores the multimodal fusion of pen, speech and 3D gestures, coupled to the dynamic construction of a semantic representation of the interaction, anchored on the sketched diagram, to provide feedback that overcomes some of the intrinsic ambiguities of pointing gestures.

References

[1]

S. Agamanolis, A. Westner, and V. M. B. Jr. Reflection of presence: Toward more natural and responsive telecollaboration. In Proc. SPIE Multimedia Networks, volume 3228A, 1997.

[2]

P. Besl and N. MacKay. A method for registration of 3-d shapes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 14:239--256, 1992.

Digital Library

[3]

S. A. Bly. A use of drawing surfaces in different collaborative settings. In CSCW '88: Proceedings of the 1988 ACM conference on Computer-supported cooperative work, pages 250--256, 1988.

Digital Library

[4]

P. Cohen, M. Johnston, D. McGee, S. Oviatt, J. Pittman, I. Smith, L. Chen, and J. Clow. Quickset: Multimodal interaction for distributed applications. In Proceedings of the Fifth ACM International Multimedia Conference, 1997.

Digital Library

[5]

D. Demirdjian, T. Ko, and T. Darrell. Constraining human body tracking. In Proc. IEEE International Conference on Computer Vision, Nice, France, 2003.

Digital Library

[6]

J. Heiser, B. Tversky, and M. Silverman. Sketeches for and from collaboration. In J. S. Gero, B. Tversky, and T. Knight, editors, Visual and spatial reasoning in design III, pages 69--78. 2004.

[7]

H. Ishii and M. Kobayashi. Clearboard: a seamless medium for shared drawing and conversation with eye contact. In CHI '92: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 525--532, 1992.

Digital Library

[8]

M. Johnston, P. Cohen, D. McGee, S. Oviatt, J. Pittman, and I. Smith. Unification-based multimodal integration. In Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics, 1997.

Digital Library

[9]

E. Kaiser, D. Demirdjian, A. Gruenstein, X. Li, J. Niekrasz, M. Wesson, and S. Kumar. Demo: A multimodal learning interface for sketch, speak and point creation of a schedule chart. In Proceedings of the Sixth International Conference on Multimodal Interfaces (ICMI 2004), pages 329--330, 2004.

Digital Library

[10]

E. Kaiser, A. Olwal, D. McGee, H. Benko, A. Corradini, X. Li, P. Cohen, and S. Feiner. Mutual disambiguation of 3d multimodal interaction in augmented and virtual reality. In Proceedings of the 5th International Conference on Multimodal Interfaces (ICMI 2003), pages 12--19, 2003.

Digital Library

[11]

E. C. Kaiser. Multimodal new vocabulary recognition through speech and handwriting in a whiteboard scheduling application. In Proceedings of the International Conference on Intelligent User Interfaces, pages 51--58, 2005.

Digital Library

[12]

P. Luft, C. Heath, H. Kuzuoka, J. HindMarsh, K. Yamazaki, and S. Oyama. Fractured ecologies: Creating environments for collaboration. Human-Computer Interaction, 18(1 & 2):51--84, 2003.

Digital Library

[13]

I. S. MacKenzie and W. Buxton. Extending fitts' law to two-dimensional tasks. In CHI '92: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 219--226, 1992.

Digital Library

[14]

D. Martin, A. Cheyer, and D. Moran. The Open Agent Architecture: A framework for building distributed software systems. Applied Artificial Intelligence, 13(1/2), 1999.

Digital Library

[15]

O. Morikawa and T. Maesako. Hypermirror: toward pleasant-to-use video mediated communication system. In CSCW '98: Proceedings of the 1998 ACM conference on Computer supported cooperative work, pages 149--158, 1998.

Digital Library

[16]

K.-I. Okada, F. Maeda, Y. Ichikawaa, and Y. Matsushita. Multiparty videoconferencing at virtual social distance: Majic design. In CSCW '94: Proceedings of the 1994 ACM conference on Computer supported cooperative work, pages 385--393, 1994.

Digital Library

[17]

S. Oviatt. Multimodal interfaces. In J. Jacko and A. Sears, editors, The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications, chapter 14, pages 286--304. Lawrence Erlbaum Assoc., Mahwah, NJ, 2003.

Digital Library

[18]

J. C. Tang and S. Minneman. Videowhiteboard: video shadows to support remote collaboration. In CHI '91: Proceedings of the SIGCHI conference on Human factors in computing systems, pages 315--322, 1991.

Digital Library

Cited By

Ferreira RPereira RBianchi Ida Silva M(2021)Decision Factors for Remote Work Adoption: Advantages, Disadvantages, Driving Forces and ChallengesJournal of Open Innovation: Technology, Market, and Complexity10.3390/joitmc70100707:1(70)Online publication date: Mar-2021
https://doi.org/10.3390/joitmc7010070
Praharaj SScheffel MDrachsler HSpecht M(2021)Literature Review on Co-Located Collaboration Modeling Using Multimodal Learning Analytics—Can We Go the Whole Nine Yards?IEEE Transactions on Learning Technologies10.1109/TLT.2021.309776614:3(367-385)Online publication date: 1-Jun-2021
https://doi.org/10.1109/TLT.2021.3097766
Defays ALeclercq PNyssen A(2018)Influence des communications multimodales sur le common groundInfluence of multimodal communications on the common ground. A suggested methodologyInterfaces numériques10.25965/interfaces-numeriques.21772:3Online publication date: 24-Jan-2018
https://doi.org/10.25965/interfaces-numeriques.2177
Show More Cited By

Index Terms

Distributed pointing for multimodal collaboration over sketched diagrams

Recommendations

Multimodal Collaboration in Expository Discourse: Verbal and Nonverbal Moves Alignment
Speech and Computer
Abstract
The paper explores multimodal collaboration in expository discourse considering verbal and nonverbal moves used by the participants in turn-taking. It reports the results of an experiment which tested speech, gesture and gaze alignment as affected ...
Collaborative multimodal photo annotation over digital paper
ICMI '06: Proceedings of the 8th international conference on Multimodal interfaces

The availability of metadata annotations over media content such as photos is known to enhance retrieval and organization, particularly for large data sets. The greatest challenge for obtaining annotations remains getting users to perform the large ...
Using self-context for multimodal detection of head nods in face-to-face interactions
ICMI '12: Proceedings of the 14th ACM international conference on Multimodal interaction

Head nods occur in virtually every face-to-face discussion. As part of the backchannel domain, they are not only used to express a 'yes', but also to display interest or enhance communicative attention. Detecting head nods in natural interactions is a ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICMI '05: Proceedings of the 7th international conference on Multimodal interfaces

October 2005

344 pages

ISBN:1595930280

DOI:10.1145/1088463

General Chairs:
Gianni Lazzari
ITC-irst, Trento (Italy)
,
Fabio Pianesi
ITC-irst, Trento (Italy)
,
Program Chairs:
James Crowley
I.N.P. Grenoble (France)
,
Kenji Mase
Nagoya University (Japan)
,
Sharon Oviatt
Oregon Health & Sciences University

Copyright © 2005 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 October 2005

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

ICMI05

Sponsor:

ICMI05: Seventh International Conference on Multimodal Interfaces 2005

October 4 - 6, 2005

Torento, Italy

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

14
Total Citations
View Citations
573
Total Downloads

Downloads (Last 12 months)1
Downloads (Last 6 weeks)0

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Ferreira RPereira RBianchi Ida Silva M(2021)Decision Factors for Remote Work Adoption: Advantages, Disadvantages, Driving Forces and ChallengesJournal of Open Innovation: Technology, Market, and Complexity10.3390/joitmc70100707:1(70)Online publication date: Mar-2021
https://doi.org/10.3390/joitmc7010070
Praharaj SScheffel MDrachsler HSpecht M(2021)Literature Review on Co-Located Collaboration Modeling Using Multimodal Learning Analytics—Can We Go the Whole Nine Yards?IEEE Transactions on Learning Technologies10.1109/TLT.2021.309776614:3(367-385)Online publication date: 1-Jun-2021
https://doi.org/10.1109/TLT.2021.3097766
Defays ALeclercq PNyssen A(2018)Influence des communications multimodales sur le common groundInfluence of multimodal communications on the common ground. A suggested methodologyInterfaces numériques10.25965/interfaces-numeriques.21772:3Online publication date: 24-Jan-2018
https://doi.org/10.25965/interfaces-numeriques.2177
Schneider JBörner Dvan Rosmalen PSpecht MZhang ZCohen PBohus DHoraud RMeng H(2015)Presentation Trainer, your Public Speaking Multimodal CoachProceedings of the 2015 ACM on International Conference on Multimodal Interaction10.1145/2818346.2830603(539-546)Online publication date: 9-Nov-2015
https://dl.acm.org/doi/10.1145/2818346.2830603
LaViola Jr. JBuchanan SPittman C(2014)Multimodal Input for Perceptual User InterfacesInteractive Displays10.1002/9781118706237.ch9(285-312)Online publication date: 12-Jul-2014
https://doi.org/10.1002/9781118706237.ch9
Ajmera RPandit PBorgaonkar SMadhvanath STripathi SJoshi A(2013)Enhancing mobile phones to support collaborative communication for micro-entrepreneurs in emerging economiesProceedings of the 11th Asia Pacific Conference on Computer Human Interaction10.1145/2525194.2525212(142-149)Online publication date: 24-Sep-2013
https://dl.acm.org/doi/10.1145/2525194.2525212
Crowe MLivingstone D(2009)Collaborative Web-Browsing in a Virtual WorldProceedings of the 2009 Conference in Games and Virtual Worlds for Serious Applications10.1109/VS-GAMES.2009.38(221-222)Online publication date: 23-Mar-2009
https://dl.acm.org/doi/10.1109/VS-GAMES.2009.38
Barthelmess POviatt S(2008)Multimodal InterfacesHCI Beyond the GUI10.1016/B978-0-12-374017-5.00012-2(391-444)Online publication date: 2008
https://doi.org/10.1016/B978-0-12-374017-5.00012-2
Tory MStaub-French SPo BWu F(2008)Physical and Digital Artifact-Mediated Coordination in Building DesignComputer Supported Cooperative Work10.1007/s10606-008-9077-417:4(311-351)Online publication date: 1-Aug-2008
https://dl.acm.org/doi/10.1007/s10606-008-9077-4
Rienks RNijholt ABarthelmess P(2008)Pro-active meeting assistants: attention please!AI & Society10.1007/s00146-007-0135-023:2(213-231)Online publication date: 7-Aug-2008
https://dl.acm.org/doi/10.1007/s00146-007-0135-0
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten