skip to main content
10.1145/2858036.2858382acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article
Open access

Enhancing Cross-Device Interaction Scripting with Interactive Illustrations

Published: 07 May 2016 Publication History

Abstract

Cross-device interactions involve input and output on multiple computing devices. Implementing and reasoning about interactions that cover multiple devices with a diversity of form factors and capabilities can be complex. To assist developers in programming cross-device interactions, we created DemoScript, a technique that automatically analyzes a cross-device interaction program while it is being written. DemoScript visually illustrates the step-by-step execution of a selected portion or the entire program with a novel, automatically generated cross-device storyboard visualization. In addition to helping developers understand the behavior of the program, DemoScript also allows developers to revise their program by interactively manipulating the cross-device storyboard. We evaluated DemoScript with 8 professional programmers and found that DemoScript significantly improved development efficiency by helping developers interpret and manage cross-device interaction; it also encourages testing to think through the script in a development process.

Supplementary Material

MP4 File (p5482-chi.mp4)

References

[1]
Azza Abouzied, Joseph Hellerstein, and Avi Silberschatz. 2012. DataPlay: interactive tweaking and example-driven correction of graphical database queries. In Proc. UIST '12, 207--218.
[2]
Apple Inc. Swift. 2015. https://developer.apple.com/swift/
[3]
Apple Inc. WatchKit. 2015. https://developer.apple.com/watchkit/
[4]
Ira D. Baxter, Andrew Yahin, Leonardo Moura, Marcelo Sant'Anna, and Lorraine Bier. 1998. Clone Detection Using Abstract Syntax Trees. In Proc. ICSM '98, 368--377.
[5]
Michael Bostock, Vadim Ogievetsky, and Jeffrey Heer. 2011. D3 Data-Driven Documents. IEEE Transactions on Visualization and Computer Graphics 17, 12, 2301--2309.
[6]
Xiang 'Anthony' Chen, Tovi Grossman, Daniel J. Wigdor, and George Fitzmaurice. 2014. Duet: exploring joint interactions on a smart phone and a smart watch. In Proc. CHI '14, 159--168.
[7]
Pei-Yu (Peggy) Chi and Yang Li. 2015. Weave: Scripting Cross-Device Wearable Interaction. In Proc. CHI '15, 3923--3932.
[8]
Stephan Diehl. 2007. Software Visualization: Visualizing the Structure, Behaviour, and Evolution of Software. Springer-Verlag NY, Inc., Secaucus, NJ, USA.
[9]
James Fogarty, Desney Tan, Ashish Kapoor, and Simon Winder. 2008. CueFlik: interactive concept learning in image search. In Proc. CHI '08, 29--38.
[10]
Jens Grubert, Matthias Heinisch, Aaron Quigley, and Dieter Schmalstieg. 2015. MultiFi: Multi Fidelity Interaction with Displays On and Around the Body. In Proc. CHI '15, 3933--3942.
[11]
Google Inc. Android Wear. 2015. https://www.android.com/wear/
[12]
Google Inc. Chrome Apps. 2015. https://developer.chrome.com/apps/
[13]
Björn Hartmann, Scott R. Klemmer, Michael Bernstein, Leith Abdulla, Brandon Burr, Avi Robinson-Mosher, and Jennifer Gee. 2006. Reflective physical prototyping through integrated design, test, and analysis. In Proc. UIST '06, 299--308.
[14]
Björn Hartmann, Loren Yu, Abel Allison, Yeonsoo Yang, and Scott R. Klemmer. 2008. Design as exploration: creating interface alternatives through parallel authoring and runtime tuning. In Proc. UIST '08, 91--100.
[15]
Ariya Hidayat. Esprima: ECMAScript parsing infrastructure for multipurpose analysis. 20 http://esprima.org/
[16]
Steven Houben and Nicolai Marquardt. 2015. WatchConnect: A Toolkit for Prototyping Smartwatch-Centric Cross-Device Applications. In Proc. CHI '15, 1247--1256.
[17]
Melody Y. Ivory and Marti A Hearst. 2001. The state of the art in automating usability evaluation of user interfaces. ACM Comput. Surv. 33, 4, 470--516.
[18]
Thorsten Karrer, Jan-Peter Krämer, Jonathan Diehl, Björn Hartmann, and Jan Borchers. 2011. Stacksplorer: call graph navigation helps increasing code maintenance efficiency. In Proc. UIST '11, 217--224.
[19]
Jun Kato, Sean McDirmid, and Xiang Cao. 2012. DejaVu: integrated support for developing interactive camera-based programs. In Proc. UIST '12, 189--196.
[20]
Scott R. Klemmer, Mark W. Newman, Ryan Farrell, Mark Bilezikjian, and James A. Landay. 2001. The designers' outpost: a tangible interface for collaborative web site. In Proc. UIST '01, 1--10.
[21]
James A. Landay and Brad A. Myers. 1996. Sketching storyboards to illustrate interface behaviors. In Proc. CHI '09, 193--194.
[22]
Brian Lee, Savil Srivastava, Ranjitha Kumar, Ronen Brafman, and Scott R. Klemmer. 2010. Designing with interactive example galleries. In Proc. CHI '10, 2257--2266.
[23]
Yang Li and James A. Landay. 2008. Activity-based prototyping of ubicomp applications for long-lived, everyday human activities. In Proc. CHI '08, 1303--1312.
[24]
Yang Li, Xiang Cao, Katherine Everitt, Morgan Dixon, and James A. Landay. 2010. FrameWire: a tool for automatically extracting interaction logic from paper prototyping tests. In Proc. CHI '10, 503--512.
[25]
Henry Lieberman and Christopher Fry. 1995. Bridging the gulf between code and behavior in programming. In Proc. CHI '95, 480--486.
[26]
James Lin and James A. Landay. 2008. Employing patterns and layers for early-stage design and prototyping of cross-device user interfaces. In Proc. CHI '08, 1313--1322.
[27]
Microsoft Inc. HoloLens. 2015. https://www.microsoft.com/microsoft-hololens/
[28]
Brad Myers, Scott E. Hudson, and Randy Pausch. 2000. Past, present, and future of user interface software tools. ACM Trans. Comput.-Hum. Interact. 7, 1, 3--
[29]
Michael Nebeling, Theano Mintsi, Maria Husmann, and Moira Norrie. 2014. Interactive development of cross-device user interfaces. In Proc. CHI '14, 2793--2802.
[30]
Donald A. Norman and Stephen W. Draper. 1986. User Centered System Design; New Perspectives on Human-Computer Interaction. L. Erlbaum Assoc. Inc., Hillsdale, NJ, USA.
[31]
Stephen Oney and Joel Brandt. 2012. Codelets: linking interactive documentation and example code in the editor. In Proc. CHI '12, 2697--2706.
[32]
Roman Rädle, Hans-Christian Jetter, Mario Schreiner, Zhihao Lu, Harald Reiterer, and Yvonne Rogers. 2015. Spatially-aware or Spatially-agnostic?: Elicitation and Evaluation of User-Defined Cross-Device Interactions. In Proc. CHI '15, 3913--3922.
[33]
Daniel Ritchie, Ankita Arvind Kejriwal, and Scott R. Klemmer. 2011. d.tour: style-based exploration of design example galleries. In Proc. UIST '11, 165--174.
[34]
Sifteo Inc. Sifteo Cubes. 2015. https://www.sifteo.com/
[35]
Michael Toomim, Andrew Begel, and Susan L. Graham. 2004. Managing Duplicated Code with Linked Editing. In Proc. VLHCC '04: the 2004 IEEE Symposium on Visual Languages - Human Centric Computing, 173--180.
[36]
Khai N. Truong, Gillian R. Hayes, and Gregory D. Abowd. 2006. Storyboarding: an empirical determination of best practices and effective guidelines. In Proc. DIS '06, 12--21.
[37]
Jishuo Yang and Daniel Wigdor. 2014. Panelrama: enabling easy specification of cross-device web applications. In Proc. CHI '14, 2783--2792.
[38]
Mark Weiser. 1981. Program slicing. In Proc. ICSE '81: the 5th International Conference on Software engineering, 439--449.

Cited By

View all
  • (2024)Multi Step Knowledge Distillation Framework For Action Recognition in Still Images2024 20th CSI International Symposium on Artificial Intelligence and Signal Processing (AISP)10.1109/AISP61396.2024.10475221(1-7)Online publication date: 21-Feb-2024
  • (2024)The Effect of Individual-Level Factors and Task Features on Interface Design for Rule-Verification Crowdsourcing TasksInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2332031(1-28)Online publication date: 16-Apr-2024
  • (2023)N-euro PredictorProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36108847:3(1-25)Online publication date: 27-Sep-2023
  • Show More Cited By

Index Terms

  1. Enhancing Cross-Device Interaction Scripting with Interactive Illustrations

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    CHI '16: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems
    May 2016
    6108 pages
    ISBN:9781450333627
    DOI:10.1145/2858036
    This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike International 4.0 License.

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 07 May 2016

    Permissions

    Request permissions for this article.

    Check for updates

    Badges

    • Best Paper

    Author Tags

    1. cross-device interaction
    2. interactive illustration
    3. scripting
    4. storyboards

    Qualifiers

    • Research-article

    Conference

    CHI'16
    Sponsor:
    CHI'16: CHI Conference on Human Factors in Computing Systems
    May 7 - 12, 2016
    California, San Jose, USA

    Acceptance Rates

    CHI '16 Paper Acceptance Rate 565 of 2,435 submissions, 23%;
    Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

    Upcoming Conference

    CHI 2025
    ACM CHI Conference on Human Factors in Computing Systems
    April 26 - May 1, 2025
    Yokohama , Japan

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)266
    • Downloads (Last 6 weeks)12
    Reflects downloads up to 20 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Multi Step Knowledge Distillation Framework For Action Recognition in Still Images2024 20th CSI International Symposium on Artificial Intelligence and Signal Processing (AISP)10.1109/AISP61396.2024.10475221(1-7)Online publication date: 21-Feb-2024
    • (2024)The Effect of Individual-Level Factors and Task Features on Interface Design for Rule-Verification Crowdsourcing TasksInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2332031(1-28)Online publication date: 16-Apr-2024
    • (2023)N-euro PredictorProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36108847:3(1-25)Online publication date: 27-Sep-2023
    • (2023)WAMS: A Flexible API for Visual Workspaces Across Multiple SurfacesProceedings of the ACM on Human-Computer Interaction10.1145/35932367:EICS(1-40)Online publication date: 19-Jun-2023
    • (2021)Understanding the Challenges and Needs of Programmers Writing Web Automation Scripts2021 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)10.1109/VL/HCC51201.2021.9576476(1-9)Online publication date: 10-Oct-2021
    • (2021)Still Image Action Recognition Using Ensemble Learning2021 7th International Conference on Web Research (ICWR)10.1109/ICWR51868.2021.9443021(125-129)Online publication date: 19-May-2021
    • (2020)"In VR, everything is possible!": Sketching and Simulating Spatially-Aware Interactive Spaces in Virtual RealityProceedings of the 2020 CHI Conference on Human Factors in Computing Systems10.1145/3313831.3376652(1-16)Online publication date: 21-Apr-2020
    • (2019)User Interface Distribution Method Based on Pi-CalculusInternational Journal of Distributed Systems and Technologies10.4018/IJDST.201907010110:3(1-20)Online publication date: Jul-2019
    • (2019)Effects of Depth Layer Switching between an Optical See-Through Head-Mounted Display and a Body-Proximate DisplaySymposium on Spatial User Interaction10.1145/3357251.3357588(1-9)Online publication date: 19-Oct-2019
    • (2019)FLUIDThe 25th Annual International Conference on Mobile Computing and Networking10.1145/3300061.3345443(1-16)Online publication date: 5-Aug-2019
    • Show More Cited By

    View Options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Login options

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media