Sketchplore: Sketch and Explore with a Layout Optimiser

ABSTRACT

This paper studies a novel concept for integrating real-time design optimisation to a sketching tool. Although optimisation methods can attack very complex design problems, their insistence on precise objectives and a point optimum is a poor fit with sketching practices. Sketchplorer is a multitouch sketching tool that uses a real-time layout optimiser. It automatically infers the designer's task to search for both local improvements to the current design and global (radical) alternatives. Using predictive models of sensorimotor performance and perception, these suggestions steer the designer toward more usable and aesthetic layouts without overriding the designer or demanding extensive input.

References

1. Seok-Hyung Bae, Ravin Balakrishnan, and Karan Singh. 2008. ILoveSketch: As-natural-as-possible Sketching System for Creating 3D Curve Models. In Proceedings of the 21st Annual ACM Symposium on User Interface Software and Technology (UIST '08). ACM, New York, NY, USA, 151--160. DOI: http://dx.doi.org/10.1145/1449715.1449740
2. Gilles Bailly, Antti Oulasvirta, Timo Kötzing, and Sabrina Hoppe. 2013. MenuOptimizer: Interactive Optimization of Menu Systems. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (UIST '13). ACM, New York, NY, USA, 331--342. DOI: http://dx.doi.org/10.1145/2501988.2502024
3. Helen Y. Balinsky. 2006. Evaluating interface aesthetics: measure of symmetry. In Electronic Imaging 2006. International Society for Optics and Photonics, 607--608.
4. Helen Y. Balinsky, Anthony J. Wiley, and Matthew C. Roberts. 2009. Aesthetic Measure of Alignment and Regularity. In Proceedings of the 9th ACM Symposium on Document Engineering (DocEng '09). ACM, New York, NY, USA, 56--65. DOI: http://dx.doi.org/10.1145/1600193.1600207
5. Rainer E. Burkhard and J. Offerman. 1977. Entwurf von schreibmaschinentastaturen mittels quadratischer zuordnungsprobleme. Operations Res 21 (1977), B121--B132.
6. Bill Buxton. 2007. Sketching User Experiences: Getting the Design Right and the Right Design. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA.
7. Daniel Cohen-Or, Olga Sorkine, Ran Gal, Tommer Leyvand, and Ying-Qing Xu. 2006. Color Harmonization. In ACM SIGGRAPH 2006 Papers (SIGGRAPH '06). ACM, New York, NY, USA, 624--630. DOI: http://dx.doi.org/10.1145/1179352.1141933
8. Nigel Cross. 2004. Expertise in design: an overview. Design studies 25, 5 (2004), 427--441.
9. Amine Drira, Henri Pierreval, and Sonia Hajri-Gabouj. 2007. Facility layout problems: A survey. Annual Reviews in Control 31, 2 (2007), 255--267.
10. Karim El Batran and Mark D. Dunlop. 2014. Enhancing KLM (Keystroke-level Model) to Fit Touch Screen Mobile Devices. In Proceedings of the 16th International Conference on Human-computer Interaction with Mobile Devices & Services (MobileHCI '14). ACM, New York, NY, USA, 283--286. DOI:http://dx.doi.org/10.1145/2628363.2628385
11. Steven K. Feiner. 1988. A Grid-based Approach to Automating Display Layout. In Proceedings on Graphics Interface '88. Canadian Information Processing Society, Toronto, Ont., Canada, Canada, 192--197. http://dl.acm.org/citation.cfm?id=102313.102339
12. Krzysztof Gajos and Daniel S. Weld. 2004. SUPPLE: Automatically Generating User Interfaces. In Proceedings of the 9th International Conference on Intelligent User Interfaces (IUI '04). ACM, New York, NY, USA, 93--100. DOI: http://dx.doi.org/10.1145/964442.964461
13. Mark D. Gross and Ellen Yi-Luen Do. 1996. Ambiguous Intentions: A Paper-like Interface for Creative Design. In Proceedings of the 9th Annual ACM Symposium on User Interface Software and Technology (UIST '96). ACM, New York, NY, USA, 183--192. DOI: http://dx.doi.org/10.1145/237091.237119
14. Pierre Hansen and Nenad Mladenović. 2001. Variable neighborhood search: Principles and applications. European journal of operational research 130, 3 (2001), 449--467.
15. Marc Hassenzahl. 2008. The Interplay of Beauty, Goodness, and Usability in Interactive Products. Hum.-Comput. Interact. 19, 4 (Dec. 2008), 319--349. DOI: http://dx.doi.org/10.1207/s15327051hci1904_2
16. Eric Horvitz. 1999. Principles of Mixed-initiative User Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '99). ACM, New York, NY, USA, 159--166. DOI: http://dx.doi.org/10.1145/302979.303030
17. Gabe Johnson, Mark D. Gross, Jason Hong, and Ellen Yi-Luen Do. 2009. Computational Support for Sketching in Design: A Review. Found. Trends Hum.-Comput. Interact. 2, 1 (Jan. 2009), 1--93. DOI: http://dx.doi.org/10.1561/1100000013
18. Levent Burak Kara, Chris M. D'Eramo, and Kenji Shimada. 2006. Pen-based Styling Design of 3D Geometry Using Concept Sketches and Template Models. In Proceedings of the 2006 ACM Symposium on Solid and Physical Modeling (SPM '06). ACM, New York, NY, USA, 149--160. DOI: http://dx.doi.org/10.1145/1128888.1128909
19. Rubaiat Habib Kazi, Fanny Chevalier, Tovi Grossman, and George Fitzmaurice. 2014. Kitty: Sketching Dynamic and Interactive Illustrations. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (UIST '14). ACM, New York, NY, USA, 395--405. DOI: http://dx.doi.org/10.1145/2642918.2647375
20. Kenneth L. Kelly. 1965. Twenty-two colors of maximum contrast. Color Engineering 3, 26 (1965), 26--27.
21. David E. Kieras and Anthony J. Hornof. 2014. Towards Accurate and Practical Predictive Models of Active-vision-based Visual Search. In Proceedings of the 32Nd Annual ACM Conference on Human Factors in Computing Systems (CHI '14). ACM, New York, NY, USA, 3875--3884. DOI: http://dx.doi.org/10.1145/2556288.2557324
22. Ranjitha Kumar, Arvind Satyanarayan, Cesar Torres, Maxine Lim, Salman Ahmad, Scott R. Klemmer, and Jerry O. Talton. 2013. Webzeitgeist: Design Mining the Web. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 3083--3092. DOI: http://dx.doi.org/10.1145/2470654.2466420
23. James A. Landay and Brad A. Myers. 1995. Interactive Sketching for the Early Stages of User Interface Design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '95). ACM Press/Addison-Wesley Publishing Co., New York, NY, USA, 43--50. DOI: http://dx.doi.org/10.1145/223904.223910
24. Lissa Light and Peter Anderson. 1993. Designing better keyboards via simulated annealing. (1993).
25. James Lin, Mark W. Newman, Jason I. Hong, and James A. Landay. 2000. DENIM: Finding a Tighter Fit Between Tools and Practice for Web Site Design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '00). ACM, New York, NY, USA, 510--517. DOI: http://dx.doi.org/10.1145/332040.332486
26. Simon Lok and Steven Feiner. 2001. A survey of automated layout techniques for information presentations. Proceedings of SmartGraphics 2001 (2001).
27. Simon Lok, Steven Feiner, and Gary Ngai. 2004. Evaluation of Visual Balance for Automated Layout. In Proceedings of the 9th International Conference on Intelligent User Interfaces (IUI '04). ACM, New York, NY, USA, 101--108. DOI: http://dx.doi.org/10.1145/964442.964462
28. Scott I. MacKenzie. 1992. Fitts' law as a research and design tool in human-computer interaction. Human-Computer Interaction 7, 1 (1992), 91--139.
29. Barbara J. Meier, Anne Morgan Spalter, and David B. Karelitz. 2004. Interactive Color Palette Tools. IEEE Comput. Graph. Appl. 24, 3 (May 2004), 64--72. DOI: http://dx.doi.org/10.1109/MCG.2004.1297012
30. Mark W. Newman and James A. Landay. 2000. Sitemaps, Storyboards, and Specifications: A Sketch of Web Site Design Practice. In Proceedings of the 3rd Conference on Designing Interactive Systems: Processes, Practices, Methods, and Techniques (DIS '00). ACM, New York, NY, USA, 263--274. DOI: http://dx.doi.org/10.1145/347642.347758
31. Peter O'Donovan, Aseem Agarwala, and Aaron Hertzmann. 2014. Learning Layouts for Single-PageGraphic Designs. Visualization and Computer Graphics, IEEE Transactions on 20, 8 (2014), 1200--1213.
32. Peter O'Donovan, Aseem Agarwala, and Aaron Hertzmann. 2015. DesignScape: Design with Interactive Layout Suggestions. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 1221--1224. DOI: http://dx.doi.org/10.1145/2702123.2702149
33. Antti Oulasvirta, Anna Reichel, Wenbin Li, Yan Zhang, Myroslav Bachynskyi, Keith Vertanen, and Per Ola Kristensson. 2013. Improving Two-thumb Text Entry on Touchscreen Devices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, New York, NY, USA, 2765--2774. DOI:http://dx.doi.org/10.1145/2470654.2481383
34. Ruth Rosenholtz, Yuanzhen Li, and Lisa Nakano. 2007. Measuring visual clutter. Journal of vision 7, 2 (2007), 17.
35. Clayton Shepard, Ahmad Rahmati, Chad Tossell, Lin Zhong, and Phillip Kortum. 2011. LiveLab: Measuring Wireless Networks and Smartphone Users in the Field. SIGMETRICS Perform. Eval. Rev. 38, 3 (Jan. 2011), 15--20. DOI: http://dx.doi.org/10.1145/1925019.1925023
36. Surya P. Singh and Renduchintala RK Sharma. 2006. A review of different approaches to the facility layout problems. The International Journal of Advanced Manufacturing Technology 30, 5--6 (2006), 425--433.
37. Ivan E. Sutherland. 1963. Sketchpad: A Man-machine Graphical Communication System. In Proceedings of the May 21--23, 1963, Spring Joint Computer Conference (AFIPS '63 (Spring)). ACM, New York, NY, USA, 329--346. DOI: http://dx.doi.org/10.1145/1461551.1461591
38. Michael Terry, Elizabeth D. Mynatt, Kumiyo Nakakoji, and Yasuhiro Yamamoto. 2004. Variation in Element and Action: Supporting Simultaneous Development of Alternative Solutions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '04). ACM, New York, NY, USA, 711--718. DOI: http://dx.doi.org/10.1145/985692.985782
39. Marc Van Droogenbroeck and Sérard. 2011.ebastien Pié Object Descriptors Based on a List of Rectangles: Method and Algorithm. In Proceedings of the 10th International Conference on Mathematical Morphology and Its Applications to Image and Signal Processing (ISMM'11). Springer-Verlag, Berlin, Heidelberg, 155--165. http: //dl.acm.org/citation.cfm?id=2023043.2023061
40. L.G. Williams. 1966. A Study of Visual Search Using Eye Movement Recordings. Technical Report. DTIC Document.
41. Jacob O. Wobbrock, Andrew D. Wilson, and Yang Li. 2007. Gestures Without Libraries, Toolkits or Training: A $1 Recognizer for User Interface Prototypes. In Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology (UIST '07). ACM, New York, NY, USA, 159--168. DOI: http://dx.doi.org/10.1145/1294211.1294238
42. Yin Yin Wong. 1992. Rough and Ready Prototypes: Lessons from Graphic Design. In Posters and Short Talks of the 1992 SIGCHI Conference on Human Factors in Computing Systems (CHI '92). ACM, New York, NY, USA, 83--84. DOI: http://dx.doi.org/10.1145/1125021.1125094
43. Shengxiang Yang and Xin Yao. 2008. Population-based incremental learning with associative memory for dynamic environments. Evolutionary Computation, IEEE Transactions on 12, 5 (2008), 542--561.
44. Yeonsoo Yang and Scott R. Klemmer. 2009. Aesthetics Matter: Leveraging Design Heuristics to Synthesize Visually Satisfying Handheld Interfaces. In CHI '09 Extended Abstracts on Human Factors in Computing Systems (CHI EA '09). ACM, New York, NY, USA, 4183--4188. DOI: http://dx.doi.org/10.1145/1520340.1520637
45. Shumin Zhai, Michael Hunter, and Barton A. Smith. 2002. Performance Optimization of Virtual Keyboards. Human-Computer Interaction 17, 2--3 (2002), 229--269.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)