Agile 3D Sketching with Air Scaffolding

ABSTRACT

Hand motion and pen drawing can be intuitive and expressive inputs for professional digital 3D authoring. However, their inherent limitations have hampered wider adoption. 3D sketching using hand motion is rapid but rough, and 3D sketching using pen drawing is delicate but tedious. Our new 3D sketching workflow combines these two in a complementary manner. The user makes quick hand motions in the air to generate approximate 3D shapes, and uses them as scaffolds on which to add details via pen-based 3D sketching on a tablet device. Our air scaffolding technique and corresponding algorithm extract only the intended shapes from unconstrained hand motions. Then, the user sketches 3D ideas by defining sketching planes on these scaffolds while appending new scaffolds, as needed. A user study shows that our progressive and iterative workflow enables more agile 3D sketching compared to ones using either hand motion or pen drawing alone.

References

1. Rahul Arora, Rubaiat Habib Kazi, Fraser Anderson, Tovi Grossman, Karan Singh, and George Fitzmaurice. 2017. Experimental evaluation of sketching on surfaces in VR. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17), 5643--5654.
2. Seok-Hyung Bae, Ravin Balakrishnan, and Karan Singh. 2008. ILoveSketch: as-natural-as-possible sketching system for creating 3D curve models. In Proceedings of the 21st annual ACM symposium on User interface software and technology (UIST '08), 151--160.
3. Seok-Hyung Bae, Ravin Balakrishnan, and Karan Singh. 2009. EverybodyLovesSketch: 3D sketching for a broader audience. In Proceedings of the 22nd annual ACM symposium on User interface software and technology (UIST '09), 59--68.
4. Bruno R. De Araùjo, Géry Casiez, and Joaquim A. Jorge. 2012. Mockup builder: direct 3D modeling on and above the surface in a continuous interaction space. In Proceedings of Graphics Interface 2012 (GI '12), 173--180.
5. Chris De Paoli and Karan Singh. 2015. SecondSkin: sketch-based construction of layered 3D models. ACM Trans. Graph. 34, 4, Article 126, 10 pages.
6. Chao Ding and Ligang Liu. 2016. A survey of sketch based modeling systems. Frontiers of Computer Science 10, 2: 985--999.
7. Julie Dorsey, Songhua Xu, Gabe Smedresman, Holly Rushmeier, and Leonard McMillan. 2007. The mental canvas: a tool for conceptual architectural design and analysis. In 15th Pacific Conference on Computer Graphics and Applications (PG '07), 201--210.
8. Mark Fuge, Mehmet Ersin Yumer, Gunay Orbay, and Levent Burak Kara. 2012. Conceptual design and modification of freeform surfaces using dual shape representations in augmented reality environments. Computer-Aided Design 44, 10, 1020--1032.
9. Google. 2016. Tilt Brush. https://www.tiltbrush.com
10. Christian Holz and Andrew Wilson. 2011. Data miming: inferring spatial object descriptions from human gesture. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11), 811--820.
11. Takeo Igarashi, Satoshi Matsuoka, and Hidehiko Tanaka. 1999. Teddy: a sketching interface for 3D freeform design. In Proceedings of the 26th annual conference on Computer graphics and interactive techniques (SIGGRAPH '99), 409--416.
12. Bret Jackson and Daniel F. Keefe. 2016. Lift-off: using reference imagery and freehand sketching to create 3D models in VR. IEEE transactions on visualization and computer graphics 22, 4, 1442--1451.
13. Kiia Kallio. 2005. 3D6B editor: projective 3D sketching with line-based rendering. In Proceedings of the 2nd Eurographics Workshop on Sketch-Based Interfaces and Modeling, 73--79.
14. Olga A. Karpenko, John F. Hughes, and Ramesh Raskar. 2004. Epipolar methods for multi-view sketching. In Proceedings of the First Eurographics Conference on Sketch-Based Interfaces and Modeling (SBIM '04), 167--173.
15. Daniel F. Keefe, Robert C. Zeleznik, and David H. Laidlaw. 2007. Drawing on air: input techniques for controlled 3D line illustration. IEEE Transactions on Visualization and Computer Graphics 13, 5, 10671081.
16. Yongkwan Kim and Seok-Hyung Bae. 2016. SketchingWithHands: 3D sketching handheld products with first-person hand posture. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16), 797--808.
17. Kevin T. McDonnell, Hong Qin, and Robert A. Wlodarczyk. 2001. Virtual clay: a real-time sculpting system with haptic toolkits. In Proceedings of the 2001 symposium on Interactive 3D graphics (I3D '01), 179190.
18. Andrew Nealen, Takeo Igarashi, Olga Sorkine, and Marc Alexa. 2007. FiberMesh: designing freeform surfaces with 3D curves. ACM Trans. Graph. 26, 3, Article 41, 9 pages.
19. Oculus. 2016. Quill. https://www.oculus.com/storystudio/quill
20. Luke Olsen, Faramarz F. Samavati, Mario Costa Sousa, and Joaquim A. Jorge. 2009. Sketch-based modeling: a survey. Computers & Graphics 33, 1, 85--103.
21. Amit Pitaru. 2003. Rhonda. http://rhondaforever.com
22. Steven Schkolne, Michael Pruett, and Peter Schröder. 2001. Surface drawing: creating organic 3D shapes with the hand and tangible tools. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '01), 261--268.
23. Johannes Schmid, Martin Sebastian Senn, Markus Gross, and Robert W. Sumner. 2011. OverCoat: an implicit canvas for 3D painting. ACM Trans. Graph. 30, 4, Article 28, 10 pages.
24. Ryan Schmidt, Azam Khan, Gord Kurtenbach, and Karan Singh. 2009. On expert performance in 3D curve-drawing tasks. In Proceedings of the 6th Eurographics Symposium on Sketch-Based Interfaces and Modeling (SBIM '09), 133--140.
25. Ryan Schmidt, Azam Khan, Karan Singh, and Gord Kurtenbach. 2009. Analytic drawing of 3D scaffolds. ACM Trans. Graph. 28, 5, Article 149, 10 pages.
26. Jonathan Taylor, Lucas Bordeaux, Thomas Cashman, Bob Corish, Cem Keskin, Toby Sharp, Eduardo Soto, David Sweeney, Julien Valentin, Benjamin Luff, Arran Topalian, Erroll Wood, Sameh Khamis, Pushmeet Kohli, Shahram Izadi, Richard Banks, Andrew Fitzgibbon, and Jamie Shotton. 2016. Efficient and precise interactive hand tracking through joint, continuous optimization of pose and correspondences. ACM Trans. Graph. 35, 4, Article 143, 12 pages.
27. Edit Varga, Imre Horváth, Zoltán Rusák, Bram de Smit, and Han Broek. 2004. Survey and investigation of hand motion processing technologies for compliance with shape conceptualization. In ASME 2004 International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, 575--587.
28. Vinayak, Sundar Murugappan, HaiRong Liu, and Karthik Ramani. 2013. Shape-It-Up: hand gesture based creative expression of 3D shapes using intelligent generalized cylinders. Computer-Aided Design 45, 2, 277--287.
29. Gerold Wesche and Hans-Peter Seidel. 2001. FreeDrawer: a free-form sketching system on the responsive workbench. In Proceedings of the ACM symposium on Virtual reality software and technology (VRST '01), 167--174.
30. Ying Wu and Thomas S. Huang. 2001. Hand modeling, analysis and recognition. IEEE Signal Processing Magazine 18, 3, 51--60.
31. Min Xin, Ehud Sharlin, and Mario Costa Sousa. 2008. Napkin sketch: handheld mixed reality 3D sketching. In Proceedings of the 2008 ACM symposium on Virtual reality software and technology (VRST '08), 223--226.

Index Terms

• Applied computing

  • Document management and text processing

    • Document capture

      • Graphics recognition and interpretation

• Computing methodologies

  • Computer graphics

• Human-centered computing

  • Human computer interaction (HCI)