Understanding and Exploiting Object Interaction Landscapes

Abstract

Interactions play a key role in understanding objects and scenes for both virtual and real-world agents. We introduce a new general representation for proximal interactions among physical objects that is agnostic to the type of objects or interaction involved. The representation is based on tracking particles on one of the participating objects and then observing them with sensors appropriately placed in the interaction volume or on the interaction surfaces. We show how to factorize these interaction descriptors and project them into a particular participating object so as to obtain a new functional descriptor for that object, its interaction landscape, capturing its observed use in a spatiotemporal framework. Interaction landscapes are independent of the particular interaction and capture subtle dynamic effects in how objects move and behave when in functional use. Our method relates objects based on their function, establishes correspondences between shapes based on functional key points and regions, and retrieves peer and partner objects with respect to an interaction.

References

1. Adobe. 2015. Mixamo. Retrieved from https://www.mixamo.com/.
2. R. Ali Al-Asqhar, T. Komura, and M. Geol Choi. 2013. Relationship descriptors for interactive motion adaptation. In Proceedings of ACM SIGGRAPH/Eurographics Symposium on Computer Animation (SCA’13). ACM, 45--53.
3. N. Amenta, M. Bern, and M. Kamvysselis. 1998. A new Voronoi-based surface reconstruction algorithm. In Proceedings of SIGGRAPH. ACM, New York, 415--421.
4. D. Anguelov, P. Srinivasan, D. Koller, S. Thrun, J. Rodgers, and J. Davis. 2005. SCAPE: Shape completion and animation of people. In Proceedings of SIGGRAPH.
5. F. G. Ashby. 1992. Multidimensional Models of Perception and Cognition. L. Erlbaum. https://books.google.com/books?id=eeQh4ESeOfsC
6. E. Bar-Aviv and E. Rivlin. 2006. Functional 3D object classification using simulation of embodied agent. In BMVC. British Machine Vision Association, 307--316.
7. I. Baran and J. Popović. 2007. Automatic rigging and animation of 3D characters. ACM Trans. Graph. 26, 3 (July 2007), 72:1--72:8.
8. S. Biasotti, A. Cerri, A. Bronstein, and M. Bronstein. 2015. Recent trends, applications, and perspectives in 3D shape similarity assessment. Comp. Graph. Forum (2015).
9. A. W. Black and P. Taylor. 1997. Automatically clustering similar units for unit selection in speech synthesis. In Eurospeech97. 601--604.
10. M. Caine. 1994. The design of shape interactions using motion constraints. In Proceedings of the 1994 IEEE International Conference on Robotics and Automation, Vol. 1. 366--371.
11. A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, J. Xiao, L. Yi, and F. Yu. 2015. ShapeNet: An information-rich 3D model repository. ArXiv e-prints (Dec. 2015).
12. Y.-W. Chao, Z. Wang, Y. He, J. Wang, and J. Deng. 2015. HICO: A benchmark for recognizing human-object interactions in images. ICCV (2015).
13. D.-Y. Chen, X.-P. Tian, Y.-T. Shen, and M. Ouhyoung. 2003. On visual similarity based 3D model retrieval. Comp. Graph. Forum 22, 3 (2003), 223--232.
14. J. Chen, X. Ge, L.-Y. Wei, B. Wang, Y. Wang, H. Wang, Y. Fei, K.-L. Qian, J.-H. Yong, and W. Wang. 2013. Bilateral blue noise sampling. ACM Trans. Graph. 32, 6, Article 216 (Nov. 2013), 11 pages.
15. S. Durrleman. 2010. Statistical Models of Currents for Measuring the Variability of Anatomical Curves, Surfaces and Their Evolution. Ph.D. dissertation. Université Nice - Sophia Antipolis, France.
16. M. Fisher, M. Savva, Y. Li, P. Hanrahan, and M. Niessner. 2015. Activity-centric scene synthesis for functional 3D scene modeling. ACM Trans. Graph. 34, 6, Article 179 (Oct. 2015), 13 pages.
17. R. Gal and D. Cohen-Or. 2006. Salient geometric features for partial shape matching and similarity. ACM Trans. Graph. 25, 1 (2006), 130--150.
18. H. Grabner, J. Gall, and L. Van Gool. 2011. What makes a chair a chair?. In CVPR. 1529--1536.
19. A. Gupta, A. Kembhavi, and L. S. Davis. 2009. Observing human-object interactions: Using spatial and functional compatibility for recognition. IEEE Trans. Pattern Anal. Mach. Intell. 31, 10 (Oct. 2009), 1775--1789.
20. E. S. L. Ho, T. Komura, and C.-L. Tai. 2010. Spatial relationship preserving character motion adaptation. ACM Trans. Graph. 29, 4, Article 33 (2010), 8 pages.
21. R. Hu, O. van Kaick, B. Wu, H. Huang, A. Shamir, and H. Zhang. 2016. Learning how objects function via co-analysis of interactions. ACM Trans. Graph. 35, 4, Article 47 (July 2016), 13 pages.
22. R. Hu, C. Zhu, O. van Kaick, L. Liu, A. Shamir, and H. Zhang. 2015. Interaction context (ICON): Towards a geometric functionality descriptor. ACM Trans. Graph. 34, 4, Article 83 (2015), 12 pages.
23. H. Jiang and D. R. Martin. 2008. Finding actions using shape flows. In Computer Vision ECCV 2008, David Forsyth, Philip Torr, and Andrew Zisserman (Eds.). Lecture Notes in Computer Science, Vol. 5303. Springer, Berlin, 278--292.
24. Germany Karlsruhe Institute of Technology, Karlsruhe. 2015. KIT Whole Human Motion Database. Retrieved from https://motion-database.humanoids.kit.edu/.
25. V. G. Kim, S. Chaudhuri, L. Guibas, and T. Funkhouser. 2014. Shape2Pose: Human-centric shape analysis. ACM Trans. Graph. 33, 4, Article 120 (2014), 12 pages.
26. H. Laga, M. Mortara, and M. Spagnuolo. 2013. Geometry and context for semantic correspondences and functionality recognition in man-made 3D shapes. ACM Trans. Graph. 32, 5, Article 150 (2013), 16 pages.
27. C. H. Lee, A. Varshney, and D. W. Jacobs. 2005. Mesh saliency. ACM Trans. Graph. 24, 3 (July 2005), 659--666.
28. M. Leordeanu and M. Hebert. 2005. A spectral technique for correspondence problems using pairwise constraints. In Proceedings of the 10th IEEE International Conference on Computer Vision (ICCV’05). Vol. 2. 1482--1489.
29. P. Li, B. Wang, F. Sun, X. Guo, C. Zhang, and W. Wang. 2015. Q-MAT: Computing medial axis transform by quadratic error minimization. ACM Trans. Graph. 35, 1, Article 8 (Dec. 2015), 16 pages.
30. Y. Li, J. L. Fu, and N. S. Pollard. 2007. Data-driven grasp synthesis using shape matching and task-based pruning. IEEE Trans. Vis. Comput. Graph. 13, 4 (July 2007), 732--747.
31. Z. Liu, C. Xie, S. Bu, X. Wang, J. Han, H. Lin, and H. Zhang. 2015. Indirect shape analysis for 3D shape retrieval. Comput. Graphics 46 (2015), 110--116. Shape Modeling International 2014.
32. N. Mitra, M. Wand, H. (Richard) Zhang, D. Cohen-Or, V. Kim, and Q.-X. Huang. 2013b. Structure-aware shape processing. In SIGGRAPH Asia 2013 Courses (SA’13). ACM, New York, Article 1, 20 pages.
33. N. J. Mitra, M. Pauly, M. Wand, and D. Ceylan. 2013a. Symmetry in 3D geometry: Extraction and applications. Comput. Graph. Forum 32, 6 (2013), 1--23.
34. J. J. Monaghan. 1992. Smoothed particle hydrodynamics. Ann. Rev. Astron. Astrophys. 30 (1992), 543--574.
35. O. Oreifej and Z. Liu. 2013. HON4D: Histogram of oriented 4D normals for activity recognition from depth sequences. In 2013 IEEE Conference on Computer Vision and Pattern Recognition (CVPR’13). 716--723.
36. R. Osada, T. Funkhouser, B. Chazelle, and D. Dobkin. 2002. Shape distributions. ACM Trans. Graph. 21, 4 (Oct. 2002), 807--832.
37. M. Ovsjanikov, M. Ben-Chen, J. Solomon, A. Butscher, and L. Guibas. 2012. Functional maps: A flexible representation of maps between shapes. ACM Trans. Graph. 31, 4 (2012).
38. M. Pechuk, O. Soldea, and E. Rivlin. 2008. Learning function-based object classification from 3D imagery. Comput. Vis. Image Underst. 110, 2 (May 2008), 173--191.
39. E. Rivlin, S. J. Dickinson, and A, Rosenfeld. 1994. Recognition by functional parts {function-based object recognition}. In Proceedings of the 1994 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’94). 267--274.
40. Columbia University USA Robotics Lab, Computer Science Laboratory. 2012. GraspIt! (2012). http://www.cs.columbia.edu/ cmatei/graspit/.
41. Y. Rubner, C. Tomasi, and L. J. Guibas. 1998. A metric for distributions with applications to image databases. In Proceedings of the 6th International Conference on Computer Vision, 1998. 59--66.
42. M. Savva, A. X. Chang, P. Hanrahan, M. Fisher, and M. Nießner. 2014. SceneGrok: Inferring action maps in 3D environments. ACM Trans. Graph. 33, 6, Article 212 (Nov. 2014), 10 pages.
43. M. Savva, A. X. Chang, P. Hanrahan, M. Fisher, and M. Nießner. 2016. PiGraphs: Learning interaction snapshots from observations. ACM Trans. Graph. 35, 4, Article 139 (July 2016), 12 pages.
44. P. Shilane and T. Funkhouser. 2007. Distinctive regions of 3D surfaces. ACM Trans. Graph. 26, 2, Article 7 (June 2007).
45. O. Sidi, O. van Kaick, Y. Kleiman, H. Zhang, and D. Cohen-Or. 2011. Unsupervised co-segmentation of a set of shapes via descriptor-space spectral clustering. ACM Trans. Graph. 30, 6, Article 126 (2011), 10 pages.
46. H. O. Song, M. Fritz, C. Gu, and T. Darrell. 2011. Visual grasp affordances from appearance-based cues. In Proceedings of the 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops). 998--1005.
47. M. Sutton, L. Stark, and K. Bowyer. 1994. GRUFF-3: Generalizing the domain of a function-based recognition system. Pattern Recog. 27, 12 (1994), 1743--1766.
48. A. Tevs, Q. Huang, M. Wand, H.-P. Seidel, and L. Guibas. 2014. Relating shapes via geometric symmetries and regularities. ACM Trans. Graph. 33, 4, Article 119 (July 2014), 12 pages.
49. A. Tversky. 1977. Features of similarity. Psych. Rev. 84 (1977), 327--352.
50. D. Tzionas and J. Gall. 2015. 3D object reconstruction from hand-object interactions. In Proceedings of the International Conference on Computer Vision (ICCV). http://files.is.tue.mpg.de/dtzionas/In-Hand-Scanning
51. Z. Wu, S. Song, A. Khosla, X. Tang, and J. Xiao. 2014. 3D shapenets for 2.5d object recognition and next-best-view prediction. CoRR abs/1406.5670 (2014). http://arxiv.org/abs/1406.5670
52. L. Xu, H. Quynh Dinh, P. Mordohai, and T. Ramsay. 2011. Detecting patterns in vector fields. 49th AIAA Aerospace Sciences Meeting Including the New Horizons Forum and Aerospace Exposition 6 (2011).
53. X. Zhao, H. Wang, and T. Komura. 2014. Indexing 3D scenes using the interaction bisector surface. ACM Trans. Graph. 33, 3, Article 22 (2014), 14 pages.
54. Y. Zheng, D. Cohen-Or, and N. J. Mitra. 2013. Smart variations: Functional substructures for part compatibility. Comp. Graph. Forum 32, 2 pt2 (2013), 195--204.
55. Y. Zhu, A. Fathi, and L. Fei-Fei. 2014. Reasoning about object affordances in a knowledge base representation. In Computer Vision ECCV 2014, D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars (Eds.). Lecture Notes in Computer Science, Vol. 8690. Springer International Publishing, 408--424.

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Computer vision representations

        • Shape representations

  • Computer graphics

    • Shape modeling

      • Shape analysis

  • Modeling and simulation

    • Model development and analysis

      • Modeling methodologies