research-article

Public Access

Learning to group discrete graphical patterns

Authors:

Evangelos Kalogerakis,

Marie-Paule Cani,

Hao ZhangAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 36, Issue 6

Article No.: 225, Pages 1 - 11

https://doi.org/10.1145/3130800.3130841

Published: 20 November 2017 Publication History

Abstract

We introduce a deep learning approach for grouping discrete patterns common in graphical designs. Our approach is based on a convolutional neural network architecture that learns a grouping measure defined over a pair of pattern elements. Motivated by perceptual grouping principles, the key feature of our network is the encoding of element shape, context, symmetries, and structural arrangements. These element properties are all jointly considered and appropriately weighted in our grouping measure. To better align our measure with human perceptions for grouping, we train our network on a large, human-annotated dataset of pattern groupings consisting of patterns at varying granularity levels, with rich element relations and varieties, and tempered with noise and other data imperfections. Experimental results demonstrate that our deep-learned measure leads to robust grouping results.

Supplementary Material

ZIP File (a225-lun.zip)

Supplemental material.

Download
22.00 MB

References

[1]

Sawsan AlHalawani, Yongliang Yang, Han Liu, and Niloy J. Mitra. 2013. Interactive Facades: Analysis and Synthesis of Semi-Regular Facades. Computer Graphics Forum (Eurographics) (2013), to appear.

[2]

Pablo Arbelaez, Michael Maire, Charless Fowlkes, and Jitendra Malik. 2011. Contour detection and hierarchical image segmentation. IEEE transactions on pattern analysis and machine intelligence 33, 5 (2011), 898--916.

Digital Library

[3]

Fan Bao, Michael Schwarz, and Peter Wonka. 2013. Procedural facade variations from a single layout. ACM Trans. on Graph 32, 1 (2013).

Digital Library

[4]

Connelly Barnes, Eli Shechtman, Dan B. Goldman, and Adam Finkelstein. 2010. The Generalized Patchmatch Correspondence Algorithm. In Proc. of ECCV.

Digital Library

[5]

M. Bokeloh, M. Wand, and H.-P. Seidel. 2010. A Connection between Partial Symmetry and Inverse Procedural Modeling. ACM Trans. on Graph 29, 4 (2010), 104:1--104:10.

Digital Library

[6]

Yilan Chen, Hongbo Fu, and Kin Chung Au. 2016. A Multi-level Sketch-based Interface for Decorative Pattern Exploration. In SIGGRAPH ASIA 2016 Technical Briefs (SA '16). Article 26, 4 pages.

Digital Library

[7]

John H. Conway, Heidi Burgiel, and Chaim Goodman-Strauss. 2008. The Symmetries of Things. A K Peters/CRC Press.

[8]

Agné Desolneux, Lionel Moisan, and Jean-Michel Morel. 2002. Gestalt theory and computer vision. Springer.

[9]

Jacob Feldman. 2003. Perceptual grouping by selection of a logically minimal model. Proc. of ICCV 55 (2003), 5--25.

Digital Library

[10]

Brendan J. Frey and Delbert Dueck. 2007. Clustering by passing messages between data points. Science 315 (2007), 2007.

[11]

John H. Graham, Shmuel Raz, Hagit Hel-Or, and Eviatar Nevo. 2010. Fluctuating Asymmetry: Methods, Theory, and Applications. Symmetry 2, 2 (2010), 466--540.

[12]

Klaus Greff, Antti Rasmus, Mathias Berglund, Tele Hao, Harri Valpola, and Juergen Schmidhuber. 2016. Tagger: Deep Unsupervised Perceptual Grouping. In Advances in Neural Information Processing Systems. 4484--4492.

Digital Library

[13]

Paul Guerrero, Gilbert Bernstein, Wilmot Li, and Niloy J. Mitra. 2016. PATEX: Exploring Pattern Variations. ACM Trans. on Graph 35, 4, Article 48 (July 2016), 48:1--48:13 pages.

Digital Library

[14]

Raia Hadsell, Sumit Chopra, and Yann LeCun. 2006. Dimensionality Reduction by Learning an Invariant Mapping. In Proc. CVPR.

Digital Library

[15]

Xufeng Han, T. Leung, Y. Jia, R. Sukthankar, and A. C. Berg. 2015. MatchNet: Unifying feature and metric learning for patch-based matching. In Proc. of CVPR.

[16]

Trevor Hastie, Robert Tibshirani, and Jerome Friedman. 2001. The Elements of Statistical Learning. Springer New York Inc., New York, NY, USA.

[17]

Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017. Image-to-Image Translation with Conditional Adversarial Networks. CVPR (2017).

[18]

Phillip Isola, Daniel Zoran, Dilip Krishnan, and Edward H Adelson. 2016. Learning visual groups from co-occurrences in space and time. International Conference on Learning Representations, Workshop paper (2016).

[19]

Gaetano Kanizsa. 1980. Grammatica del Vedere. Il Mulino.

[20]

Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. CoRR abs/1412.6980 (2014).

[21]

W. Köhler. 1929. Gestalt Psychology. Liveright, New York.

[22]

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Proc. NIPS.

Digital Library

[23]

Michael Kubovy and Martin van den Berg. 2008. The whole is equal to the sum of its parts: A probabilistic model of grouping by proximity and similarity in regular patterns. Psychological Review 115, 1 (2008), 131--154.

[24]

Yanxi Liu, Hagit Hel-Or, Craig S. Kaplan, and Luc J. Van Gool. 2010. Computational Symmetry in Computer Vision and Computer Graphics. Foundations and Trends in Computer Graphics and Vision, Vol. 5. 1--195.

[25]

Michal Lukac, Daniel Sykora, Kalyan Sunkavalli, Eli Shechtman, Ondrej Jamriska, Nathan Carr, and Tomas Pajdla. 2017. Nautilus: Recovering Regional Symmetry Transformations for Image Editing. ACM Trans. on Graph to appear (2017).

Digital Library

[26]

Michael Maire, Takuya Narihira, and Stella X Yu. 2016. Affinity CNN: Learning pixel-centric pairwise relations for figure/ground embedding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 174--182.

[27]

Kevis-Kokitsi Maninis, Jordi Pont-Tuset, Pablo Arbeláez, and Luc Van Gool. 2016. Convolutional oriented boundaries. In European Conference on Computer Vision. 580--596.

[28]

Niloy J. Mitra, Mark Pauly, Michael Wand, and Duygu Ceylan. 2012. Symmetry in 3D Geometry: Extraction and Applications. In Proc. of Eurographics STAR Report.

[29]

Liangliang Nan, Andrei Sharf, Ke Xie, Tien-Tsin Wong, Oliver Deussen, Daniel Cohen-Or, and Baoquan Chen. 2011. Conjoining Gestalt Rules for Abstraction of Architectural Drawings. ACM Trans. on Graph 30, 6 (2011). 185:1--185:10.

Digital Library

[30]

S. Palmer. 1992. Common region: a new principle of perceptual grouping. Cognitive Psychology 24 (1992), 436--447.

[31]

Stephen E. Palmer. 1977. Hierarchical structure in perceptual representation. Cognitive Psychology 9, 4 (1977), 441--474.

[32]

Joshua Podolak, Philip Shilane, Aleksey Golovinskiy, Szymon Rusinkiewicz, and Thomas Funkhouser. 2006. A Planar-reflective Symmetry Transform for 3D Shapes. ACM Trans. on Graph 25, 3 (2006).

Digital Library

[33]

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV) 115, 3 (2015), 211--252.

Digital Library

[34]

Jianbo Shi and Jitendra Malik. 2000. Normalized Cuts and Image Segmentation. IEEE Trans. Pat. Ana. & Mach. Int. (2000).

Digital Library

[35]

Patricio Simari, Evangelos Kalogerakis, and Karan Singh. 2006. Folding meshes: hierarchical mesh segmentation based on planar symmetry. Symp. on Geom. Proc. (2006), 111--119.

Digital Library

[36]

Edgar Simo-Serra, Eduard Trulls, Luis Ferraz, Iasonas Kokkinos, Pascal Fua, and Francesc Moreno-Noguer. 2015. Discriminative Learning of Deep Convolutional Feature Point Descriptors. In Proc. of ICCV.

Digital Library

[37]

O. Stava, B. Benes, R. Mech, D. Aliga, and P. Kristof. 2010. Inverse Procedural Modeling by Automatic Generation of L-systems. Computer Graphics Forum 29, 2 (2010), 665--674.

Digital Library

[38]

Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik G. Learned-Miller. 2015. Multi-view Convolutional Neural Networks for 3D Shape Recognition. In Proc. ICCV.

Digital Library

[39]

Oliver van Kaick, Kai Xu, Hao Zhang, Yanzhen Wang, Shuyang Sun, Ariel Shamir, and Daniel Cohen-Or. 2013. Co-Hierarchical Analysis of Shape Structures. ACM Trans. on Graph 32, 4 (2013), Article 69.

Digital Library

[40]

Kaijun Wang. 2010. Fast Affinity Propagation Clustering under Given Number of Clusters. (2010). https://www.mathworks.com/matlabcentral/fileexchange/25722-fast-affinity-propagation-clustering-under-given-number-of-clusters

[41]

Yanzhen Wang, Kai Xu, Jun Li, Hao Zhang, Ariel Shamir, Ligang Liu, Zhiquan Cheng, and Yueshan Xiong. 2011. Symmetry Hierarchy of Man-Made Objects. Computer Graphics Forum (Eurographics) 30, 2 (2011), 287--296.

[42]

M. Wertheimer. 1938. Laws of organization in perceptual forms. 71--88.

[43]

H. Weyl. 1952. Symmetry. Princeton University Press.

[44]

Fuzhang Wu, Dong-Ming Yan, Weiming Dong, Xiaopeng Zhang, and Peter Wonka. 2014. Inverse procedural modeling of facade layouts. ACM Trans. on Graph 33, 4 (2014), 121:1--121:10.

Digital Library

[45]

Junyuan Xie, Ross Girshick, and Ali Farhadi. 2016. Unsupervised Deep Embedding for Clustering Analysis. In Proc. of ICML.

Digital Library

[46]

Pengfei Xu, Hongbo Fu, Chiew-Lan Tai, and Takeo Igarashi. 2015. GACA: Group-Aware Command-based Arrangement of Graphic Elements. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, CHI 2015, Seoul, Republic of Korea, April 18-23, 2015. 2787--2795.

Digital Library

[47]

Jianwei Yang, Devi Parikh, and Dhruv Batra. 2016. Joint Unsupervised Learning of Deep Representations and Image Clusters. In Proc. CVPR.

[48]

Yi-Ting Yeh, Katherine Breeden, Lingfeng Yang, Matthew Fisher, and Pat Hanrahan. 2013. Synthesis of Tiled Patterns Using Factor Graphs. ACM Trans. on Graph 32, 1, Article 3 (Feb. 2013), 3:1--3:13 pages.

Digital Library

[49]

Kwang Moo Yi, Eduard Trulls, Vincent Lepetit, and Pascal Fua. 2016. LIFT: Learned Invariant Feature Transform. In IEEE ECCV.

[50]

S. Zagoruyko and N. Komodakis. 2015. Learning to compare image patches via convolutional neural networks. In Proc. of CVPR. 4353--4361.

[51]

Hao Zhang, Kai Xu, Wei Jiang, Jinjie Lin, Daniel Cohen-Or, and Baoquan Chen. 2013. Layered Analysis of Irregular Facades via Symmetry Maximization. ACM Trans. on Graph 32, 4 (2013), Article 121.

Digital Library

[52]

Arthur Zimek, Erich Schubert, and Hans-Peter Kriegel. 2012. A Survey on Unsupervised Outlier Detection in High-dimensional Numerical Data. Stat. Anal. Data Min. 5, 5 (2012).

Digital Library

[53]

C. Lawrence Zitnick and Piotr Dollár. 2014. Edge Boxes: Locating Object Proposals from Edges. In Proc. of ECCV.

Cited By

Wang ZPeng YFang LGao L(2025)Computational optical imaging: on the convergence of physical and digital layersOptica10.1364/OPTICA.54494312:1(113)Online publication date: 17-Jan-2025
https://doi.org/10.1364/OPTICA.544943
Xia FLi XDu ZZhao L(2024)Intelligent pattern reconstruction based on primitive spatial relationship adaptive transformationFifteenth International Conference on Graphics and Image Processing (ICGIP 2023)10.1117/12.3021534(92)Online publication date: 25-Mar-2024
https://doi.org/10.1117/12.3021534
Fei BYang WChen WMa LMagalhães Jdel Bimbo ASatoh SSebe NAlameda-Pineda XJin QOria VToni L(2022)VQ-DcTr: Vector-Quantized Autoencoder With Dual-channel Transformer Points Splitting for 3D Point Cloud CompletionProceedings of the 30th ACM International Conference on Multimedia10.1145/3503161.3548181(4769-4778)Online publication date: 10-Oct-2022
https://dl.acm.org/doi/10.1145/3503161.3548181
Show More Cited By

Index Terms

Learning to group discrete graphical patterns
1. Computing methodologies
  1. Computer graphics
    1. Shape modeling
      1. Shape analysis

Recommendations

Mining Mid-level Visual Patterns with Deep CNN Activations

The purpose of mid-level visual element discovery is to find clusters of image patches that are representative of, and which discriminate between, the contents of the relevant images. Here we propose a pattern-mining approach to the problem of ...
Discrete-to-deep reinforcement learning methods
Abstract
Neural networks are effective function approximators, but hard to train in the reinforcement learning (RL) context mainly because samples are correlated. In complex problems, a neural RL approach is often able to learn a better solution than ...
Combining local belief from low-level primitives for perceptual grouping

Segmentation is usually unable to cope with artifacts due to slight change in lighting conditions or object occlusion for instance. That is why perceptual grouping is often used to overcome segmentation's lacks. This refers to the ability of human ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 36, Issue 6

December 2017

973 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/3130800

Editor:
Kavita Bala

Issue’s Table of Contents

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 November 2017

Published in TOG Volume 36, Issue 6

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation
the Science and Technology Plan Project of Hunan Province
the Program of Key Disciplines in Hunan Province
Canadian Network for Research and Innovation in Machining Technology, Natural Sciences and Engineering Research Council of Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

14
Total Citations
View Citations
725
Total Downloads

Downloads (Last 12 months)119
Downloads (Last 6 weeks)16

Reflects downloads up to 02 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wang ZPeng YFang LGao L(2025)Computational optical imaging: on the convergence of physical and digital layersOptica10.1364/OPTICA.54494312:1(113)Online publication date: 17-Jan-2025
https://doi.org/10.1364/OPTICA.544943
Xia FLi XDu ZZhao L(2024)Intelligent pattern reconstruction based on primitive spatial relationship adaptive transformationFifteenth International Conference on Graphics and Image Processing (ICGIP 2023)10.1117/12.3021534(92)Online publication date: 25-Mar-2024
https://doi.org/10.1117/12.3021534
Fei BYang WChen WMa LMagalhães Jdel Bimbo ASatoh SSebe NAlameda-Pineda XJin QOria VToni L(2022)VQ-DcTr: Vector-Quantized Autoencoder With Dual-channel Transformer Points Splitting for 3D Point Cloud CompletionProceedings of the 30th ACM International Conference on Multimedia10.1145/3503161.3548181(4769-4778)Online publication date: 10-Oct-2022
https://dl.acm.org/doi/10.1145/3503161.3548181
Deng HZheng YChen JYu SXiao KMao X(2022)Learning 3D mineral prospectivity from 3D geological models using convolutional neural networks: Application to a structure-controlled hydrothermal gold depositComputers & Geosciences10.1016/j.cageo.2022.105074(105074)Online publication date: Feb-2022
https://doi.org/10.1016/j.cageo.2022.105074
Hu PMiao YHou M(2021)Reconstruction of Complex Roof Semantic Structures from 3D Point Clouds Using Local Convexity and ConsistencyRemote Sensing10.3390/rs1310194613:10(1946)Online publication date: 17-May-2021
https://doi.org/10.3390/rs13101946
Li LYuan CShen HZhuang YSmith JYang YCesar PMetze FPrabhakaran B(2021)VQMG: Hierarchical Vector Quantised and Multi-hops Graph Reasoning for Explicit Representation LearningProceedings of the 29th ACM International Conference on Multimedia10.1145/3474085.3475224(5029-5037)Online publication date: 17-Oct-2021
https://dl.acm.org/doi/10.1145/3474085.3475224
Gao YZhang ZLin HZhao XDu SZou C(2021)Hypergraph Learning: Methods and PracticesIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2020.3039374(1-1)Online publication date: 2021
https://doi.org/10.1109/TPAMI.2020.3039374
de Melo CTorralba AGuibas LDiCarlo JChellappa RHodgins J(2021)Next-generation deep learning based on simulators and synthetic dataTrends in Cognitive Sciences10.1016/j.tics.2021.11.008Online publication date: Dec-2021
https://doi.org/10.1016/j.tics.2021.11.008
Yang ZDai QZhang J(2021)Visual perception driven collage synthesisComputational Visual Media10.1007/s41095-021-0226-88:1(79-91)Online publication date: 27-Oct-2021
https://doi.org/10.1007/s41095-021-0226-8
Hu PYang B(2020)Visual perception driven 3D building structure representation from airborne laser scanning point cloudVirtual Reality & Intelligent Hardware10.1016/j.vrih.2020.05.0042:3(261-275)Online publication date: Jun-2020
https://doi.org/10.1016/j.vrih.2020.05.004
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Figures

Tables

Media

View Issue’s Table of Contents