Interpretable Partitioned Embedding for Customized Multi-item Fashion Outfit Composition

ABSTRACT

Intelligent fashion outfit composition becomes more and more popular in these years. Some deep learning based approaches reveal competitive composition recently. However, the uninterpretable characteristic makes such deep learning based approach cannot meet the designers, businesses and consumers' urge to comprehend the importance of different attributes in an outfit composition. To realize interpretable and customized multi-item fashion outfit compositions, we propose a partitioned embedding network to learn interpretable embeddings from clothing items. The network consists of two vital components: attribute partition module and partition adversarial module. In the attribute partition module, multiple attribute labels are adopted to ensure that different parts of the overall embedding correspond to different attributes. In the partition adversarial module, adversarial operations are adopted to achieve the independence of different parts. With the interpretable and partitioned embedding, we then construct an outfit composition graph and an attribute matching map. Extensive experiments demonstrate that 1) the partitioned embedding have unmingled parts which corresponding to different attributes and 2) outfits recommended by our model are more desirable in comparison with the existing methods.

References

1. Kaori Abe, Teppei Suzuki, Shunya Ueta, Akio Nakamura, Yutaka Satoh, and Hirokatsu Kataoka. 2017. Changing Fashion Cultures. arXiv preprint arXiv:1703.07920 (2017).
2. Lukas Bossard, Matthias Dantone, Christian Leistner, Christian Wengert, Till Quack, and Luc Van Gool. 2012. Apparel classification with style. In Asian conference on computer vision. Springer, 321--335.
3. Diane Bouchacourt, Ryota Tomioka, and Sebastian Nowozin. 2017. Multi-level variational autoencoder: Learning disentangled representations from grouped observations. arXiv preprint arX-iv:1705.08841 (2017).
4. Huizhong Chen, Andrew Gallagher, and Bernd Girod. 2012. Describing clothing by semantic attributes. Computer Vision ECCV 2012 (2012), 609--623.
5. Qiang Chen, Junshi Huang, Rogerio Feris, Lisa M Brown, Jian Dong, and Shuicheng Yan. 2015. Deep domain adaptation for describing people based on fine-grained clothing attributes. In Proceedings of the IEEE conference on computer vision and pattern recognition. 5315--5324.
6. Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. 2016. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in Neural Information Processing Systems. 2172--2180.
7. Zunlei Feng, Wolong Yuan, Chunli Fu, Jie Lei, and Mingli Song. 2017. Finding intrinsic color themes in images with human visual perception. Neurocomputing (2017).
8. Jianlong Fu, Jinqiao Wang, Zechao Li, Min Xu, and Hanqing Lu. 2012. Efficient clothing retrieval with semantic-preserving visual phrases. In Asian Conference on Computer Vision. Springer, 420--431.
9. M Hadi Kiapour, Xufeng Han, Svetlana Lazebnik, Alexander C Berg, and Tamara L Berg. 2015. Where to buy it: Matching street clothing photos in online shops. In Proceedings of the IEEE International Conference on Computer Vision. 3343--3351.
10. Ruining He and Julian McAuley. 2016. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In proceedings of the 25th international conference on world wide web. International World Wide Web Conferences Steering Committee, 507--517.
11. Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexan- der Lerchner. 2016. beta-vae: Learning basic visual concepts with a constrained variational framework. (2016).
12. Junshi Huang, Rogerio S Feris, Qiang Chen, and Shuicheng Yan. 2015. Cross-domain image retrieval with a dual attribute-aware ranking network. In Proceedings of the IEEE International Conference on Computer Vision. 1062--1070.
13. Tomoharu Iwata, Shinji Wanatabe, and Hiroshi Sawada. 2011. Fashion coordinates recommender system using photographs from fashion magazines. In IJCAI, Vol. 22. 2262.
14. Vignesh Jagadeesh, Robinson Piramuthu, Anurag Bhardwaj, Wei Di, and Neel Sundaresan. 2014. Large scale visual recommenda- tions from street fashion images. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 1925--1934.
15. M Hadi Kiapour, Kota Yamaguchi, Alexander C Berg, and Tamara L Berg. 2014. Hipster wars: Discovering elements of fashion styles. In European conference on computer vision. Springer, 472--488.
16. Diederik P Kingma and Max Welling. 2014. Auto-Encoding Variational Bayes. stat 1050 (2014), 1.
17. Tejas D Kulkarni, William F Whitney, Pushmeet Kohli, and Josh Tenenbaum. 2015. Deep convolutional inverse graphics network. In Advances in Neural Information Processing Systems. 2539--2547.
18. Yuncheng Li, Liangliang Cao, Jiang Zhu, and Jiebo Luo. 2017. Mining Fashion Outfit Composition Using An End-to-End Deep Learning Approach on Set Data. IEEE Transactions on Multi- media (2017).
19. Qiang Liu, Shu Wu, and Liang Wang. 2017. DeepStyle: Learning User Preferences for Visual Recommendation. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 841--844.
20. Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. 2016. Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1096--1104.
21. Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of machine learning research 9, Nov (2008), 2579--2605.
22. Kevin Matzen, Kavita Bala, and Noah Snavely. 2017. Street- Style: Exploring world-wide clothing styles from millions of photos. arXiv preprint arXiv:1706.01869 (2017).
23. Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. 2015. Image-based recommendations on styles and substitutes. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 43--52.
24. Jan Morovic. 1998. To develop a universal gamut mapping algorithm. (1998).
25. Jose Oramas and Tinne Tuytelaars. 2016. Modeling visual com- patibility through hierarchical mid-level elements. arXiv preprint arXiv:1604.00036 (2016).
26. Guim Perarnau, Joost van de Weijer, Bogdan Raducanu, and Jose M Álvarez. 2016. Invertible conditional gans for image editing. arXiv preprint arXiv:1611.06355 (2016).
27. Jürgen Schmidhuber. 2008. Learning factorial codes by predictability minimization. Learning 4, 6 (2008).
28. N Siddharth, Brooks Paige, Alban Desmaison, Jan-Willem van de Meent, Frank Wood, Noah D Goodman, Pushmeet Kohli, and Philip HS Torr. 2016. Learning Disentangled Representations in Deep Generative Models. (2016).
29. Edgar Simo-Serra, Sanja Fidler, Francesc Moreno-Noguer, and Raquel Urtasun. 2015. Neuroaesthetics in fashion: Modeling the perception of fashionability. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 869--877.
30. Edgar Simo-Serra and Hiroshi Ishikawa. 2016. Fashion style in 128 floats: Joint ranking and classification using weak data for feature extraction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 298--307.
31. Louis L Thurstone. 1927. A law of comparative judgment. Psychological review 34, 4 (1927), 273.
32. Andreas Veit, Serge Belongie, and Theofanis Karaletsos. 2017. Conditional similarity networks. Computer Vision and Pattern Recognition (CVPR 2017) (2017).
33. Andreas Veit, Balazs Kovacs, Sean Bell, Julian McAuley, Kavita Bala, and Serge Belongie. 2015. Learning visual clothing style with heterogeneous dyadic co-occurrences. In Proceedings of the IEEE International Conference on Computer Vision. 4642--4650.
34. Sirion Vittayakorn, Kota Yamaguchi, Alexander C Berg, and Tamara L Berg. 2015. Runway to realway: Visual analysis of fashion. In Applications of Computer Vision (WACV), 2015 IEEE Winter Conference on. IEEE, 951--958.
35. Chaoyue Wang, Chaohui Wang, Chang Xu, and Dacheng Tao. 2017. Tag disentangled generative adversarial network for ob- ject image re-rendering. In Proceedings of the Twenty-Sixth International Joint Conference on Arti cial Intelligence, IJCAI. 2901--2907.
36. Xiaolong Wang and Abhinav Gupta. 2016. Generative image modeling using style and structure adversarial networks. In European Conference on Computer Vision. Springer, 318--335.
37. Xianwang Wang and Tong Zhang. 2011. Clothes search in con- sumer photos via color matching and attribute learning. In Proceedings of the 19th ACM international conference on Multi- media. ACM, 1353--1356.
38. Kota Yamaguchi, M Hadi Kiapour, Luis E Ortiz, and Tamara L Berg. 2012. Parsing clothing in fashion photographs. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Con- ference on. IEEE, 3570--3577.
39. Kota Yamaguchi, Takayuki Okatani, Kyoko Sudo, Kazuhiko Murasaki, and Yukinobu Taniguchi. 2015. Mix and Match: Joint Model for Clothing and Attribute Recognition.. In BMVC. 51--1.
40. Zhengzhong Zhou, Yifei Xu, Jingjin Zhou, and Liqing Zhang. 2016. Interactive Image Search for Clothing Recommendation. In Proceedings of the 2016 ACM on Multimedia Conference. ACM, 754--756

Index Terms

• Computing methodologies

  • Artificial intelligence

    • Computer vision

      • Computer vision representations

        • Image representations

  • Machine learning

    • Machine learning approaches

      • Learning latent representations

      • Neural networks