Browse-to-Search: Interactive Exploratory Search with Visual Entities

Abstract

With the development of image search technology, users are no longer satisfied with searching for images using just metadata and textual descriptions. Instead, more search demands are focused on retrieving images based on similarities in their contents (textures, colors, shapes etc.). Nevertheless, one image may deliver rich or complex content and multiple interests. Sometimes users do not sufficiently define or describe their seeking demands for images even when general search interests appear, owing to a lack of specific knowledge to express their intents. A new form of information seeking activity, referred to as exploratory search, is emerging in the research community, which generally combines browsing and searching content together to help users gain additional knowledge and form accurate queries, thereby assisting the users with their seeking and investigation activities. However, there have been few attempts at addressing integrated exploratory search solutions when image browsing is incorporated into the exploring loop. In this work, we investigate the challenges of understanding users' search interests from the images being browsed and infer their actual search intentions. We develop a novel system to explore an effective and efficient way for allowing users to seamlessly switch between browse and search processes, and naturally complete visual-based exploratory search tasks. The system, called Browse-to-Search enables users to specify their visual search interests by circling any visual objects in the webpages being browsed, and then the system automatically forms the visual entities to represent users' underlying intent. One visual entity is not limited by the original image content, but also encapsulated by the textual-based browsing context and the associated heterogeneous attributes. We use large-scale image search technology to find the associated textual attributes from the repository. Users can then utilize the encapsulated visual entities to complete search tasks. The Browse-to-Search system is one of the first attempts to integrate browse and search activities for a visual-based exploratory search, which is characterized by four unique properties: (1) in session—searching is performed during browsing session and search results naturally accompany with browsing content; (2) in context—the pages being browsed provide text-based contextual cues for searching; (3) in focus—users can focus on the visual content of interest without worrying about the difficulties of query formulation, and visual entities will be automatically formed; and (4) intuitiveness—a touch and visual search-based user interface provides a natural user experience. We deploy the Browse-to-Search system on tablet devices and evaluate the system performance using millions of images. We demonstrate that it is effective and efficient in facilitating the user's exploratory search compared to the conventional image search methods and, more importantly, provides users with more robust results to satisfy their exploring experience.

References

1. A. Agarawala and R. Balakrishnan. 2006. Keepin' it real: Pushing the desktop metaphor with physics, piles and the pen. In Proceedings of the SIGCHI Conference on Human Factors in Computing System (CHI'06). ACM, New York, NY, 1283--1292.
2. D. Bainbridge, M. B. Twidale, and D. M. Nichols. 2011. A User-driven context-aware approach to erroneous metadata in digital libraries. In Proceedings of the ACM/IEEE Joint Conference on Digital Libraries. 39--48.
3. D. Bainbridge, M. B. Twidale, and D. M. Nichols. 2012. Interactive context-aware user-driven metadata correction in digital libraries. Int. J. Digital Lib. 13, 1, 17--32.
4. H. Cao, D. H. Hu, D. Shen, D. Jiang, J.-T. Sun, E. Chen, and Q. Yang. 2009. Context-aware query classification. In Proceedings of the 32nd International Conference on Research and Development in Information Retrieval (SIGIR). 3--10.
5. Y. Cao, H. Wang, C. Wang, Z. Li, L. Zhang, and L. Zhang. 2010. MindFinder: Interactive sketch-based image search on millions of images. In Proceedings of the International Conference on ACM Multimedia. 1605--1608.
6. V. Chandrasekhar, G. Takacs, D. M. Chen, S. S. Tsai, R. Grzeszczuk, and B. Girod. 2009. CHoG: Compressed histogram of gradients A low bit-rate feature descriptor. In Proceedings of the IEEE International Conference on Computer Vision and Pattern Recoginition (CVPR). 2504--2511.
7. E. Cheng, F. Jing, and L. Zhang. 2009. A unified relevance feedback framework for web image retrieval. IEEE Trans. Image Process. 18, 6, 1350--1357.
8. M. Dörk, C. Williamson, and S. Carpendale. 2012. Navigating tomorrow's Web: From searching and browsing to visual exploration. ACM Trans. Web 6, 3, 13:1--13:28.
9. W.-T. Fu, T. G. Kannampallil, and R. Kang. 2010. Facilitating exploratory search by model-based navigational cues. In Proceedings of the 14th International Conference on Intelligent User Interfaces (IUI'10). ACM, New York, NY, 199--208.
10. G. Golovchinsky, A. Dunnigan, and A. Diriye. 2012. Designing a tool for exploratory information seeking. In Proceedings of the Extended Abstracts on Human Factors in Computing Systems (CHI). 1799--1804.
11. Google Related. 2012. http://www.google.com/related.
12. R. Ji, L.-Y. Duan, J. Chen, H. Yao, Y. Rui, S.-F. Chang, and W. Gao. 2011. Towards low bit rate mobile visual search with multiple-channel coding. In Proceedings of the International Conference on Multimedia. 573--582.
13. A. Kerne, E. Koh, S. M. Smith, A. Webb, and B. Dworaczyk. 2008. combinFormation: Mixed-initiative composition of image and text surrogates promotes information discovery. ACM Trans. Inf. Syst. 27, 1, 5:1--5:45
14. B. Kules and B. Shneiderman. 2008. Users can change their web search tactics: Design guidelines for categorized overviews. Int. J. Inf. Process. Manag. 44, 2, 463--484.
15. S. Kullback and R. A. Leibler. 1951. On information and sufficiency. Ann. Math. Statist. 22, 1, 79--86.
16. X. Li. 2010. Understanding the semantic structure of noun phrase queries. In Proceedings of the 48th Annual Meeting of the ACL (ACL). 1337--1345.
17. Y. Liu, T. Mei, and X.-S. Hua. 2009. CrowdReranking: Exploring multiple search engines for visual search reranking. In Proceedings of SIGIR. 500--507.
18. F. Loumakis, S. Stumpf, and D. Grayson. 2011. This image smells good: Effects of image information scent in search engine results pages. In Proceedings of the 20th ACM International Conference on Information and Knowledge Management (CIKM). 475--484.
19. D. G. Lowe. 2004. Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vision 60, 2, 91--110.
20. S. Lu, T. Mei, J. Wang, J. Zhang, Z. Wang, D. D. Feng, J.-T. Sun, and S. Li. 2012. Browse-to-Search. (Video demo). In Proceedings of the International Conference on Multimedia.
21. W. Lu, J. Wang, X.-S. Hua, S. Wang, and S. Li. 2011. Contextual image search. In Proceedings of the International Conference on Multimedia. 513--522.
22. G. Marchionini. 2006. Exploratory search: From finding to understanding. Commun. ACM 49, 4, 41--46.
23. G. Marchionini and G. Geisler. 2002. The open video digital library. D-Lib Mag. 8, 12.
24. T. Mei, Y. Rui, S. Li, and Q. Tian. 2014. Multimedia search reranking: A literature survey. ACM Comput. Surv. 46, 3, 38:1--38:38.
25. T. Mei, B. Yang, X.-S. Hua, and S. Li. 2011. Contextual video recommendation by multimodal relevance and user feedback. ACM Trans. Inf. Syst. 29, 2, 10:1--10:24.
26. D. Milne, D. M. Nichols, and I. H. Witten. 2008. A competitive environment for exploratory query expansion. In Proceedings of the 8th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL). 197--200.
27. P. Pirolli. 2009. An elementary social information foraging model. In Proceedings of the SIGCHI Conference on Human Factors on Computing Systems. 605--614.
28. P. Pirolli, S. K. Card, and M. M. Van Der Wege. 2003. The effects of information scent on visual search in the hyperbolic tree browser. ACM Trans. Comput. Human Interact. 10, 1, 20--53.
29. D. Qin, S. Gammeter, L. Bossard, T. Quack, and L. J. Van Gool. 2011. Hello neighbor: Accurate object retrieval with k-reciprocal nearest neighbors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 777--784.
30. J Sang, T. Mei, Y.-Q. Xu, C. Zhao, C. Xu, and S. Li. 2013. Interaction design for mobile visual search. IEEE Trans. Multiamedia 15, 7, 1665--1676.
31. G. Schindler, M. Brown, and R. Szeliski. 2007. City-scale location recognition. In Proceedings of (CVPR).
32. D. Shen, J.-T. Sun, Q. Yang, and Z. Chen. 2006. Building bridges for web query classification. In Proceedings of SIGIR. 131--138.
33. D. A. Smith, A. Owens, M. C. Schraefel, P. Sinclair, P. André, M. Wilson, A. Russell, K. Martinez, and P. Lewis. 2007. Challenges in supporting faceted semantic browsing of multimedia collections. In Proceedings of the 2nd International Conference on Semantic and Digital Media Technologies (SAMT). Lecture Notes in Computer Science, Vol. 4816. Springer, Berlin, 280--283.
34. J. Wang and S. Li. 2012. Query-driven iterated neighborhood graph search for large scale indexing. In Proceedings of the International Conference on Multimedia.
35. R. W. White, B. Kules, S. M. Drucker, and M. M. C. Schraefel. 2006. Supporting exploratory search. Commun. ACM 49, 4.
36. R. W. White and R. A. Roth. 2009. Exploratory search: Beyond the query-response paradigm. Morgan & Claypool Publishers.
37. M. L. Wilson, P. André, and M. C. Schraefel. 2008. Backward highlighting: Enhancing faceted search. In Proceedings of the ACM Symposium on User Interface Software and Technology (UIST). 235--238.
38. M. L. Wilson and M. C. Schraefel. 2010. Evaluating collaborative information-seeking interfaces with a search-oriented inspection method and re-framed information seeking theory. Int. J. Inf. Process. Manag. 46, 6, 718--732.
39. H. Xu, J. Wang, X.-S. Hua, and S. Li. 2010. Image search by concept map. In Proceedings of the SIGIR. 275--282.
40. H. Xu, J. Wang, Z. Li, G. Zeng, S. Li, and N. Yu. 2011. Complementary hashing for approximate nearest neighbor search. In Proceedings of the IEEE International Conference on Computer Vision (ICCV). 1631--1638.
41. J. Xu and W. B. Croft. 1996. Query expansion using local and global document analysis. In Proceedings of the SIGIR. 4--11.
42. F. X. Yu, R. Ji, and S.-F. Chang. 2011. Active query sensing for mobile location search. In Proceedings of the International Conference on Multimedia. 3--12.
43. Z.-J. Zha, L. Yang, T. Mei, M. Wang, and Z. Wang. 2009. Visual query suggestion. In Proceedings of the International Conference on Multimedia. 15--24.
44. W. Zhou, H. Li, Y. Lu, and Q. Tian. 2013. SIFT match verification by geometric coding for large-scale partial-duplicate web image search. ACM Trans. Multimedia Comput. Commun. Appl. 9, 1, 4:1--4:18.

Index Terms

• Human-centered computing

  • Human computer interaction (HCI)

• Information systems

  • Information retrieval