skip to main content
research-article

WebTables: exploring the power of tables on the web

Published:01 August 2008Publication History
Skip Abstract Section

Abstract

The World-Wide Web consists of a huge number of unstructured documents, but it also contains structured data in the form of HTML tables. We extracted 14.1 billion HTML tables from Google's general-purpose web crawl, and used statistical classification techniques to find the estimated 154M that contain high-quality relational data. Because each relational table has its own "schema" of labeled and typed columns, each such table can be considered a small structured database. The resulting corpus of databases is larger than any other corpus we are aware of, by at least five orders of magnitude.

We describe the WEBTABLES system to explore two fundamental questions about this collection of databases. First, what are effective techniques for searching for structured data at search-engine scales? Second, what additional power can be derived by analyzing such a huge corpus?

First, we develop new techniques for keyword search over a corpus of tables, and show that they can achieve substantially higher relevance than solutions based on a traditional search engine. Second, we introduce a new object derived from the database corpus: the attribute correlation statistics database (AcsDB) that records corpus-wide statistics on co-occurrences of schema elements. In addition to improving search relevance, the AcsDB makes possible several novel applications: schema auto-complete, which helps a database designer to choose schema elements; attribute synonym finding, which automatically computes attribute synonym pairs for schema matching; and join-graph traversal, which allows a user to navigate between extracted schemas using automatically-generated join links.

References

  1. E. Agichtein, L. Gravano, V. Sokolovna, and A. Voskoboynik. Snowball: A prototype system for extracting relations from large text collections. In SIGMOD Conference, 2001. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. S. Agrawal, S. Chaudhuri, and G. Das. Dbxplorer: A system for keyword-based search over relational databases. In ICDE, 2002. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. S. Bell and P. Brockhausen. Discovery of data dependencies in relational databases. In European Conference on Machine Learning, 1995.Google ScholarGoogle Scholar
  4. T. Brants, A. C. Popat, P. Xu, F. J. Och, and J. Dean. Large language models in machine translation. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Language Learning, pages 858--867, 2007.Google ScholarGoogle Scholar
  5. M. Cafarella, A. Halevy, Z. Wang, E. Wu, and Y. Zhang. Uncovering the relational web. In under review, 2008.Google ScholarGoogle Scholar
  6. M. J. Cafarella, D. Suciu, and O. Etzioni. Navigating extracted data with schema discovery. In Web DB, 2007.Google ScholarGoogle Scholar
  7. H. Chen, S. Tsai, and J. Tsai. Mining tables from large scale html texts. In 18th International Conference on Computational Linguistics (COLING), pages 166--172, 2000. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. K. W. Church and P. Hanks. Word association norms, mutual information, and lexicography. In Proceedings of the 27th Annual Association for Computational Linguistics, 1989. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. R. Dhamankar, Y. Lee, A. Doan, A. Y. Halevy, and P. Domingos. imap: Discovering complex mappings between database schemas. In SIGMOD Conference, 2004. Google ScholarGoogle ScholarDigital LibraryDigital Library
  10. A. Doan, P. Domingos, and A. Y. Halevy. Reconciling schemas of disparate data sources: A machine-learning approach. In SIGMOD Conference, 2001. Google ScholarGoogle ScholarDigital LibraryDigital Library
  11. O. Etzioni, M. Cafarella, D. Downey, S. Kok, A. Popescu, T. Shaked, S. Soderland, D. Weld, and A. Yates. Web-scale information extraction in knowitall (preliminary results). In Thirteenth International World Wide Web Conference, 2004. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. W. Gatterbauer, P. Bohunsky, M. Herzog, B. Krüpl, and B. Pollak. Towards domain-independent information extraction from web tables. In Proceedings of the 16th International World Wide Web Conference (WWW 2007), pages 71--80, 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. B. He, Z. Zhang, and K. C.-C. Chang. Knocking the door to the deep web: Integration of web query interfaces. In SIGMOD Conference, pages 913--914, 2004. Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. V. Hristidis and Y. Papakonstantinou. Discover: Keyword search in relational databases. In VLDB, 2002. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. D. Lin and P. Pantel. Dirt: Discovery of inference rules from text. In KDD, 2001. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. J. Madhavan, P. A. Bernstein, A. Doan, and A. Y. Halevy. Corpus-based schema matching. In ICDE, 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. J. Madhavan, P. A. Bernstein, and E. Rahm. Generic schema matching with cupid. In VLDB, 2001. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. J. Madhavan, A. Y. Halevy, S. Cohen, X. L. Dong, S. R. Jeffery, D. Ko, and C. Yu. Structured data meets the web: A few observations. IEEE Data Eng. Bull., 29(4): 19--26, 2006.Google ScholarGoogle Scholar
  19. C. Manning and H. Schütze. Foundations of Statistical Natural Language Processing. MIT Press, 1999. Google ScholarGoogle ScholarDigital LibraryDigital Library
  20. I. R. Mansuri and S. Sarawagi. Integrating unstructured data into relational databases. In ICDE, 2006. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. R. Miller and P. Andritsos. Schema discovery. IEEE Data Eng. Bull., 26(3):40--45, 2003.Google ScholarGoogle Scholar
  22. A. Nandi and H. V. Jagadish. Assisted querying using instant-response interfaces. In SIGMOD Conference, pages 1156--1158, 2007. Google ScholarGoogle ScholarDigital LibraryDigital Library
  23. G. Penn, J. Hu, H. Luo, and R. McDonald. Flexible web document analysis for delivery to narrow-bandwidth devices. In International Conference on Document Analysis and Recognition (ICDAR01), pages 1074--1078, 2001. Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. E. Rahm and P. A. Bernstein. A survey of approaches to automatic schema matching. VLDB Journal, 10(4):334--350, 2001. Google ScholarGoogle ScholarDigital LibraryDigital Library
  25. P. D. Turney. Mining the web for synonyms: Pmi-ir versus Isa on toefl. In Proceedings of the Twelfth European Conference on Machine Learning, 2001. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Y. Wang and J. Hu. A machine learning based approach for table detection on the web. In Eleventh International World Wide Web Conference, 2002. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. I. Witten and E. Frank. Data Mining: Practical machine learning tools and techniques. Morgan Kaufman, San Francisco, 2nd edition edition, 2005. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. S. Wong, C. Butz, and Y. Xiang. Automated database schema design using mined data dependencies. Journal of the American Society of Information Science, 49(5):455--470, 1998. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. M. Yoshida and K. Torisawa. A method to integrate tables of the world wide web. In Proceedings of the 1st International Workshop on Web Document Analysis, pages 31--34, 2001.Google ScholarGoogle Scholar
  30. R. Zanibbi, D. Blostein, and J. Cordy. A survey of table recognition: Models, observations, transformations, and inferences, 2003.Google ScholarGoogle Scholar

Index Terms

  1. WebTables: exploring the power of tables on the web

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader