Indexing and searching tera-scale Grid-Based Digital Libraries

ABSTRACT

The University of California, Berkeley and the University of Liverpool in conjunction with the San Diego Supercomputer Center are developing a framework for Grid-Based Digital Library systems and Information Retrieval Services (Cheshire3) that operates in both single-processor and distributed computing environments. In this paper we discuss some results of testing Grid-based parallel approaches in indexing and retrieval for a variety of information resources, ranging from small test collections like the TREC and INEX collections, to medium-scale metadata collections like Medline and a test version of University of California Online Union Catalog, MELVYL (with 15 million and 16.5 million records respectively) ranging up to large-scale collections like the US National Records and Archives Administration (NARA) Preservation Prototype. This paper examines our approaches to indexing and retrieving from these collections and the architecture of the system that supports them.

REFERENCES

• 1: A. Rajasekar, M. Wan, R. Moore, W. Schroeder, G. Kremenek, A. Jagatheesan, C. Cowart, B. Zhu, S.-Y. Chen, and R. Olschanowsky, "Storage resource broker - managing distributed data in a grid," Computer Society of India Journal, Vol. 33, no. 4, pp. 42--54, 2003.
• 2: Ian Foster , Carl Kesselman, The Grid 2: Blueprint for a New Computing Infrastructure, Morgan Kaufmann Publishers Inc., San Francisco, CA, 2003
• 3: Ray R. Larson , Jerome McDonough , Paul O'Leary , Lucy Kuntz , Ralph Moon, Cheshire II: designing a next-generation online catalog, Journal of the American Society for Information Science, v.47 n.7, p.555-567, July 1996 [doi>10.1002/(SICI)1097-4571(199607)47:7<555::AID-ASI7>3.0.CO;2-T]
• 4: C. Lagoze, H. Van de Sompel, M. Nelson, S. Warner, "The Open Archives Initiative Protocol for Metadata Harvesting, Version 2.0". June 2002. http://www.openarchives.org/OAI/2.0/openarchivesprotocol.htm
• 5: R. Denenberg, R. Sanderson, M. Dovey et al. (eds), "SRW - Search/Retrieve Webservice", February 2004. http://www.loc.gov/srw
• 6: OASIS WSBPEL Technical Committee, "Web Services Business Process Execution Language". 2005. http://www.oasis-open.org/committees/wsbpel/charter.php
• 7: Bertram Ludäscher , Ilkay Altintas , Chad Berkley , Dan Higgins , Efrat Jaeger , Matthew Jones , Edward A. Lee , Jing Tao , Yang Zhao, Scientific workflow management and the Kepler system: Research Articles, Concurrency and Computation: Practice & Experience, v.18 n.10, p.1039-1065, August 2006 [doi>10.1002/cpe.v18:10]
• 8: Ray R. Larson , Robert Sanderson, Grid-based digital libraries: cheshire3 and distributed retrieval, Proceedings of the 5th ACM/IEEE-CS joint conference on Digital libraries, June 07-11, 2005, Denver, CO, USA [doi>10.1145/1065385.1065409]
• 9: A. Geist, A. Beguelin, J. Dongarra, W. Jiang, R. Manchek, V. Sunderam, "PVM: Parallel Virtual Machine" MIT Press, 1995
• 10: Message Passing Interface Forum. "MPI: A Message-Passing Interface standard (version 1.1)", Technical report, 1995. http://www.mpiforum.org.
• 11: W3C, "SOAP Specifications" June 2003. http://www.w3.org/TR/soap/
• 12: N. Nassar, G. Newby, K. Gamiel, M. Dovey, J. Morris, "Grid Information Retrieval Architecture", 2003, urlhttp://www.gir-wg.org/
• 13: DILIGENT Project, "Architectural Overview", April 2005, http://diligentproject.org/content/view/71/99
• 14: SleepyCat Software, "Berkeley DB", 2005, http://sleepycat.com/products/db.shtml
• 15: L. Declerck, C. Frymann, "DSpace/SRB Integration", CNI Fall Task Force Meeting, http://libnet.ucsd.edu/nara/
• 16: "About the TeraGrid", 2005. http://www.teragrid.org/about/index.html
• 17: "National Centre for Text Mining", 2005. http://www.nactem.ac.uk
• 18: National Library of Medicine, "Medline Factsheet". http://www.nlm.nih.gov/pubs/factsheets/medline.html
• 19: Tsujii, Jun'ichi, "Tsujii Laboratory", 2005. http://www-tsujii.is.s.u-tokyo.ac.jp/
• 20: "PyMPI", 2005, http://pympi.sourceforge.net/
• 21: The National Archives, "National Archives Electronic Records Archives Option Award Announcement" 2005. http://www.archives.gov/era/acquisition/option-award.html
• 22: T. Phelps, P. Watry, "A No-Compromises Architecture for Digital Document Preservation" in Research and Advanced Technology for Digital Libraries 9th European Conference, ECDL2005, Proceedings 2005, pp. 266--277
• 23: P Tooby, "Building a 'Memory' for the National Science Digital Library", in NPACI Online Vol. 7, issue 2, January 2003. http://www.npaci.edu/online/v7.2/nsdl.html

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.1 Content Analysis and Indexing
          Subjects: Indexing methods

Additional Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.3 Information Search and Retrieval
          Subjects: Search process
      H.3.4 Systems and Software
          Subjects: Distributed systems
      H.3.7 Digital Libraries

General Terms:
Algorithms, Design, Management