Benchmarking XML processors for applications in grid web services

ABSTRACT

Web services based specifications have emerged as the underlying architecture for core grid services and standards, such as WSRF. XML is inextricably inter-twined with Web services based specifications, and as a result the design and implementation of XML processing tools plays a significant role in grid applications. These applications use XML in a wide variety of ways, including workflow specifications, WS-Security based documents, service descriptions in WSDL, and on-the-wire format in SOAP-based communication. The application characteristics also vary widely in the use of XML messages in their performance, memory, size, and processing requirements. Numerous XML processing tools exist today, each of which is optimized for specific features. To make the right decisions, grid application and middleware developers must thus understand the complex dependencies between XML features and the application. We propose a standard benchmark suite for quantifying, comparing, and contrasting the performance of XML processors under a wide range of representative use cases. The benchmarks are defined by a set of XML schemas and conforming documents. To demonstrate the utility of the benchmarks and to provide a snapshot of the current XML implementation landscape, we report the performance of many different XML implementations, on the benchmarks, and draw conclusions about their current performance characteristics. We also present a brief analysis on the current shortcomings and required critical design changes for multi-threaded XML processing tools to run efficiently on emerging multi-core architectures.¹

REFERENCES

• 1: Abu-Ghazaleh, N., Govindaraju, M., and Lewis, M. J. 2004. Optimizing performance of web services with chunk-overlaying and pipelined-send. Proceedings of the International Conference on Internet Computing (ICIC) (June), 482--485.
• 2: Nayef Abu-Ghazaleh , Michael J. Lewis , Madhusudhan Govindaraju, Differential Serialization for Optimized SOAP Performance, Proceedings of the 13th IEEE International Symposium on High Performance Distributed Computing (HPDC'04), p.55-64, June 04-06, 2004 [doi>10.1109/HPDC.2004.8]
• 3: Abu-Ghazaleh, N., Lewis, M. J., and Govindaraju, M. 2004. Performance of Dynamic Resizing of Message Fields for Differential Serialization of SOAP Messages. Proceedings of the International Symposium on Web Services and Applications (June), 783--789.
• 4: Axis Java, 2002. The Apache Project. http://ws.apache.org/axis/.
• 5: Bailey, D., Barszcz, E., Barton, J., Browning, D., Carter, R., Dagum, L., Fatoohi, R., Fineberg, S., Frederickson, P., Lasinski, T., Schreiber, R., Simon. H., Venkatakrishnan, V., and Weeratunga, S., 1994. The NAS Parallel Benchmarks. http://www.nas.nasa.gov/Software/NPB/.
• 6: Barron, E. J., Battisti, D. S., Boville, B. A., Bryan, K., Carrier, G. F., Cess, R. D., Davis, R. E., Ghil, M., Hall, M. M., Karl, T. R., Kiehl, J. T., Martinson, D. G., Parkinson, C. L., Saltzman, B., and Turco, R. P. 1994. Global ocean-atmosphere- land system (GOALS) for predicting seasonal-to-interannual climate. National Academy Press, Washington, D.C.
• 7: Fran Berman , Geoffrey Fox , Anthony J. G. Hey , Tony Hey, Grid Computing: Making the Global Infrastructure a Reality, John Wiley & Sons, Inc., New York, NY, 2003
• 8: Chilingaryan. S. A., 2003, XML benchmark. http://xmlbench.sourceforge.net/.
• 9: Kenneth Chiu , Madhusudhan Govindaraju , Randall Bramley, Investigating the Limits of SOAP Performance for Scientific Computing, Proceedings of the 11 th IEEE International Symposium on High Performance Distributed Computing HPDC-11 20002 (HPDC'02), p.246, July 24-26, 2002
• 10: Christensen, E., Curbera, F., Meredith, G., and Weerawarana, S., 2001. Web Services Description Language (WSDL) 1.1, March, http://www.w3.org/TR/wsdl.
• 11: Chun, G., Dail, H., Casanova, H., and Snavel, A. 2004. Benchmark probes for grid assessmen. In In Proceedings of the High-Performance Grid Computing Workshop.
• 12: Clark, J., 1998. The expat xml parser. http://expat.sourceforge.net/.
• 13: Devsphere, 2000. The XML parsing benchmark. http://www.devsphere.com/xml/benchmark/.
• 14: Ian Foster , Carl Kesselman, The Grid 2: Blueprint for a New Computing Infrastructure, Morgan Kaufmann Publishers Inc., San Francisco, CA, 2003
• 15: Foster, I., Kishimoto, H., Savva, A., Berry, D., Djaoui, A., Grimshaw, A., Horn, B., Maciel, F., Siebenlist, F., Subramaniam, R., Treadwell, J., and Reich, J. V. 2005. The open grid services architecture, version 1.0. Global Grid Forum (January). http://www.gridforum.org/documents/GWD-I-E/GWD-I.030.pdf.
• 16: Michael Frumkin , Rob F. Van der Wijngaart, NAS Grid Benchmarks: A Tool for Grid Space Exploration, Cluster Computing, v.5 n.3, p.247-255, July 2002 [doi>10.1023/A:1015669003494 ]
• 17: Gannon, D., Krishnan, S., Fang, L., Kandaswamy, G., Simmhan, Y., and Slominski, A. 2004. On building parallel and grid applications: Component technology and distributed services. In CLADE 2004, Challenges of Large Applications in Distributed Environments. IEEE Computer Society Press.
• 18: Globus Toolkit, 2002. Globus Alliance. http://wwwunix.globus.org/toolkit/downloads/.
• 19: Madhusudhan Govindaraju , Aleksander Slominski , Venkatesh Choppella , Randall Bramley , Dennis Gannon, Requirements for and evaluation of RMI protocols for scientific computing, Proceedings of the 2000 ACM/IEEE conference on Supercomputing (CDROM), p.61-es, November 04-10, 2000, Dallas, Texas, United States
• 20: Govindaraju, M., Lewis, M., Chiu, K., Engelen, R., Lang, S., and Jackson, K. 2005. Web services performance aspects. In The Proceedings of GlobusWorld.
• 21: Gudgin, M., Hadley, M., Mendelsohn, N., Moreau, J.-J., Canon, and Nielsen, H. F., 2003. Simple object access protocol 1.1, June. http://www.w3.org/TR/SOAP.
• 22: Hapmap, 2003. International HapMap Project. http://www.hapmap.org/abouthapmap.html.
• 23: Haustein, S., 2000. kxml pull parser, July. http://kxml.sourceforge.net/.
• 24: Michael R. Head , Madhusudhan Govindaraju , Aleksander Slominski , Pu Liu , Nayef Abu-Ghazaleh , Robert van Engelen , Kenneth Chiu , Michael J. Lewis, A Benchmark Suite for SOAP-based Communication in Grid Web Services, Proceedings of the 2005 ACM/IEEE conference on Supercomputing, p.19, November 12-18, 2005 [doi>10.1109/SC.2005.2]
• 25: Tony Hey , David Lancaster, The Development of Parkbench and Performance Prediction, International Journal of High Performance Computing Applications, v.14 n.3, p.205-215, August 2000 [doi>10.1177/109434200001400304]
• 26: Yi Huang , Aleksander Slominski , Chathura Herath , Dennis Gannon, WS-Messenger: A Web Services-Based Messaging System for Service-Oriented Grid Computing, Proceedings of the Sixth IEEE International Symposium on Cluster Computing and the Grid (CCGRID'06), p.166-173, May 16-19, 2006 [doi>10.1109/CCGRID.2006.109]
• 27: Humphrey, M., and Wasson, G. 2005. Architectural foundations of wsrf.net. International Journal of Web Services Research 2, 2 (April-June), 83--97.
• 28: Illinca, F., Hetu, J.-F., and Bramley, R., 1997. Simulation of 3-d mold-filling and solidification processes on distributed memory parallel architectures, November. Proceedings of International Mechanical Engineering Congress & Exposition.
• 29: Kepler, 2003. The Kepler Project. http://www.kepler-project.org/.
• 30: Kohlhoff, C., and Steele, R. 2004. Evaluating SOAP for High Performance Applications in Capital Markets. Journal of Computer Systems, Science, and Engineering 63, 4 (July), (241--251).
• 31: Krishnan, S., Baldridge, K., Greenberg, J., Stearn. B., and Bhatia, K. 2005. An end-to-end web services-based infrastructure for biomedical applications. In In Grid 2005, 6th IEEE/ACM International Workshop on Grid Computing.
• 32: LEAD Events, 2003. Indiana University Extreme Computing Laboratory. http://www.extreme.indiana.edu/xgws/messenger/.
• 33: Luszczek, P., Dongarra, J., Koester, D., Raben-seifner. R., Lucas, B., Kepner, J., McCalpin, J., Bailey, D., and Takahashi, D., 2005. Introduction to the HPC Challenge Benchmark Suite, March. http://ic1.cs.utk.edu/hpcc/pubs/index.htm.
• 34: McCalpin, J. D., 1997. STREAM: Sustainable Memory Bandwidth in High Performance Computers, June. http://www.cs.virginia.edu/stream.
• 35: Nagaratnam, N., and Humphrey, M., 2003. Open grid service architecture security working group (ogsasec-wg). http://www.cs.virginia.edu/humphrey/ogsa-secwg/.
• 36: Oren, Y., 2002. Piccolo XML Parser for Java, March. http://piccolo.sourceforge.net/.
• 37: Petitet, A., Whaley, R. C., Dongarra, J., and Clear, A. 2004. Hpl - a portable implementation of the high-performance linpack benchmark for distributed-memory computers. Tech. rep., Innovative Computing Lab, University of Tennessee, January. http://www.netlib.org/benchmark/hpl/.
• 38: A. R. Schmidt , Florian Waas , Martin L. Kersten , D. Florescu , I. Manolescu , M. J. Carey , R. Busse, The XML benchmark project, CWI (Centre for Mathematics and Computer Science), Amsterdam, The Netherlands, 2001
• 39: Gurmeet Singh , Shishir Bharathi , Ann Chervenak , Ewa Deelman , Carl Kesselman , Mary Manohar , Sonal Patil , Laura Pearlman, A Metadata Catalog Service for Data Intensive Applications, Proceedings of the 2003 ACM/IEEE conference on Supercomputing, p.33, November 15-21, 2003
• 40: Slominski, A., Govindaraju, M., Gannon, D., and Bramley, R. 2001. Design of an XML based Interoperable RMI System: SoapRMI C++/Java 1.1. In Proceedings of PDPTA, 1661-1667.
• 41: Slominski, A., 2004. XSOAP Toolkit. http://www.extreme.indiana.edu/xgws/.
• 42: Slominski, A., 2005. Scientific workflows survey. http://www.extreme.indiana.edu/swf-survey/.
• 43: SoapWare.org. 2001. The Leading Directory for SOAP 1.1 Developers, May. http://www.soapware.org/directory/4/implementations.
• 44: SPEC, 1992. The SPEC Benchmarks. http://www.specbench.org.
• 45: Trolltech, 1998. Qt C++ Application Development Framework, October. http://www.trolltech.com/products/qt/.
• 46: Robert A. Van Engelen , Kyle A. Gallivan, The gSOAP Toolkit for Web Services and Peer-to-Peer Computing Networks, Proceedings of the 2nd IEEE/ACM International Symposium on Cluster Computing and the Grid, p.128, May 21-24, 2002
• 47: Van Engelen, R., Zhang, W., and Govindaraju, M. 2006. Toward remote object coherence with compiled object serialization for distributed computing with xml web services. In in the proceedings of Compilers for Parallel Computing (CPC), 441--455.
• 48: Van Engelen, R. 2003. Pushing the SOAP envelope with Web services for scientific computing. In proceedings of the International Conference on Web Services (ICWS), 346--352.
• 49: Robert van Engelen, Code generation techniques for developing light-weight XML Web services for embedded devices, Proceedings of the 2004 ACM symposium on Applied computing, March 14-17, 2004, Nicosia, Cyprus [doi>10.1145/967900.968075]
• 50: Van Engelen, R., 2004. Constructing finite state automata for high performance xml web services.
• 51: Veillard, D., 1998. The XML C Parser and toolkit of Gnome, February. http://xmlsoft.org/. W3C. Canonical XML. http://www.w3.org/TR/xml-c14n.
• 52: Steven Cameron Woo , Moriyoshi Ohara , Evan Torrie , Jaswinder Pal Singh , Anoop Gupta, The SPLASH-2 programs: characterization and methodological considerations, Proceedings of the 22nd annual international symposium on Computer architecture, p.24-36, June 22-24, 1995, S. Margherita Ligure, Italy
• 53: WSRF, 2004. Web services resource framework 1.2, December. http://www.oasis-open.org/committees/wsrf/.
• 54: Xerces, 2003. Xerces XML Parser, September. http://xerces.apache.org/.
• 55: XMethods.com, 2001. SOAPBuilders Interoperability Lab. http://www.xmethods.com/ilab/.
• 56: Wei Zhang , Robert A. van Engelen, TDX: a high-performance table-driven XML parser, Proceedings of the 44th annual Southeast regional conference, March 10-12, 2006, Melbourne, Florida [doi>10.1145/1185448.1185606]
• 57: Zhang, J., 2003. Virtual Token Descriptor (VTD) XML Parser. http://vtd-xml.sourceforge.net/.

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.5 On-line Information Services
          Subjects: Web-based services

Additional Classification:
  H. Information Systems
  H.2 DATABASE MANAGEMENT
      H.2.1 Logical Design
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.4 Systems and Software
          Subjects: Performance evaluation (efficiency and effectiveness)

  I. Computing Methodologies
  I.7 DOCUMENT AND TEXT PROCESSING
      I.7.2 Document Preparation

          Nouns: XML

General Terms:
Design, Performance

Keywords:
XML, benchmarking, multi-core