J2EE infrastructure scalability and throughput estimation

ABSTRACT

ECperf, the widely recognized industry standard J2EE benchmark, has attracted a large number of results submissions and their subsequent publication. However, ECperf places little restriction on the hardware platform, operating systems and databases utilized in the benchmarking process. This, combined with the existence of only two primary metrics, makes it difficult to accurately compare the results, or the performance of the Application Server products themselves. By mining the full-disclosure archives for trends and correlations we have discovered that J2EE technology is very scalable with increasing middle-tier resources, as long as the database has sufficient resources to prevent it becoming a bottleneck. Other observed trends include, a linear correlation between middle-tier total processing power and throughput, as well as between J2EE Application Server license costs and throughput. However, the results clearly indicate that there is an increasing cost per user with increasing capacity systems. Finally, the correlation between middle-tier processing power and throughput, combined with results obtained from a different 'lighter-weight' benchmark, facilitates an estimate of throughput for different types of J2EE applications.

REFERENCES

• 1: ECperf Specification and Kit, <u>http://java.sun.com/j2ee/ecperf/</u>
• 2: ECperf results: <u>www.theserverside.com/ecperf</u>
• 3: JSR 4 - ECperf Benchmark Specification, <u>http://jcp.org/en/jsr/detail?id=004</u>
• 4: J2EE compatibility, <u>http://java.sun.eom/j2ee/compatibility.html</u>
• 5: JBoss ECperf kit from CSIRO, <u>http://www.cmis.csiro.au/adsat/jbossecperf.htm</u>
• 6: Paul Brebner et al, Evaluating J2EE Application Servers, CSIRO MTE Report, CSIRO Publishing and Cutter Consortium, 2002: <u>http://www.cmis.csiro.au/mte</u>
• 7: Nile benchmark, <u>http://www.gotdotnet.com/team/compare/Nile%20Benchmark%20Results.doc</u>
• 8: NASDAQ transaction rate, <u>http://www.nasdaqnews.com/news/pr2002/nesection02165.html</u>
• 9: TPC-C analysis, <u>http://www.benchmarkresources.com/handbook/chapterl2.pdf</u>
• 10: Gunther, N, Commerical Clusters and Scalability, at: <u>http://www.teamquest.com/html/gunther/scalability.shtml</u>
• 11: Brebner, P, "Is your AppServer being Crippled by the JVM?", in proceedings of: BorCon2002, 5th Annual Asia Pacific Borland Conference, Australia, Sydney.
• 12: SPEC#8482; Information: <u>http://www.spec.org/spec/trademarks.html</u>
• 13: SPECjAppServer2001: <u>http://www.spec.org/osg/jApp Server2001/</u>
• 14: Samuel D. Kounev and Alejandro P. Buchmann, Implementing and Optimising Sun's ECperf Benchmark with WebLogic Server, BEA Technology Conference, 14 November 2001, Frankfurt: <u>http://de.bea.com/events/techkonf2001/presentations/pdf/hik 2c.pdf</u>
• 15: Samuel D. Kounev, Eliminating ECperf Persistence Bottlenecks when using RDBMS with Pessimistic Concurrency Control, <u>http://www.dvsl.informatik.tudarmstadt.de/DVS 1/staff/skounev/pub/ecperf1.0-opt-proposal.pdf</u>
• 16: Samuel D. Kounev and Alejandro P. Buchmann, Performance Issues in E-Business Systems, in: SSGRR 2002W (International Conference on Advances in Infrastructure for e-Business, e-Education, e-Science, e-Medicine, and Mobile Technologies on the Internet): <u>http://www.ssgrr.it/en/ssgrr2002w/papers/1.pdf</u>
• 17: Samuel D. Kounev and Alejandro P. Buchmann, Improving Data Access of J2EE Applications by Exploiting Asynchronous Messaging and Caching Services, in: VLDB 2002: <u>http://www.cs.ust.hk/vldb2002/VLDB2002-papers/S16P03.pdf</u>
• 18: Brebner, P., Chen, S., Stock-Online Open Source Software, available from the ObjectWeb Java Middleware Open Benchmarking project: <u>http://jmob.objectweb.org/</u>
• 19: Ian Gorton , Sean Baker, Enterprise Transaction Processing Systems: Putting the Cobra Ots, Encina++ and Orbixotm to Work, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, 2000
• 20: Brebner, P, Ran, S, Entity Bean A, B, C's: Enterprise Java Beans Commit Options and Caching, in: R. Guerraoui (Ed): Middleware 2001, LNCS 2218, pp 36--55, 2001. 2218, pp 36--55, 2001.
• 21: SPEC: <u>http://www.benchmarkresources.com/handbook/chapter9.pdf</u>
• 22: ECperf: <u>http://www.jinspired.corn/products/idbinsight/ecperfuncovered.pdf</u>
• 23: Shanti Subramanyam, Rarnesh Ramachandran and Akara Sucharitakul, Performance Analysis and Tuning of the Java#8482; 2 Platform, Enterprise Edition (J2EE#8482;), JavaOne, 2001 and 2002, San Francisco.
• 24: Ricardo Morin, Hardeep Reehal, and Kumar Shiv, Java#8482; 2 Platform, Enterprise Edition (J2EE) Performance Evaluation Workloads--Analysis and Lessons Learned, JavaOne 2002, San Francisco
• 25: Yan Zhang , Anna Liu , Wei Qu, Comparing industry benchmarks for J2EE application server: IBM's trade2 vs Sun's ECperf, Proceedings of the twenty-sixth Australasian conference on Computer science: research and practice in information technology, p.199-206, February 01, 2003, Adelaide, Australia
• 26: EJB Benchmarking: <u>http://nenya.ms.mff.cuni.cz/~prochazk/papers/TREJBComp.pdf</u>
• 27: 2002 WebSphere eBU (Trade2, ECperf and other benchmarks): <u>http://www-900.ibm.com/developerWorks/cn/wsdd/download/pdf/ws-ebu/scene4-perfscal-ch.pdf</u>
• 28: Martin Karlsson, Kevin E. Moore, Erik Hagersten, and David A. Wood, Memory Characterization of the ECperf Benchmark, <u>http://www.cs.wisc.edu/multifacetlpapers/WMPI02 ECperfSEQ12.pdf</u>
• 29: Shuping Ran , Paul Brebner , Ian Gorton, The Rigorous Evaluation of Enterprise Java Bean Technology, Proceedings of the The 15th International Conference on Information Networking, p.93, January 31-February 02, 2001
• 30: Ian Gorton , Anna Liu , Paul Brebner, Rigorous Evaluation of COTS Middleware Technology, Computer, v.36 n.3, p.50-55, March 2003 [doi>10.1109/MC.2003.1185217 ]

INDEX TERMS

Keywords:
ECperf benchmark, Enterprise Java Beans (EJB), J2EE, scalability, throughput