What makes the differences: benchmarking XML database implementations

ABSTRACT

XML is emerging as a major standard for representing data on the World Wide Web. Recently, many XML storage models have been proposed to manage XML data. In order to assess an XML database's abilities to deal with XML queries, several benchmarks have also been proposed, including XMark and XMach. However, no reported studies using those benchmarks were found that can provide users with insights on the impacts of a variety of storage models on XML query performance. In this article, we report our first set of results on benchmarking a set of XML database implementations using two XML benchmarks. The selected implementations represent a wide range of approaches, including RDBMS-based systems with document-independent and document-dependent XML-relational schema mapping approaches, and XML native engines based on an Object-Oriented Model and the Document Object Model. Comprehensive experiments were conducted to study relative performance of different approaches and the important issues that affect XML query performance, such as path expression query processing, effectiveness of various partitioning, label-path, and indexing structures.

REFERENCES

• 1: Abiteboul, S., Quass, D., McHugh, J., Widom, J., and Wiener, J. L. 1997. The Lorel query language for semistructured data. International Journal on Digital Libraries 1, 1, 68--88.
• 2: Timo Böhme , Erhard Rahm, XMach-1: A Benchmark for XML Data Management, Datenbanksysteme in Büro, Technik und Wissenschaft (BTW), 9. GI-Fachtagung,, p.264-273, March 07-09, 2001
• 3: Berglund, A., Boag, S., Chamberlin, D., Fernandez, M. F., Kay, M., Robie, J., and Simeon, J. 2002. XML path language (XPath) 2.0. Tech. rep.
• 4: Boag, S., Chamberlin, D., Fernandez, M. F., Florescu, D., Robie, J., and Simeon, J. 2002. XQuery 1.0: An XML query language. In W3C Working Draft 16 August 2002.
• 5: Timo Böhme , Erhard Rahm, Multi-user Evaluation of XML Data Management Systems with XMach-1, Proceedings of the VLDB 2002 Workshop EEXTT and CAiSE 2002 Workshop DTWeb on Efficiency and Effectiveness of XML Tools and Techniques and Data Integration over the Web-Revised Papers, p.148-158, January 2003
• 6: Angela Bonifati , Stefano Ceri, Comparative analysis of five XML query languages, ACM SIGMOD Record, v.29 n.1, p.68-79, March 2000 [doi>10.1145/344788.344822]
• 7: Stéphane Bressan , Mong-Li Lee , Ying Guang Li , Zoé Lacroix , Ullas Nambiar, The XOO7 Benchmark, Proceedings of the VLDB 2002 Workshop EEXTT and CAiSE 2002 Workshop DTWeb on Efficiency and Effectiveness of XML Tools and Techniques and Data Integration over the Web-Revised Papers, p.146-147, January 2003
• 8: Stefano Ceri , Sara Comai , Ernesto Damiani , Piero Fraternali , Stefano Paraboschi , Letizia Tanca, XML-GL: a graphical language for querying and restructuring XML documents, Proceeding of the eighth international conference on World Wide Web, p.1171-1187, May 1999, Toronto, Canada
• 9: Donald D. Chamberlin , Jonathan Robie , Daniela Florescu, Quilt: An XML Query Language for Heterogeneous Data Sources, Selected papers from the Third International Workshop WebDB 2000 on The World Wide Web and Databases, p.1-25, May 18-19, 2000
• 10: Chien, S. Y., Vagena, Z., Zhang, D., Tsotras, V., and Zaniolo, C. 2002. Efficient structural joins on indexed XML documents. In Proceedings of the 28th International Conference on Very Large Data Bases. Hong Kong, China. 263--274.
• 11: V. Christophides , S. Abiteboul , S. Cluet , M. Scholl, From structured documents to novel query facilities, Proceedings of the 1994 ACM SIGMOD international conference on Management of data, p.313-324, May 24-27, 1994, Minneapolis, Minnesota, United States
• 12: Alin Deutsch , Mary Fernandez , Daniela Florescu , Alon Levy , Dan Suciu, A query language for XML, Proceeding of the eighth international conference on World Wide Web, p.1155-1169, May 1999, Toronto, Canada
• 13: Alin Deutsch , Mary Fernandez , Dan Suciu, Storing semistructured data with STORED, Proceedings of the 1999 ACM SIGMOD international conference on Management of data, p.431-442, May 31-June 03, 1999, Philadelphia, Pennsylvania, United States
• 14: Mary Fernández , Daniela Florescu , Jaewoo Kang , Alon Levy , Dan Suciu, Catching the boat with Strudel: experiences with a Web-site management system, Proceedings of the 1998 ACM SIGMOD international conference on Management of data, p.414-425, June 01-04, 1998, Seattle, Washington, United States
• 15: Florescu, D. and Kossmann, D. 1999. A performance evaluation of alternative mapping schemes for storing XML data in a relational database. Survey report.
• 16: Torsten Grust, Accelerating XPath location steps, Proceedings of the 2002 ACM SIGMOD international conference on Management of data, June 03-06, 2002, Madison, Wisconsin [doi>10.1145/564691.564705]
• 17: Haifeng Jiang , Hongjun Lu , Wei Wang , Jeffrey Xu Yu, Path materialization revisited: an efficient storage model for XML data, Proceedings of the thirteenth Australasian database conference, p.85-94, January 01, 2002, Melbourne, Victoria, Australia
• 18: XParent: An Efficient RDBMS-Based XML Database System, Proceedings of the 18th International Conference on Data Engineering, p.335, February 26-March 01, 2002
• 19: Kanda Runapongsa , Jignesh M. Patel , H. V. Jagadish , Shurug Al-Khalifa, The Michigan Benchmark: A Microbenchmark for XML Query Processing Systems, Proceedings of the VLDB 2002 Workshop EEXTT and CAiSE 2002 Workshop DTWeb on Efficiency and Effectiveness of XML Tools and Techniques and Data Integration over the Web-Revised Papers, p.160-161, January 2003
• 20: Efficient Storage of XML Data, Proceedings of the 16th International Conference on Data Engineering, p.198, February 28-March 03, 2000
• 21: Dao Dinh Kha , Masatoshi Yoshikawa , Shunsuke Uemura, An XML Indexing Structure with Relative Region Coordinate, Proceedings of the 17th International Conference on Data Engineering, p.313-320, April 02-06, 2001
• 22: Dongwon Lee , Wesley W. Chu, Comparative analysis of six XML schema languages, ACM SIGMOD Record, v.29 n.3, p.76-87, Sept. 2000 [doi>10.1145/362084.362140]
• 23: Quanzhong Li , Bongki Moon, Indexing and Querying XML Data for Regular Path Expressions, Proceedings of the 27th International Conference on Very Large Data Bases, p.361-370, September 11-14, 2001
• 24: Hongjun Lu , Guoren Wang , Ge Yu , Yubin Bao , Jianhua Lv , Yaxin Yu, XBase: making your gigabyte disk queriable, Proceedings of the 2002 ACM SIGMOD international conference on Management of data, June 03-06, 2002, Madison, Wisconsin [doi>10.1145/564691.564785]
• 25: Jianhua Lv , Guoren Wang , Jeffrey Xu Yu , Ge Yu , Hongjun Lu , Bing Sun, Performance Evaluation of a DOM-Based XML Database: Storage, Indexing and Query Optimization, Proceedings of the Third International Conference on Advances in Web-Age Information Management, p.13-24, August 11-13, 2002
• 26: Jason McHugh , Serge Abiteboul , Roy Goldman , Dallas Quass , Jennifer Widom, Lore: a database management system for semistructured data, ACM SIGMOD Record, v.26 n.3, p.54-66, Sept. 1997 [doi>10.1145/262762.262770]
• 27: Albrecht Schmidt , Martin L. Kersten , Menzo Windhouwer , Florian Waas, Efficient Relational Storage and Retrieval of XML Documents, Selected papers from the Third International Workshop WebDB 2000 on The World Wide Web and Databases, p.137-150, May 18-19, 2000
• 28: Schmidt, A., Waas, F., Kersten, M., Carey, M. J., Manolescu, I., and Busse, R. 2002. Xmark: A benchmark for XML data management. In Proceedings of the 28th International Conference on Very Large Data Bases. (Hong Kong, China). 974--985.
• 29: A. R. Schmidt , Florian Waas , Martin L. Kersten , D. Florescu , I. Manolescu , M. J. Carey , R. Busse, The XML benchmark project, CWI (Centre for Mathematics and Computer Science), Amsterdam, The Netherlands, 2001
• 30: Jayavel Shanmugasundaram , Kristin Tufte , Chun Zhang , Gang He , David J. DeWitt , Jeffrey F. Naughton, Relational Databases for Querying XML Documents: Limitations and Opportunities, Proceedings of the 25th International Conference on Very Large Data Bases, p.302-314, September 07-10, 1999
• 31: Structural Joins: A Primitive for Efficient XML Query Pattern Matching, Proceedings of the 18th International Conference on Data Engineering, p.141, February 26-March 01, 2002
• 32: Tian, F., DeWitt, D. J., Chen, J., and Zhang, C. 2000. The design and performance evaluation of altervative XML storage strategies. Tech. rep., Computer Science Department, University of Wisconsin, Madison, WI.
• 33: W3C. Document object model (DOM). http://www.w3.org/DOM/.
• 34: Benjamin B. Yao , M. Tamer Özsu , John Keenleyside, XBench - A Family of Benchmarks for XML DBMSs, Proceedings of the VLDB 2002 Workshop EEXTT and CAiSE 2002 Workshop DTWeb on Efficiency and Effectiveness of XML Tools and Techniques and Data Integration over the Web-Revised Papers, p.162-164, January 2003
• 35: XRel: a path-based approach to storage and retrieval of XML documents using relational databases, ACM Transactions on Internet Technology (TOIT), v.1 n.1, p.110-141, Aug. 2001 [doi>10.1145/383034.383038]
• 36: Chun Zhang , Jeffrey Naughton , David DeWitt , Qiong Luo , Guy Lohman, On supporting containment queries in relational database management systems, Proceedings of the 2001 ACM SIGMOD international conference on Management of data, p.425-436, May 21-24, 2001, Santa Barbara, California, United States
• 37: Aoying Zhou , Hongjun Lu , Shihui Zheng , Yuqi Liang , Long Zhang , Wenyun Ji , Zengping Tian, VXMLR: A Visual XML-Relational Database System, Proceedings of the 27th International Conference on Very Large Data Bases, p.719-720, September 11-14, 2001

INDEX TERMS

Primary Classification:
  H. Information Systems
  H.3 INFORMATION STORAGE AND RETRIEVAL
      H.3.4 Systems and Software
          Subjects: Performance evaluation (efficiency and effectiveness)

General Terms:
Management, Measurement, Performance

Keywords:
XML query processing, XML storage model, benchmark