CDx: a family of real-time Java benchmarks

ABSTRACT

Java is becoming a viable platform for hard real-time computing. There are production and research real-time Java VMs, as well as applications in both military and civil sector. Technological advances and increased adoption of Real-time Java contrast significantly with the lack of real-time benchmarks. The few benchmarks that exist are either low-level synthetic micro-benchmarks, or benchmarks used internally by companies, making it difficult to independently verify and repeat reported results.

This paper presents the _x (Collision Detector) benchmark suite, an open source application benchmark suite that targets different hard and soft real-time virtual machines. _x is, at its core, a real-time benchmark with a single periodic task, which implements aircraft collision detection based on simulated radar frames. The benchmark can be configured to use different sets of real-time features and comes with a number of workloads. We describe the architecture of the benchmark and characterize the workload based on input parameters.

REFERENCES

• 1: AICAS. The Jamaica virtual machine. http://www.aicas.com.
• 2: Chris Andreae, Yvonne Coady, Celina Gibbs, James Noble, Jan Vitek, and Tian Zhao. Scoped types and aspects for real-time Java memory management. Realtime Systems Journal, 37(1):1--44, October 2007.
• 3: AONIX. The PERC virtual machine. http://www.aonix.com.
• 4: Austin Armbruster, Jason Baker, Antonio Cunei, David Holmes, Chapman Flack, Filip Pizlo, Edward Pla, Marek Prochazka, and Jan Vitek. A Real-time Java virtual machine with applications in avionics. ACM Transactions in Embedded Computing Systems (TECS), 7(1):1--49, 2007.
• 5: Joshua Auerbach, David F. Bacon, Florian Bömers, and Perry Cheng. Real-time music synthesis in Java using the Metronome garbage collector. In Proceedings of the International Computer Music Conference, Copenhagen, Denmark, August 2007.
• 6: Joshua Auerbach, David F. Bacon, Perry Cheng, David Grove, Ben Biron, Charlie Gracie, Bill McCloskey, Aleksandar Micic, and Ryan Sciampacone. Tax-and-spend: democratic scheduling for real-time garbage collection. In Proceedings of the ACM International Conference on Embedded Software (EMSOFT), pages 245--254, October 2008.
• 7: Joshua S. Auerbach, David F. Bacon, Rachid Guerraoui, Jesper Honig Spring, and Jan Vitek. Flexible task graphs: a unified restricted thread programming model for Java. pages 1--11. ACM, June 2008.
• 8: Jason Baker, Antonio Cunei, Filip Pizlo, and Jan Vitek. Accurate garbage collection in uncooperative environments. Concurrency and Computation: Practice and Experience, 2009.
• 9: S. M. Blackburn, R. Garner, C. Hoffman, A. M. Khan, K. S. McKinley, R. Bentzur, A. Diwan, D. Feinberg, D. Frampton, S. Z. Guyer, M. Hirzel, A. Hosking, M. Jump, H. Lee, J. E. B. Moss, A. Phansalkar, D. Stefanović, T. VanDrunen, D. von Dincklage, and B. Wiedermann. The DaCapo benchmarks: Java benchmarking development and analysis. In Proceedings of the ACM SIGPLAN Conference on Object-Oriented Programing, Systems, Languages, and Applications (OOPSLA), volume 41, pages 169--190, October 2006.
• 10: Greg Bollella, Tim Canham, Vanessa Carson, Virgil Champlin, Daniel Dvorak, Brian Giovannoni, Mark Indictor, Kenny Meyer, Alex Murray, and Kirk Reinholtz. Programming with non-heap memory in the real-time specification for Java. In Companion of the 18th annual ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications, pages 361--369, 2003.
• 11: J. E. Bresenham. Algorithm for computer control of a digital plotter. pages 1--6, 1998.
• 12: Alan Burns, Ken Tindell, and Andy Wellings. Effective analysis for engineering real-time fixed priority schedulers. IEEE Transactions on Software Engineering, 21(5):475--480, 1995.
• 13: S. R. Chidamber and C. F. Kemerer. A metrics suite for object oriented design. IEEE Transactions on Software Engineering, 20(6):476--493, June 1994.
• 14: Angelo Corsaro and Doug Schmidt. Evaluating Real-Time Java features and performance for real-time embedded systems. In The 8th IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS), 2002.
• 15: Brian P. Doherty. A real-time benchmark for Java#8482; In JTRES '07: Proceedings of the 5th international workshop on Java technologies for real-time and embedded systems, pages 35--46, New York, NY, USA, 2007. ACM.
• 16: Colin J. Fidge. Real-time schedulability tests for preemptive multitasking. Real-Time Systems, 14(1):61--93, January 1998.
• 17: Sven Gestegard Robertz, Roger Henriksson, Klas Nilsson, Anders Blomdell, and Ivan Tarasov. Using real-time Java for industrial robot control. In Proceedings of the 5th international workshop on Java technologies for real-time and embedded systems (JTRES), pages 104--110, 2007.
• 18: Roger Henriksson. Scheduling Garbage Collection in Embedded Systems. PhD thesis, Lund University, July 1998.
• 19: IBM. WebSphere Real Time. http://www.ibm.com/software/webservers/realtime.
• 20: IBM. DDG1000 Next generation navy destroyers. http://www.ibm.com/press/us/en/pressrelease/21033.wss, 2007.
• 21: Raj Jain. The Art of Computer Systems Performance Analysis. John Wiley & Sons, 1991.
• 22: Java Grande Forum. Java Grande Forum benchmark suite. http://www2.epcc.ed.ac.uk/computing/research-activities/java_grande/index_1.html, 2001.
• 23: Jet Propulsion Laboratories, Golden Gate Project. Suramadu benchmarking framework. http://www.opengroup.org/projects/suramadu/, 2006.
• 24: Nicolas Juillerat, Stefan Müller Arisona, and Simon Schubiger-Banz. Real-time, low latency audio processing in Java. In Proceedings of the International Computer Music Conference, Copenhagen, Denmark, August 2007.
• 25: David J. Lilja. Measuring Computer Performance: A Practitioner's Guide. Cambridge University Press, 2000.
• 26: NIST. SciMark 2.0 benchmarks. http://math.nist.gov/scimark2, 2000.
• 27: Filip Pizlo, Jason Fox, David Holmes, and Jan Vitek. Real-time Java scoped memory: design patterns and semantics. In Proceedings of the IEEE International Symposium on Object-oriented Real-Time Distributed Computing (ISORC), Vienna, Austria, May 2004.
• 28: Purdue. The Ovm virtual machine, www.ovmj.org.
• 29: David C. Sharp, Edward Pla, Kenn R. Luecke, and Ricardo J. Hassan II. Evaluating real-time java for mission-critical large-scale embedded systems. In RTAS '03: Proceedings of the The 9th IEEE Real-Time and Embedded Technology and Applications Symposium, page 30, Washington, DC, USA, 2003. IEEE Computer Society.
• 30: SPEC. SPECjvm98 benchmarks, 1998.
• 31: SPEC. SPECjbb2000 benchmarks, 2005. http://www.spec.org/jbb2005.
• 32: SPEC. SPECjvm2008 benchmarks, 2008. http://www.spec.org/jvm2008.
• 33: D. D. Spinellis. ckjm Chidamber and Kemerer metrics Software, v 1.6. Technical report, Athens University of Economics and Business, 2005.
• 34: Sun Microsystems. Sun java real-time system. http://java.sun.com/javase/technologies/realtime, 2008.
• 35: Tian Zhao, James Noble, and Jan Vitek. Scoped types for real-time Java. In Proceedings of the 25th IEEE International Real-Time Systems Symposium (RTSS), Lisbon, Portugal, December 2004.