« XD first fix pack ships (6.0.1.1) | Main | Redhat acquires JBoss, whos next »

April 11, 2006

Importance of warm up when benchmarking

It's absolutely vital that when running benchmarks in Java, that time be given to allow the JIT to work.

The operations being measured should be run in a loop continously while outputing the current speed. You'll notice pretty horrible numbers are the beginning. Here is an example of an ObjectGrid test I did. These are timing for a loop running on an ObjectGrid client doing a fixed number of operations against an ObjectGrid server.

Finished. Elapsed time = 56201.
Inserting Objects.
Finished. Elapsed time = 4637.
Inserting Objects.
Finished. Elapsed time = 3976.
Inserting Objects.
Finished. Elapsed time = 3796.
Inserting Objects.
Finished. Elapsed time = 4106.
Inserting Objects.
Finished. Elapsed time = 3815.
Inserting Objects.
Finished. Elapsed time = 3815.
Inserting Objects.
Finished. Elapsed time = 3936.
Inserting Objects.
Finished. Elapsed time = 3735.
Inserting Objects.
Finished. Elapsed time = 4005.

You can see the first run taking 56 seconds, the second takes 4.6 seconds and then we settle around 3.8 seconds. This is because of the JIT. It takes a while to start optimizing and it's important to do enough loops for performance to settle.

Anyway, let it warm up first and loop until performance settles. Don't take the initial numbers as they are way off.

April 11, 2006 | Permalink

Comments

Hi Billy,

You should also consider the impact of random GC cycles during the test runs. This could explain why the test results have some deviation - assuming everything else is constant.

Quick solution for better performance testing
1. Disable GC
2. Use an APM solution that adjusts for GC
3. Take the average

You might find this article interesting:
http://www.jinspired.com/products/jxinsight/callstackbenchmark.html

There are many other performance related articles available here:
http://www.jinspired.com/products/jxinsight/insights.html

Kind regards,

William Louth
JXInsight Product Architect
CTO, JInspired

Posted by: William Louth | Apr 11, 2006 5:53:56 PM

Post a comment