Published: Sep 05 2009 / 09:30
A fine good example of how tests should not be done.
Could you give some comments on why this is so bad?
For starters, how often do you see Java programs in which Strings are built up from individual chars? Its measuring something that is pretty much never done in Java programs.
And second, considering you are concerned about memory usage (which is going to vary tremendously due to the unpredictability of the gc), 5 trials way too small.
Third, I don't think Runtime.totalMemory() returns what you think it returns.
True that the individual chars are probably not entirely real world.
I don agree that the memory usage is unpredictable especially as you don't know when/if the garbage collector will run. However, the results were all similar enough to make me believe the tendencies they show.
Well, if concatenation is being tested then loops should contain just that, concatenations. I'm seeing much more inside them. If done correctly I'm sure the results would be different and so would be your conclusions.
What more? You see a no-op function call which is included in the hope that his forces the string to be used. Admittedly the optimizer may be too smart for this too, but you have to stop somewhere.
Doesn't Java 6 use StringBuilder under the covers for String concatenation anyway?
Hell Java 5 does, and previous versions of Java used StringBuffers. Well, it usually does, in some cases it will optimize it so that String concatenation is not even needed. For instance String myString = "my" + " " + "string" will become String myString = "my string".
That is what I thought as well.
I expect the results to be changed significantly when the StringBuilder can be reused. However, that would have made the comparison somewhat unfair I think.
With the assumption that the StringBuilder is indeed used, that only increases my surprise about the results.
sorry, yet another suspicious microbenchmark with an unsubstantiated conclusion. Microbenchmarking is hard, very hard, they can't be slapped together in a few minutes. It requires deep analysis of how the code is being treated by each of the layers in the system. There has to be an analysis of the code to ensure that what is being measured is really the effect you believe is being measured or is there another over-riding effect that is introducing random noise into the result. Is this benchmark even answering the question asked of it? I wouldn't comment until I saw that some steps had been taken to reconcile the results with the code.
Hence the lack of "conclusions" in the article.
I make no claims about knowing about benchmarking. I did a simple test as I wanted to "prove" some things I always assumed. Like the "string additions are bad", "StringBuilder is better than StringBuffer", and javolution's claim about being fast and using less memory.
Seems the little test at least proved that these claims at least are not always true.
I think the OP is saying, basically, that micro-benchmarks are *very* difficult to do right, if it's even possible. If you don't "know about benchmarking" it's even harder, so you haven't really proven anything. Plus the JVM version makes a *huge* difference--go back and "benchmark" on 1.3 and I'll bet javolution compares better.
Without picking apart the bytecode I'm not really sure we know what's going on.
Html tags not supported. Reply is editable for 5 minutes. Use [code lang="java|ruby|sql|css|xml"][/code] to post code snippets.