Performance tests in ActiveMQ

Wanting to see what kind of performance ActiveMQ has (in preparation for making changes mentioned in the scaling post), we did some performance testing.  All of these tests were run using versions of the jmeter script in another post and ActiveMQ 5.6.0

We've used jmeter's throughput metric, the total number of actions, as the main indicator of performance. Since our threads only put messages on a queue or took messages off a queue, we'll use that as a close guide to the number of messages that ActiveMQ can send through the system.

Using my relatively new Windows7 workstation and KahaDB, we saw ~70k messages/min move - that was putting them on and taking them off the queues.

On one of our live servers (otherwise not in use) again with KahaDB and with an extra queue or two caused memory issues due to too many threads.  While it failed to run entirely, it showed how threading can quickly create issues.
Changing to more loops and less threads and running the test several times saw: ~100k/minute for 80s test using kahadb. (125k samples, dev 50, sample 7 average 25, median 12)

Running several tests on that live system with a SQL Server back data store seemed to run at 32k/minute, but finished at 50k/minute.  (122k samples, dev 100, sample 6, aver 45 median 16, took 150 secs).

So far it looks like a definite performance gain for KahaDB, but further tests showed that SQL Server knew how to optimise for our tests and close that gap significantly.  The servers are all virtual machines running on the same hardware; however, the ActiveMQ/KahaDB server has 1 cpu and a few GBs of memory while the SQL Server has several cpus dedicated to it and 32GB of memory - more than enough to hold all of that data in memory. For running under normal loads, the concern was that SQL Server wouldn't hold our data in memory in live use and would show a clearly slower peformance.

Larger tests with more queues and fewer threads which looped more:
KahaDB:
Write only - throughput on write is directly a function of the number of threads:
3 thread each (multiple runs): 35k/min throughput, 41secs
5 threads each: 50k/min
10 threads each: 92k/min
15 threads each: 132k/min
20 threads each: 145k/min
25 threads each: memory issues, 60k/min then 0k/min

Read only
3 threads: 12k, 3k/min, dec 288, ave 91, median 0, 230 secs
10 threads each: 94k/min
Further tests ran into a limitation with the setup.

SQL Server: with long write only job and 20 threads, SQL Serverl eventually gets close to KahaDB performance despite starting much lower.  Read only achieves ~12k/min which is far less.

Read/Write:
repeated runs of a read/write version have seen maximum values for KahaDB hit 200k/min (34s) and SQL Server around 150k/min.  Readers seem faster on kahadb as they close sooner.
With fresh data (tables), SQL Server runs at an average of ~80k/min (70k/min (68s), 2nd run: 101k/min (47s), 3rd 88k/min (55s). 4th 66k/min)
With fresh data stores, KahaDB was averaging 150k/min (140k/min (34s), 2nd 148k/min, 3rd 156k/min, 4th 169k/min).  Making it definitely the performance winner.

We've glossed over a few details with the tests largely because there are so many other details (time of day, other load, order of execution, etc) that might affect any one or many tests.  In other words, this is an indicator of performance differences, any particular setup will differ.