Oracle load testing - part 3 Hammerora results

I learned something very important when doing testing with Hammerora. The documentation is quite good and has a simple, but important point, the importance of planning and preparation. To quote the documentation for Oracle OLTP testing:

Planning and Preparation is one of the most crucial stages of successful testing but is often overlooked. Firstly you should fully document the configuration of your entire load testing environment including details such as hardware, operating system versions and settings and Oracle version and parameters. Once you have fully documented your configuration you should ensure that the configuration is not changed for an entire series of measured tests. This takes discipline but is an essential component of conducting accurate and measured tests.

From this I conclude that many tests I've seen and done myself have not been accurate. As stated earlier the goal for this testing was just to compare performance before and after migration from EVA to 3PAR.

Since this customer did not have the required license to run AWR I did the following change in the driver scipt to create a Statspack snapshot in place of a AWR snapshot. Search for string containing dbms_workload, replace

set sql1 "BEGIN dbms_workload_repository.create_snapshot(); END;"

with

set sql1 "BEGIN perfstat.statspack.snap; END;"

When testing with Hammerora I decided to run each test three times to see if the numbers where consistent. I recorded the numbers of each run in an spread sheet as shown in the following table for the tests on EVA:

Vusers	Run	Report	tpm	nopm	Avg_tpm
1	1	1_2	7158	2369	7634
1	2	11_12	7874	2645
1	3	21_22	7868	2804
3	1	31_32	16478	5765	17317
3	2	41_42	17678	6256
3	3	43_44	17794	6130
10	1	45_46	27847	9959	33225
10	2	51_61	32581	11600
10	3	71_81	39248	13701
20	1	91_101	47489	17441	47075
20	2	111_121	63062	22658
20	3	131_141	30674	11116
30	1	151_161	54349	19756	44186
30	2	171_181	45628	17331
30	3	191_201	32581	12733

Vusers is the number of virtual users in Hammerora, Run is 1 - 3 for each new setting of Vusers. Report  refers to the Statspack report created on snapshots before and after. Tpm and nopm as reported from Hammerora and finally Avg_tpm is the average in each group. Compare this to the numbers for the 3PAR:

Vusers	Run	Report	tpm	nopm	Avg_tpm
1	1	9_10	8246	2815	8262
1	2	11_12	7983	2717
1	3	13_14	8556	2956
3	1	15_16	22652	7854	22881
3	2	17_18	22652	7831
3	3	19_20	23339	7994
10	1	21_22	33539	11767	33191
10	2	25_26	39054	13729
10	3	27_28	26981	9428
20	1	29_30	47134	16462	47356
20	2	31_32	46436	16330
20	3	33_34	48497	17023
30	1	35_36	53197	18902	50788
30	2	37_38	44980	15994
30	3	39_40	54187	19033

The repeated tests for the same number of virtual users do not vary as much on the 3PAR compared to the EVA. Also the numbers for the EVA seemed to improve for each run, maybe due to some caching taking place.

The 3PAR seemed to be more reliable for the same number of virtual users as can be seen in these screen captures, the first for 20 virtual users on EVA:

The peformance on the 3PAR does not change much during the test (20 virtual users):

You'll see that in one instant the EVA seems to perform better, but I rather have stable and less erratic performance with the 3PAR than a system with occasional good performance.

All in all it was very easy to play around with Hammerora, it is very easy to set up so you can spend time on planning and executing the tests. Also I like how you can observe change of performance over time. Clearly Hammerora is a tool I will use more later.