After cloning the LIVE server to TEST, the users are finding the TEST instance performance a lot slower

After cloning LIVE to TEST servers we find that the users on the new TEST servers are struggling for performance; seeing spinning wheels and general slow responses. The LIVE and TEST servers are all virtual machines running in the same virtual infrastructure.

1.  We first identified by running "Performance Index" (AIOBENCH) the TEST servers were far worse than LIVE. Investigations with the IT provider confirmed the TEST servers were running on lower spec physical hardware.  After moving the TEST servers to the production infrastructure we got similar results from AIOBENCH and better overall performance for the users on TEST servers, but was still significantly slower.

2.  We then took one of the worse performing processes and ran Engine Trace (openlog) for both LIVE and TEST servers.   On the TEST trace we see 90% time in SYSTEME2 and GET_OS calls, but there are no such calls in the LIVE server.  We identified that V1 software was enabled in TEST but not LIVE.  On TEST server we disabled X3EDM activity code and ran the dictionary validation for this activity code and found this resolved the problem

1.  Always run "Performance Index" (AIOBENCH) to confirm the underlying system performance is the same for the two systems being compared.  NOTE: This only compares the Runtime and Database components.

2.  Confirm the X3 setup is the same on both servers under comparison.  (Also the data volumes, but in this case it was a recent clone)     One way to do this is to take one of the worse performing functions that can be easily run multiple times and compare an engine trace (which should both follow the same code path and have similar Timing Traces as well)