Hi,
I have two similar machines running vmserver, and one is exhibiting very high CPU load when all machines are idle. What should I look for to identify the problem ?
Details: I have two servers to host 15 virtual machines, which run daily builds and qa tests for large finite element solver project. Most of the day, both machines sit idle, the builds are scheduled at night where load of both servers is high. During idle time machine A shows about 100+% CPU usage (out of 400%), and the other is at 5% (out of 400%)
Machine A: 2 dual core Opterons 8GB RAM, running OpenSuSE 11.1, 7 guests with different distros of Linux (SLES/RHEL/Fedora, 32/64). Till last week, this machine ran OpenSuse 10.1 with vmserver 1.0.7 and behaved (to get remote access, the vmware service had to be restarted 3-4 times a week). I added new rootdisk, installed new SuSE 11.1 + vmserver 2.0.1, mounted old disks, registered all guests, upgraded guests hardware, installed vm-tools. Previously idle load was 5-10%, after upgrade it jumped to 100-120%, uniformly distributed for all guests. On guest side, there is no load showed.
Machine B: Quad core PHENOM with 8GB RAM (this machine is about 2.5 times faster than the other one), running OpenSuSE 11.1, 7 virtual machines, with different versions of Windows (XP/Vista, 32/64). I was forced to reinstall SuSE because unidentified upgrade prevented recompilation of vmserver. Wiped system partition, reinstalled Linux, and vmserver 2.0.1, mounted old disks, registered all guests, upgraded hardware, installed vm-tools. Idle load is at 5% total (out of 400%).
Both machines are left at the CD distro version ( 2.6.27.7-9-default ) and have nearly identical set of rpm's installed. The hardware is different, but not vastly so, and each has plenty of spare disk space. No RAID, just 3 or 4 disks mounted as separate partitions, so guests are distributed between different drives. During vmserver installation, I had to force gcc version (4.3 vs. 4.3.2) and it did not build vmsock module. In both cases config was left at defaults, except remote port, which is configured at default 902 and A, and changed to 904 on B. I registered the same set of users, the same roles and permissions on both machines. In both cases I had to manually edit authorizations.xml because of the bug in 'NextRoleId' (funny that they fixed NextAceId, but left NextRoleId with the same bug)
To verify the issue I pulled small Linux guest machine from old backups and copied to each machine. Hardware level 4, no vm-tools, Fedora 8 - 64-bit. When started on Machine A it shows 20% idle cpu usage, when started on B it quickly drops to 1-2%. Same test with Win2K machine, same results. I am at loss as to where to look for the source of the problem, but it is clearly in the config for vmserver.
Attached is the snapshot of CPU load history over the last week for both servers to show the dramatic difference between the old and new behavior.
Any suggestions will be welcomed
Tadeusz