VMware Server 2.0.1 and latest RHEL5 kernel (both host & guest) = RANDOM REBOOTS !
I have VMware Server 2.0.1 running on CentOS 5.3 (x64 kernel-2.6.18-128.2.1.el5).
Lately I started experiencing random reboots of host and had to investigate it a bit.
At first I suspected hardware problems and specificly RAM corruption. 24h memtest86+ and no errors.
I had my guests migrated to another VMware-server 2.0.1 running on FC11 (2.6.29.6-213.fc11.x86_64).
No more host reboots but the guests (CentOS 5.3 x86 2.6.18-128.2.1.el5) still keep random rebooting.
Had a look at the guest logs and found this:
-
Aug 01 11:08:40.120: Worker#1| Caught signal 6 -- tid 5362
Aug 01 11:08:40.120: Worker#1| SIGNAL: eip 0x369b0332f5 esp 0x7fa1e0b01d88 ebp 0x2
Aug 01 11:08:40.120: Worker#1| SIGNAL: eax 0x0 ebx 0x369b132efe ecx 0xffffffffffffffff edx 0x6 esi 0x14f2 edi 0x14ed
Aug 01 11:08:40.120: Worker#1| r8 0x2c55020 r9 0x3 r10 0x8 r11 0x3206 r12 0xa57c9a000 r13 0x10000 r14 0x284f4d0 r15 0x7fa1e0b01f20
Aug 01 11:08:40.120: Worker#1| SIGNAL: stack 7FA1E0B01D88 : 0x9b034b20 0x00000036 0x00000000 0x00000000
Aug 01 11:08:40.120: Worker#1| SIGNAL: stack 7FA1E0B01D98 : 0x00000000 0x00000000 0x00000000 0x00000000
Aug 01 11:08:40.120: Worker#1| SIGNAL: stack 7FA1E0B01DA8 : 0x00000000 0x00000000 0x00000000 0x00000000
Aug 01 11:08:40.120: Worker#1| SIGNAL: stack 7FA1E0B01DB8 : 0x00003000 0x00000000 0x9b369e80 0x00000036
Aug 01 11:08:40.120: Worker#1| SIGNAL: stack 7FA1E0B01DC8 : 0x00010000 0x00000000 0x9b369e80 0x00000036
Aug 01 11:08:40.120: Worker#1| SIGNAL: stack 7FA1E0B01DD8 : 0x00010010 0x00000000 0xe0b01f20 0x00007fa1
Aug 01 11:08:40.120: Worker#1| SIGNAL: stack 7FA1E0B01DE8 : 0x00000000 0x00000000 0x9b369ee8 0x00000036
Aug 01 11:08:40.120: Worker#1| SIGNAL: stack 7FA1E0B01DF8 : 0x9b077f32 0x00000036 0x00000000 0x00000000
Aug 01 11:08:40.120: Worker#1| Backtrace:
Aug 01 11:08:40.120: Worker#1| Backtrace[0] 00007fa1e0b01830 rip=000000000041521c rbx=0000000000000000 rbp=0000000000415500 r12=0000000000000000 r13=00007fa1e0b01940 r14=0000000000000025 r15=0000000000000006
Aug 01 11:08:40.120: Worker#1| Backtrace[1] 00007fa1e0b01850 rip=0000000000468cf5 rbx=0000000000000080 rbp=0000000000000006 r12=0000000000000000 r13=00007fa1e0b01940 r14=0000000000000025 r15=0000000000000006
Aug 01 11:08:40.120: Worker#1| Backtrace[2] 00007fa1e0b01940 rip=000000369bc0ee90 rbx=000000369b132efe rbp=0000000000000002 r12=0000000a57c9a000 r13=0000000000010000 r14=000000000284f4d0 r15=00007fa1e0b01f20
Aug 01 11:08:40.120: Worker#1| Backtrace[3] 00007fa1e0b01d88 rip=000000369b0332f5 rbx=000000369b132efe rbp=0000000000000002 r12=0000000a57c9a000 r13=0000000000010000 r14=000000000284f4d0 r15=00007fa1e0b01f20
Aug 01 11:08:40.120: Worker#1| Backtrace[4] 00007fa1e0b01d90 rip=000000369b034b20 rbx=000000369b132efe rbp=0000000000000002 r12=0000000a57c9a000 r13=0000000000010000 r14=000000000284f4d0 r15=00007fa1e0b01f20
Aug 01 11:08:40.120: Worker#1| Backtrace[5] 00007fa1e0b01ec0 rip=000000369b075a50 rbx=000000369b132efe rbp=0000000000000002 r12=0000000a57c9a000 r13=0000000000010000 r14=000000000284f4d0 r15=00007fa1e0b01f20
Aug 01 11:08:40.120: Worker#1| Backtrace[6] 00007fa1e0b01f00 rip=000000000047eb93 rbx=0000000000000000 rbp=0000000002c55020 r12=0000000a57c9a000 r13=0000000000010000 r14=000000000284f4d0 r15=00007fa1e0b01f20
Aug 01 11:08:40.120: Worker#1| Backtrace[7] 00007fa1e0b01f70 rip=000000000054cc9b rbx=000000000274ebd0 rbp=000000000274ed50 r12=0000000000000001 r13=000000000000000c r14=0000000000000000 r15=ffffffffffffffff
Aug 01 11:08:40.120: Worker#1| Backtrace[8] 00007fa1e0b01f90 rip=00000000004f165d rbx=000000000274ed80 rbp=000000000274ed50 r12=0000000000000001 r13=000000000000000c r14=0000000000000000 r15=ffffffffffffffff
Aug 01 11:08:40.120: Worker#1| Backtrace[9] 00007fa1e0b01fe0 rip=00000000004f186c rbx=000000000274ed80 rbp=000000000274ed50 r12=0000000000000000 r13=00000000027f4244 r14=0000000000edfca0 r15=0000000000000003
Aug 01 11:08:40.120: Worker#1| Backtrace10 00007fa1e0b02010 rip=000000000048542a rbx=0000000000ee0c08 rbp=0000000000000000 r12=00007fff609f6b50 r13=0000000000000000 r14=0000000000000000 r15=0000000000000003
Aug 01 11:08:40.120: Worker#1| Backtrace11 00007fa1e0b02110 rip=000000369bc0686a rbx=0000000000000000 rbp=0000000000000000 r12=00007fff609f6ad0 r13=0000000000000000 r14=0000000000000000 r15=0000000000000003
Aug 01 11:08:40.120: Worker#1| Backtrace12 00007fa1e0b02220 rip=000000369b0de25d rbx=00007fa1e0b02910 rbp=0000000000000000 r12=00007fff609f6ad0 r13=0000000000000000 r14=0000000000000000 r15=0000000000000003
Aug 01 11:08:40.120: Worker#1| SymBacktrace[0] 00007fa1e0b01830 rip=000000000041521c in function (null) in object /usr/lib/vmware/bin/vmware-vmx loaded at 0000000000400000
Aug 01 11:08:40.120: Worker#1| SymBacktrace[1] 00007fa1e0b01850 rip=0000000000468cf5 in function (null) in object /usr/lib/vmware/bin/vmware-vmx loaded at 0000000000400000
Aug 01 11:08:40.120: Worker#1| SymBacktrace[2] 00007fa1e0b01940 rip=000000369bc0ee90 in function (null) in object /lib64/libpthread.so.0 loaded at 000000369bc00000
Aug 01 11:08:40.121: Worker#1| SymBacktrace[3] 00007fa1e0b01d88 rip=000000369b0332f5 in function gsignal in object /lib64/libc.so.6 loaded at 000000369b000000
Aug 01 11:08:40.121: Worker#1| SymBacktrace[4] 00007fa1e0b01d90 rip=000000369b034b20 in function abort in object /lib64/libc.so.6 loaded at 000000369b000000
Aug 01 11:08:40.121: Worker#1| SymBacktrace[5] 00007fa1e0b01ec0 rip=000000369b075a50 in function (null) in object /lib64/libc.so.6 loaded at 000000369b000000
Aug 01 11:08:40.121: Worker#1| SymBacktrace[6] 00007fa1e0b01f00 rip=000000000047eb93 in function (null) in object /usr/lib/vmware/bin/vmware-vmx loaded at 0000000000400000
Aug 01 11:08:40.121: Worker#1| SymBacktrace[7] 00007fa1e0b01f70 rip=000000000054cc9b in function (null) in object /usr/lib/vmware/bin/vmware-vmx loaded at 0000000000400000
Aug 01 11:08:40.121: Worker#1| SymBacktrace[8] 00007fa1e0b01f90 rip=00000000004f165d in function (null) in object /usr/lib/vmware/bin/vmware-vmx loaded at 0000000000400000
Aug 01 11:08:40.121: Worker#1| SymBacktrace[9] 00007fa1e0b01fe0 rip=00000000004f186c in function (null) in object /usr/lib/vmware/bin/vmware-vmx loaded at 0000000000400000
Aug 01 11:08:40.121: Worker#1| SymBacktrace10 00007fa1e0b02010 rip=000000000048542a in function (null) in object /usr/lib/vmware/bin/vmware-vmx loaded at 0000000000400000
Aug 01 11:08:40.121: Worker#1| SymBacktrace11 00007fa1e0b02110 rip=000000369bc0686a in function (null) in object /lib64/libpthread.so.0 loaded at 000000369bc00000
Aug 01 11:08:40.121: Worker#1| SymBacktrace12 00007fa1e0b02220 rip=000000369b0de25d in function clone in object /lib64/libc.so.6 loaded at 000000369b000000
Aug 01 11:08:40.121: Worker#1| Unexpected signal: 6.
Aug 01 11:08:40.121: Worker#1| Panic: can't get userlevel lock.
Aug 01 11:08:40.121: Worker#1| Core dump limit is 0 KB.
Aug 01 11:08:40.139: Worker#1| Child process 13550 failed to dump core (status 0x6).
Aug 01 11:08:40.139: Worker#1| Backtrace:
Aug 01 11:08:40.139: Worker#1| Backtrace[0] 00007fa1e0b01350 rip=000000000041521c rbx=0000000000000000 rbp=0000000000415500 r12=0000000000000000 r13=00007fa1e0b01940 r14=0000000000000025 r15=0000000000000006
Aug 01 11:08:40.139: Worker#1| Backtrace[1] 00007fa1e0b01370 rip=00000000004db650 rbx=00007fa1e0b01880 rbp=0000000000000006 r12=0000000000000000 r13=00007fa1e0b01940 r14=0000000000000025 r15=0000000000000006
Aug 01 11:08:40.139: Worker#1| Backtrace[2] 00007fa1e0b01850 rip=0000000000468ea8 rbx=00007fa1e0b01880 rbp=0000000000000006 r12=0000000000000000 r13=00007fa1e0b01940 r14=0000000000000025 r15=0000000000000006
Aug 01 11:08:40.139: Worker#1| Backtrace[3] 00007fa1e0b01940 rip=000000369bc0ee90 rbx=000000369b132efe rbp=0000000000000002 r12=0000000a57c9a000 r13=0000000000010000 r14=000000000284f4d0 r15=00007fa1e0b01f20
Aug 01 11:08:40.139: Worker#1| Backtrace[4] 00007fa1e0b01d88 rip=000000369b0332f5 rbx=000000369b132efe rbp=0000000000000002 r12=0000000a57c9a000 r13=0000000000010000 r14=000000000284f4d0 r15=00007fa1e0b01f20
Aug 01 11:08:40.139: Worker#1| Backtrace[5] 00007fa1e0b01d90 rip=000000369b034b20 rbx=000000369b132efe rbp=0000000000000002 r12=0000000a57c9a000 r13=0000000000010000 r14=000000000284f4d0 r15=00007fa1e0b01f20
Aug 01 11:08:40.139: Worker#1| Backtrace[6] 00007fa1e0b01ec0 rip=000000369b075a50 rbx=000000369b132efe rbp=0000000000000002 r12=0000000a57c9a000 r13=0000000000010000 r14=000000000284f4d0 r15=00007fa1e0b01f20
Aug 01 11:08:40.139: Worker#1| Backtrace[7] 00007fa1e0b01f00 rip=000000000047eb93 rbx=0000000000000000 rbp=0000000002c55020 r12=0000000a57c9a000 r13=0000000000010000 r14=000000000284f4d0 r15=00007fa1e0b01f20
Aug 01 11:08:40.139: Worker#1| Backtrace[8] 00007fa1e0b01f70 rip=000000000054cc9b rbx=000000000274ebd0 rbp=000000000274ed50 r12=0000000000000001 r13=000000000000000c r14=0000000000000000 r15=ffffffffffffffff
Aug 01 11:08:40.139: Worker#1| Backtrace[9] 00007fa1e0b01f90 rip=00000000004f165d rbx=000000000274ed80 rbp=000000000274ed50 r12=0000000000000001 r13=000000000000000c r14=0000000000000000 r15=ffffffffffffffff
Aug 01 11:08:40.139: Worker#1| Backtrace10 00007fa1e0b01fe0 rip=00000000004f186c rbx=000000000274ed80 rbp=000000000274ed50 r12=0000000000000000 r13=00000000027f4244 r14=0000000000edfca0 r15=0000000000000003
Aug 01 11:08:40.139: Worker#1| Backtrace11 00007fa1e0b02010 rip=000000000048542a rbx=0000000000ee0c08 rbp=0000000000000000 r12=00007fff609f6b50 r13=0000000000000000 r14=0000000000000000 r15=0000000000000003
Aug 01 11:08:40.139: Worker#1| Backtrace12 00007fa1e0b02110 rip=000000369bc0686a rbx=0000000000000000 rbp=0000000000000000 r12=00007fff609f6ad0 r13=0000000000000000 r14=0000000000000000 r15=0000000000000003
Aug 01 11:08:40.139: Worker#1| Backtrace13 00007fa1e0b02220 rip=000000369b0de25d rbx=00007fa1e0b02910 rbp=0000000000000000 r12=00007fff609f6ad0 r13=0000000000000000 r14=0000000000000000 r15=0000000000000003
Aug 01 11:08:40.139: Worker#1| SymBacktrace[0] 00007fa1e0b01350 rip=000000000041521c in function (null) in object /usr/lib/vmware/bin/vmware-vmx loaded at 0000000000400000
Aug 01 11:08:40.139: Worker#1| SymBacktrace[1] 00007fa1e0b01370 rip=00000000004db650 in function (null) in object /usr/lib/vmware/bin/vmware-vmx loaded at 0000000000400000
Aug 01 11:08:40.139: Worker#1| SymBacktrace[2] 00007fa1e0b01850 rip=0000000000468ea8 in function (null) in object /usr/lib/vmware/bin/vmware-vmx loaded at 0000000000400000
Aug 01 11:08:40.139: Worker#1| SymBacktrace[3] 00007fa1e0b01940 rip=000000369bc0ee90 in function (null) in object /lib64/libpthread.so.0 loaded at 000000369bc00000
Aug 01 11:08:40.139: Worker#1| SymBacktrace[4] 00007fa1e0b01d88 rip=000000369b0332f5 in function gsignal in object /lib64/libc.so.6 loaded at 000000369b000000
Aug 01 11:08:40.139: Worker#1| SymBacktrace[5] 00007fa1e0b01d90 rip=000000369b034b20 in function abort in object /lib64/libc.so.6 loaded at 000000369b000000
Aug 01 11:08:40.139: Worker#1| SymBacktrace[6] 00007fa1e0b01ec0 rip=000000369b075a50 in function (null) in object /lib64/libc.so.6 loaded at 000000369b000000
Aug 01 11:08:40.139: Worker#1| SymBacktrace[7] 00007fa1e0b01f00 rip=000000000047eb93 in function (null) in object /usr/lib/vmware/bin/vmware-vmx loaded at 0000000000400000
Aug 01 11:08:40.139: Worker#1| SymBacktrace[8] 00007fa1e0b01f70 rip=000000000054cc9b in function (null) in object /usr/lib/vmware/bin/vmware-vmx loaded at 0000000000400000
Aug 01 11:08:40.139: Worker#1| SymBacktrace[9] 00007fa1e0b01f90 rip=00000000004f165d in function (null) in object /usr/lib/vmware/bin/vmware-vmx loaded at 0000000000400000
Aug 01 11:08:40.139: Worker#1| SymBacktrace10 00007fa1e0b01fe0 rip=00000000004f186c in function (null) in object /usr/lib/vmware/bin/vmware-vmx loaded at 0000000000400000
Aug 01 11:08:40.139: Worker#1| SymBacktrace11 00007fa1e0b02010 rip=000000000048542a in function (null) in object /usr/lib/vmware/bin/vmware-vmx loaded at 0000000000400000
Aug 01 11:08:40.139: Worker#1| SymBacktrace12 00007fa1e0b02110 rip=000000369bc0686a in function (null) in object /lib64/libpthread.so.0 loaded at 000000369bc00000
Aug 01 11:08:40.139: Worker#1| SymBacktrace13 00007fa1e0b02220 rip=000000369b0de25d in function clone in object /lib64/libc.so.6 loaded at 000000369b000000
Aug 01 11:08:40.139: Worker#1| Msg_Post: Error
Aug 01 11:08:40.139: Worker#1| http://msg.log.error.unrecoverable VMware Server unrecoverable error: (Worker#1)
Aug 01 11:08:40.139: Worker#1| Unexpected signal: 6.
Aug 01 11:08:40.139: Worker#1| http://msg.panic.haveLog A log file is available in "/storage/VirtualMachines/srv01.ctg.ee/vmware.log". http://msg.panic.requestSupport.withLog Please request support and include the contents of the log file. http://msg.panic.requestSupport.vmSupport.windowsOrLinux
Aug 01 11:08:40.139: Worker#1| To collect data to submit to VMware support, select Help > About and click "Collect Support Data". You can also run the "vm-support" script in the Workstation folder directly.
Aug 01 11:08:40.139: Worker#1| http://msg.panic.response We will respond on the basis of your support entitlement.
Aug 01 11:08:40.221: vmx| VTHREAD watched thread 37 "Worker#1" died
Aug 01 11:08:40.222: vcpu-0| VTHREAD watched thread 0 "vmx" died
Aug 01 11:08:40.320: Worker#13| VTHREAD watched thread 4 "vcpu-0" died
Aug 01 11:08:40.320: Worker#8| VTHREAD watched thread 4 "vcpu-0" died
Aug 01 11:08:40.320: Worker#16| VTHREAD watched thread 4 "vcpu-0" died
Aug 01 11:08:40.321: Worker#3| VTHREAD watched thread 0 "vmx" died
Aug 01 11:08:40.321: Worker#0| VTHREAD watched thread 0 "vmx" died
Aug 01 11:08:40.321: Worker#9| VTHREAD watched thread 0 "vmx" died
Aug 01 11:08:40.321: Worker#2| VTHREAD watched thread 4 "vcpu-0" died
Aug 01 11:08:40.322: Worker#7| VTHREAD watched thread 4 "vcpu-0" died
Aug 01 11:08:40.322: Worker#12| VTHREAD watched thread 4 "vcpu-0" died
Aug 01 11:08:40.322: Worker#15| VTHREAD watched thread 4 "vcpu-0" died
Aug 01 11:08:40.322: Worker#10| VTHREAD watched thread 0 "vmx" died
Aug 01 11:08:40.322: Worker#5| VTHREAD watched thread 4 "vcpu-0" died
Aug 01 11:08:40.322: Worker#14| VTHREAD watched thread 4 "vcpu-0" died
Aug 01 11:08:40.323: Worker#4| VTHREAD watched thread 0 "vmx" died
Aug 01 11:08:40.323: Worker#11| VTHREAD watched thread 4 "vcpu-0" died
Aug 01 11:08:40.324: Worker#6| VTHREAD watched thread 4 "vcpu-0" died
Aug 01 11:08:40.554: mks| VTHREAD watched thread 0 "vmx" died
-
I've looked around and that seems to be quite a common problem here and there?