Warranty Void if....

Our VMware vSphere 5.1 server died over the weekend. Basically the vmware-vpxd service wouldn't initialize. This prompted a call to VMware support. VMware support was on the phone with us for two days, at which point they issued the dreaded Prima Nocte of support. "Oh Sorry, it seems you are not running a supported config, we can not go any further with this case, do you agree with this statement?"

We aren't running a supported config? Ouch. The VMware tech pointed us to an article outlining the maximum hosts and guests for the embedded database on vSphere 5.1. Unfortunately for us, we are slightly over the allowed limits for 5.1.












vSphere is very central to what we do, so we went down the road of restoring backups. Unfortunately that avenue ended in the same place. That darn vmware-vpxd service just would not initialize. Time to start digging.

 Let see what this Linux box is doing now that it is booted.


That doesn't seem normal. Let me guess, some long running query is stopping the vmware-vpxd service from starting.























Connect to the database and see if there are any queries running.





Sorry for the line wrap, but as expected there is a long running query.












Wow 357324 rows on a recursive query, that could explain our issue.










Before everyone freaks out about me nuking the table contents, I did a bit of research. There is actually a vSphere 5.5 article instructing users to delete the contents in certain scenarios. We are also quite desperate at this point. Time for a reboot.

Back in business! Time to upgrade to vSphere 5.5.

Comments

Post a Comment

Popular posts from this blog

Cisco VRF-Lite Guest Network and OpenDNS

My basic goal of doing this was to create a branded/managed OpenDNS enabled guest network on VRF-Lite. I didn't want to put any dependencies on the global routing process, I also wanted to avoid using any kind of OpenDNS Update Client for my cheapo adsl drop. In case anyone is curious this is a working config on c2900-universalk9-mz.SPA.153-3.M1.bin. Not sure why anyone else would ever try to do this, but this may be helpful if you do. ip vrf GUEST_VRF ip dhcp excluded-address 192.168.124.1 ip dhcp pool GUEST_NETWORK vrf GUEST_VRF network 192.168.124.0 255.255.255.0 dns-server 208.67.222.222 208.67.220.220 default-router 192.168.124.1 Create a vrf and guest network dhcp pool. ip domain lookup source-interface FastEthernet0/0/0 ip domain name guestvrf.net ip name-server 208.67.222.222 ip name-server 208.67.220.220 ip ddns update method DDNS_OPENDNS HTTP add https://USERNAME:PASSWORD@updates.opendns.com/nic/update?hostname=SITECODE interval maximum 0 1 0 0 I wanted...
Read more

Work Folders, Folder Redirection, Symbolic Links, Oh My!

Image
Roaming profiles are torture for admins, but users want their files and settings, and old habits die hard as they say. This is my dilemma. Roaming profiles have created legions of users who can't do basic configuration of many desktop applications, as they only had to tough it out once four years ago. Don't even get me started on the large mass of files and yo-yo copy pattern. Enter folder redirection, sure as long as you have good connectivity. A small branch of 10 users with a 10Mbps circuit and 10ms latency is enough to cause user revolt. Okay then, how bout we add in offline files? Get ready to cry  and  don't just take my word.... Along comes Work Folders , which i must say is something that can deal with many scenarios. Sure it is a blatant copy of Dropbox et al., but that is a good thing! Now back to dealing with my dilemma. How can I give a roaming profile like experience with good performance, while maintaining most if not all application configurations. ...
Read more

Ansible: Good Things Come to Those Who Wait

I have been a windows admin for many years. This tenure has much to do with local job opportunities and less so about a staunch love of one side. As a person who attempts to stay current in both worlds, I am currently dabbling with Configuration Management or Desired State Configuration (MS Speak). Because my place of employment doesn't have a decent sized GNU/Linux deployment I am really only able to go it alone in lab environments, but hey, I still enjoy it! CM is not new, the Chef and Puppet camps have been around for quite some time. I simply never started the journey with either product, thus I won't comment on them, other than to say it was probably because I was too lazy. :) Fast forward to today and there are several in the CM space. Salt, Ansible, Puppet, Chef, etc. I did some basic research and landed on Ansible. There isn't a ton of thought that went into this conclusion, but as a long time windows admin with a disdain for agents... well you get my point. ...
Read more