Scaleoutme

Our VMware vSphere 5.1 server died over the weekend. Basically the vmware-vpxd service wouldn't initialize. This prompted a call to VMware support. VMware support was on the phone with us for two days, at which point they issued the dreaded Prima Nocte of support. "Oh Sorry, it seems you are not running a supported config, we can not go any further with this case, do you agree with this statement?" We aren't running a supported config? Ouch. The VMware tech pointed us to an article outlining the maximum hosts and guests for the embedded database on vSphere 5.1 . Unfortunately for us, we are slightly over the allowed limits for 5.1. vSphere is very central to what we do, so we went down the road of restoring backups. Unfortunately that avenue ended in the same place. That darn vmware-vpxd service just would not initialize. Time to start digging.  Let see what this Linux box is doing now that it is booted. That doesn't seem normal. Let me ...

My story begins with slowness, like many IT stories before it. Let me give some context. This is regarding a custom application hosted at an MSP. The problem started about a week after production launch. Staff and the broader customer base began experiencing slowness and timeouts. The usual answer ensues. "The servers look good" says MSP. "CPU doesn't go above 5%, tons of free memory, disk time is less than 1%, network utilization is low. It isn't anything on our side." Now with the onus successfully shifted, the problem lands internally. Pinging the application host at the MSP (MPLS circuit) produces a 1ms response time, impressive latency for sure. Next up, I was able to capture a "slow" event in wireshark: The first line is a query from my client to the server. The next two packets are from the server to my client. Strange result for a machine "doing nothing" as the MSP said. It took over half a second to start returning t...