Scaleoutme

Our VMware vSphere 5.1 server died over the weekend. Basically the vmware-vpxd service wouldn't initialize. This prompted a call to VMware support. VMware support was on the phone with us for two days, at which point they issued the dreaded Prima Nocte of support. "Oh Sorry, it seems you are not running a supported config, we can not go any further with this case, do you agree with this statement?" We aren't running a supported config? Ouch. The VMware tech pointed us to an article outlining the maximum hosts and guests for the embedded database on vSphere 5.1 . Unfortunately for us, we are slightly over the allowed limits for 5.1. vSphere is very central to what we do, so we went down the road of restoring backups. Unfortunately that avenue ended in the same place. That darn vmware-vpxd service just would not initialize. Time to start digging.  Let see what this Linux box is doing now that it is booted. That doesn't seem normal. Let me ...

My story begins with slowness, like many IT stories before it. Let me give some context. This is regarding a custom application hosted at an MSP. The problem started about a week after production launch. Staff and the broader customer base began experiencing slowness and timeouts. The usual answer ensues. "The servers look good" says MSP. "CPU doesn't go above 5%, tons of free memory, disk time is less than 1%, network utilization is low. It isn't anything on our side." Now with the onus successfully shifted, the problem lands internally. Pinging the application host at the MSP (MPLS circuit) produces a 1ms response time, impressive latency for sure. Next up, I was able to capture a "slow" event in wireshark: The first line is a query from my client to the server. The next two packets are from the server to my client. Strange result for a machine "doing nothing" as the MSP said. It took over half a second to start returning t...

Visio to represent network segmentation using firewalls and VRFs.

If you have read some of my other blog posts then you know I have some design philosophies that I abide by. Let me introduce to you a new one. Irresponsibility. If your design takes into account the manual actions of people, then failure is imminent, and failure is further exacerbated by admin turnover. This leads me to my story. How do I protect user data from workstation admins tasked with supporting workstations? No this is not a rhetorical question. The design takes into account drive-by admins, i.e. connect as local admin and steal the data. I'm not really sure how many of these admins exist, however someone in the IT security department is adamant they are plentiful, and made it clear they must be stopped. So the usual EFS mantra is that a CA/SubCA should be used for the EFS/Recovery Agent certificates. I wholeheartedly agree, but in many IT shops this is a lot of complexity and headache. In addition to the complexity there is the problem of remembering to renew the data ...