We have a pretty normal single master MySQL setup.
Since we have a read heavy application it makes sense. Everyone writes to the master and reads from a large pool of read-only slaves.
But, with more and more slaves it becomes hard to manage what nodes read from what slaves. It can get unmanageable pretty quick when configuring the app servers.
If we lose a MySQL slave, we have to redirect all of those servers to the new one… which descends into a bunch of temporary app config or DNS changes that sometimes are not temporary :/
The stuff in this article isn’t my bit of magic, but it is what we have been using in one of our three datacenters for about a year now and am hoping to migrate the others to the scheme. My boss and an ex co-worker set it up an I think it is pretty nice.
Keepalived is a very handy piece of ops-sauce. Dash some on your operations project and it adds a bit of tangy high availability and an aroma of robust fail-over.
It implements a VRRPv2 stack to handle LVS director failover and acts as a userspace daemon for LVS cluster nodes healthchecks and LVS directors failover.
While trying to reverse engineer how a previous co-worker setup a MySQL load balancing scheme using keepalived I discovered how difficult it was to find rpms for it (I found 1.1.10 out there). I’ll be posting later on the MySQL HA scheme later.
I tried building the latest version (1.2.2) which continually broke in RHEL5 (despite there being a RHEL6 rpm)… so I gave in and built the latest version of the previous release (1.1.20).
Here we go…
Ha! First day of my long awaited vacation and what do I do? Write a blog post about stuff I do at work of course!
A good portion of our team prefers to interface with Hive programatically using the Hive Thrift Server
The more we rely on it, the more we need to harden it.
It is not really setup or packaged for this so we need to go to town on it.