I love Cobbler.
Cobbler + Chef in my environment means that I can go from bare metal to an active cluster node in moments with little effort.
It is a powerful system for managing kickstart profiles, pxeboot, power, dhcp, dns etc..
Below are some notes to help get you going with just the basic feature set. It is a system you can easy go nuts with to automate a lot of your infrastructure.
Following up from my last post on creating a simple yum repository, here is how to setup a local CentOS mirror.
Here is some low hanging fruit to improve your RHEL environment and simplify your work… setup a simple Yum repository.
If you follow this blog, you can see I get to play with all sorts of awesome stuff.
Y U JELLY? No need to be.
There is so much awesome stuff to do… I don’t have time to do all the awesome stuff there is to do.
Come work with me and do awesome stuff.
It’ll be awesome.
Awesome Web Operations Engineer
Awesome Junior Web Operations Engineer
Director of Awesome IT
I do not apologize for my gratuitous use of the word awesome. If you can’t take this much awesome, you probably couldn’t handle working at Outbrain.
I’m pretty hard on vendors.
I can be down right mean sometimes when arguing for what I think is right. When I am fed buzzwords and technobabbel or smell BS I call them out on it.
Free drinks, swag or fancy steak dinners won’t get my business. Price, performance and responsiveness to my input and problems will. (Though, I’ll certainly let you guys keep trying since I love steak).
When I find technologies or vendors I like I latch on to them and nerdgush about them on this blog or to my peers. The C-Series servers are a good example. They’re pretty cool and allow me to hit my goals for pricing, density and other features, but the real value is the Dell Data Center Solutions team. Other vendors may have good-enough products and better pricing, but they don’t have anything like these guys.
Over the last year Google Analytics says I’ve been getting a lot of hits from search that indicate there are some folks who want to know how the C-series DRAC works.
It is easy enough to setup like any other IPMI/DRAC system.
First you’ll need to plug the IPMI/Management Ethernet port into your network (preferably an our of band (OOB) network seperate from your production network). In the BIOS, make sure the management port is set to ‘Dedicated’, earlier ones shipped with it set to ‘Shared’ by default which precluded the dedicated IPMI port.
We have a pretty normal single master MySQL setup.
Since we have a read heavy application it makes sense. Everyone writes to the master and reads from a large pool of read-only slaves.
But, with more and more slaves it becomes hard to manage what nodes read from what slaves. It can get unmanageable pretty quick when configuring the app servers.
If we lose a MySQL slave, we have to redirect all of those servers to the new one… which descends into a bunch of temporary app config or DNS changes that sometimes are not temporary :/
The stuff in this article isn’t my bit of magic, but it is what we have been using in one of our three datacenters for about a year now and am hoping to migrate the others to the scheme. My boss and an ex co-worker set it up an I think it is pretty nice.
Gorilla Party Rocking your logs like an open-source mogul.
Graylog2‘s moto should be LMFAO (logging my freaking apps off).
Graylog2 is lovely little Splunk-like server that collects your logs and provides a nice interface for searching and analyzing them.
From the site
Graylog2 is an open source log management solution that stores your logs in ElasticSearch. It consists of a server written in Java that accepts your syslog messages via TCP, UDP or AMQP and stores it in the database. The second part is a web interface that allows you to manage the log messages from your web browser.
They have lovely screen shots here.
The only problem with it is it has quite a few moving parts that need to be installed that are not traditionally easy to get going on CentOS.
So, here is my guide.
Keepalived is a very handy piece of ops-sauce. Dash some on your operations project and it adds a bit of tangy high availability and an aroma of robust fail-over.
It implements a VRRPv2 stack to handle LVS director failover and acts as a userspace daemon for LVS cluster nodes healthchecks and LVS directors failover.
While trying to reverse engineer how a previous co-worker setup a MySQL load balancing scheme using keepalived I discovered how difficult it was to find rpms for it (I found 1.1.10 out there). I’ll be posting later on the MySQL HA scheme later.
I tried building the latest version (1.2.2) which continually broke in RHEL5 (despite there being a RHEL6 rpm)… so I gave in and built the latest version of the previous release (1.1.20).
Here we go…
I find myself running more and more Cassandra clusters and when we were on Chef 0.9.8 I was being lazy and just cloning my Cassandra cookbook per cluster. Not exactly a way to scale the manageability of your config
Now I’ve refactored the cookbook to allow me to manage multiple clusters by extracting the
initial_token from a databag. Once we start implementing the new Environments feature in Chef 0.10 I’ll be able to simplify this further.
I’m debating having the cookbook auto-generate tokens and assign them as well as re-generate/nodetool move/re-balance when I’ve added another node with that cluster specified in the databag. That’s a big project and for now I’m too much of a control freak to automate that, but I’m thinking on it.
I’ve also made it so the cookbook auto-generates the
cassandra-topology.properties for the
PropertyFileSnitch based off of location info stored in the databag.