Setting up ZFS on Debian in 10 minutes

We run a small Citrix XenServer cluster at work, for our internal servers, and we had been running just a simple raid-1 array on the backend server. However the idea of SSD cache peaked my interest, so I backed up our storage repo one weekend, and reinstalled the server.

Here’s how I installed it:

apt-get install build-essential gawk alien fakeroot linux-headers-$(uname -r) zlib1g-dev uuid-dev libblkid-dev libselinux-dev parted lsscsi

#Install SPL
wget http://github.com/downloads/zfsonlinux/spl/spl-0.6.0-rc11.tar.gz
tar -xzvf spl-0.6.0-rc11.tar.gz
cd spl-0.6.0-rc11/
./configure
make deb
dpkg -i *.deb

cd ..
# Install ZFS
wget http://github.com/downloads/zfsonlinux/zfs/zfs-0.6.0-rc11.tar.gz
tar -xzvf zfs-0.6.0-rc11.tar.gz
cd zfs-0.6.0-rc11/
./configure
make deb
dpkg –I *.deb

# have a look at /dev/disk/by-id, to get physical location mappings to drive names
ls -l /dev/disk/by-id/

# and shove them in here:
#e.g.

vi /etc/zfs/zdev.conf
1tb_1 pci-0000:03:06.0-scsi-0:0:0:0
1tb_2 pci-0000:03:06.0-scsi-1:0:0:0
1tb_3 pci-0000:03:06.0-scsi-2:0:0:0
ssd_1 pci-0000:00:11.0-scsi-2:0:0:0

zpool create storagepool raidz 1tb_1 1tb_2 1tb_3
zpool attach storagepool cache ssd_1

zfs create storagepool/pool

# Install iscsitarget to point our xenserver cluster at
apt-get install iscsitarget iscsitarget-dkms

# and create a 500gb backing volume
zfs create -V 500G storagepool/iscsivol01

vi /etc/iet/ietd.conf
Target iqn.2012-01.local.icongroup.icon-szfs01:iscsivol01
Alias iscsivol01
Lun 0 Path=/dev/storagepool/iscsivol01,Type=fileio,ScsiId=2012110201,ScsiSN=2012110201

vi /etc/default/iscsitarget
ISCSITARGET_ENABLE=true

# also remember to set targets.allow and initiators.allow as needed

/etc/init.d/iscsitarget restart

# all good to go!

MySQL Multi-Master Replication Setup

So we have a bunch of websites for different markets, running wordpress, which we would ideally like hosted in their home market. BUT, we want to be able to fail them over to a different country, should the servers in their country go down. Failover in a MySQL master-slave relationship is always a bit of a pain (as it is with any DB engine) – once you’ve failed, you really can’t ‘go back’ to the original master, until you’ve re-synced it all. Which isn’t overly easy when you only have a 1 hour per 24-hr-period maintenance window, across all the markets your company operates in.

Enter MySQL Multi-Master replication. Make a change on on server? It appears on the other. Make a change on the other server? It appears on the first!

The way this works, is that each MySQL server can be both a Master, AND a Slave. So Take two servers, A and B. Any changes made on A are played via logs to the B server. Similarly, any changes on the B server are pushed to the A server. Well, actually it’s a little more than that, as Server A will send the updates it receives from Server B, on to Server B. Why does it do this?

Well, we might have six Masters! Going Master A -> Master B -> Master C -> Master D -> Master E -> Master F. And Master F is feeding Master A. All a nice big circle. So when you make a change on Master B, it propagates to C, D, E, F, and then to A. AND back to B. But B knows not to replicate its own changes on again, and they stop there.

It’s easiest to set this all up with fresh, clean, servers.

I installed MySQL-server on two clean Debian VM’s, one in Australia, one in Ireland.

Configuration

Server A – /etc/my.cnf
Add the following:

[mysqld]
# ... other configuration, tuning, etc ...
server-id = 10
# Make sure this partition has space to log bin, relay and whatever else!
log-bin = /var/lib/mysql/bin.log
relay-log = /var/lib/mysql/slave-relay.log
relay-log-index = /var/lib/mysql/slave-relay-log.index
# Creating some room between pk ids, we can always manually insert if need be.
auto_increment_increment = 10
auto_increment_offset = 1
# This is the default, but let's be safe and ensure it's on
replicate-same-server-id = FALSE
# Want more slaves in the future with writes going to both masters?
log-slave-updates = TRUE
# If there's a reboot, let's not auto start replication. - we need to make sure of where we are, and start it manually..
skip-slave-start = TRUE

Server B – /etc/my.cnf

[mysqld]
# ... other configuration, tuning, etc ...
server-id = 11
log-bin = /var/lib/mysql/bin.log
relay-log = /var/lib/mysql/slave-relay.log
relay-log-index = /var/lib/mysql/slave-relay-log.index
auto_increment_increment = 10
auto_increment_offset = 2
replicate-same-server-id = FALSE
log-slave-updates = TRUE
skip-slave-start = TRUE

You could add multiple more servers here, just increment the server-id, and the auto_increment_offset.

Starting Replication

To start replication, we first need to create a replication user on both servers, then setup the replication attributes.

First, create the replication user on both servers:
Server A

# mysql -u root -p
mysql> GRANT REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO repluser@'serverb.ip.address' IDENTIFIED BY 'replpassword';

Server B

# mysql -u root -p
mysql> GRANT REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO repluser@'servera.ip.address' IDENTIFIED BY 'replpassword';

Find the master info on Server B:

mysql> show master status;
+------------+----------+--------------+------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+------------+----------+--------------+------------------+
| bin.000001 | 294 | | |
+------------+----------+--------------+------------------+
1 row in set (0.00 sec)

Now we want to start the replication on Server A, using the info from Server B:

mysql>CHANGE MASTER TO
MASTER_HOST='92.1.1.1',
MASTER_USER='repluser',
MASTER_PASSWORD='replpassword',
MASTER_LOG_FILE='bin.000001',
MASTER_LOG_POS=294;
mysql>start slave;
mysql>show slave status\G

You may need to run the show slave status\G a few times, before the slave drops into the standard ‘Waiting for master to send event’ state.

Once this is done, you can then work on repeating this process to start Server B slaving from Server A.

Find the master info on Server A:

mysql> show master status;
+------------+----------+--------------+------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+------------+----------+--------------+------------------+
| bin.000001 | 293 | | |
+------------+----------+--------------+------------------+
1 row in set (0.00 sec)

Now we want to start the replication on Server A, using the info from Server B:

mysql>CHANGE MASTER TO
MASTER_HOST='202.62.1.1',
MASTER_USER='repluser',
MASTER_PASSWORD='replpassword',
MASTER_LOG_FILE='bin.000001',
MASTER_LOG_POS=293;
mysql>start slave;
mysql>show slave status\G

Exporting/Importing the Data

Now you want to create any databases, users, grants, and then import any data you want. Keep an eye on the ‘show slave status\G’ on the server opposite to where you’re doing all this, to make sure it is replicating correctly :)

And you’re done!

Enabling SNMP on Xenserver 6

I spent an hour this afternoon cleaning up our OpManager install, making sure the new servers in the NZ office are monitored correctly.

And then I realised that we don’t have SNMP enabled on the XenServer hosts over there.

Turns out it’s not too hard though:

Edit /etc/sysconfig/iptables

Add in amongst the other allows:

-A RH-Firewall-1-INPUT -m state –state NEW -m udp -p udp –dport 161 -j ACCEPT

# service iptables restart

Edit /etc/snmp/snmpd.conf (aka change public to something more secure)

# service snmpd restart
# chkconfig snmpd on

Yup, that was easy :)

Getting involved in SOTA

So a few years ago, I discovered SOTA. Summits on the Air. Idea is to get some activity back into Ham Radio, by doing a scoring system based on how hard it is to get to a particular summit. For example, say Canberra, you can drive to the top of Mt Ainslie. And that’s classified as a Summit. Same goes for Black Mountain. However they are only scored at 1 point each.

That said, Mt Ginini is also a Summit to which you can drive. However that is only for 4, MAYBE 5 months of the year. And it’s a fairly hellish trip even during summer. Mt Ginini is scored at 8 points!

Back when i first discovered it, i asked on the SOTA lists about getting SOTA happening in Australia, but it was pretty much a case of ‘well, if you want it, you organise it’. Dunno about you, but I have no interest in trying to start something of that sort! I see the sh!tfighting that goes on on the North-American SOTA lists, and I’m not real interested in that..

However last week, I saw that VK1 finally had SOTA summits going live, and, well, I know the Vk1 guys, and they’re alright! So I asked ‘what about Vk2?’. And apparently VK2 is on the way, but needs folks to map out summits.

So its time to get my arse into gear and help out :)

First thing, you need some maps!

So off you head to Geoscience Australia. On the Right hand side of their homepage, is a section titled ‘Products’. In that list, you want to click on ‘Data/Applications’. Once on the Data and Applications page, you want to click on ‘Free data downloads’.

You’ll want to select ‘Topography’ here, and then pop in the area you’re interested in. I’m thinking I’ll look at Sydney, as that’s close to home, and noone is looking after it as yet. So I’ll pop in ‘sydney’, and click ‘Show me the results’.

On the Product Selection screen, I’ll select ‘Sydney Special’, and click on Continue to File Selection.

I use GRASS as a GIS tool in Windows, and occasionally MapInfo (I have a copy of 6.5, which is pretty old, but it still does the job for what I do..), so I’ll select the MapInfo native format, and ESRI personal geodatabase to pull into GRASS. I’ll also use the Georeferenced PDF, so i can dump it into dropbox, and use it on my ipad later :)

You’ll want to select a Sector and Industry which make some sense to keep GA happy.

And then off to download!.

What a crazy weekend

So a Classic Industries ‘mini tourer’ popped up on ebay last week for $1200 Buy It Now. It was an ’86 model, so fairly old shape, but I still couldn’t resist – that was basically what I had budgeted for building a trailer! So I phoned the guy, had a good chat, and bought it. I’ll ride down on the weekend to pick it up, thinks I – it’s in Canberra, but that’s only 4 hours each way. Do it on the bike, and it’ll cost 80 bucks, max.

7am Saturday morning rolls around, and off I ride.30 mins down the road, and all my auxiliary power devices shut down. iPod, GPS, autocom, etc. Check fuses, nope those are all good. Ended up with the tank raised, inside what I call the ‘Brain box’ – whee all the relays live on the k100. I have NO idea what I was thinking when I first wired the bike up, but I had a daisy chain cum star wiring next of earth wires to all the relays and various circuit feeds. And this was iffy. Messed with it for a while, thinking I’ll fix it tomorrow over a relaxing day, and got everything working. Back on the road again, 30 mins behind.

Roll off the throttle 15km before goulburn, and the bike dies. UGH! Pull over, she starts fine. Next time I roll off, same thing! Babied it into goulburn, removed the fuel filter up-ended it onto the driveway, and wow, that’s a lot of water and other crap!

Continue to Canberra, no probs. It’s way too hot, so I strap my jacket across the back seat, and continue on in my tshirt. I haven’t burnt in over 10 years, so I’m not scared of the sun. I remember back to my Summernats days, the Maori guys would be lathering up with sunscreen, I never bothered, I just get more brown, if anything..

Arrive in Canberra at the sellers house, bugger me, I’m going red! UV must be through the roof today! Trailer is a little beauty! Went across to Duffy to catch up with my best man for a while, then headed off northbound.

Wow, the trailer moves around! The seller, Allan, did say you need to have a certain amount of weight in it. Apparently the bag of soil he put in for me wasn’t enough :( ‘well, I have an esky’ I think, though it isn’t currently mounted – it’s sitting inside the trailer. I buy three bags of ice at the next servo (and some drinks), and the trailer is now towing perfectly!

20km short of Goulburn, the bike starts playing up again – the same old dying on power-off problem as on the way down! I manage to baby it to goulburn, empty the filter, yup it’s got water again! I go buy a 20l jerry can (cost me 60 bucks!!), remove the tank, and drain the tank into it – wow there was a lot of gunk! Reassemble the bike, and off I go again. By this time, it’s 8pm! I get 10km down the road, and the problem re-appears! I managed to baby it another 15km, then it gave up totally, and would not fire at all.

Ended up getting towed to goulburn, mrs drove down and picked me up, and towed the trailer home, then I did another trip down to get the bike on the Sunday. Time to look at the bike during the week!

All up, I was on the road for 34hr continuously – not a good weekend!