Category Archives: IT

Exporting a Sophos UTM Configuration into Something You Can Actually Migrate From

Migrating away from Sophos UTM can be awkward. The WebAdmin interface gives you a configuration backup, but that backup is not especially friendly if your goal is to understand firewall rules, NAT, aliases, services, routing, WAF objects, or VPN leftovers.

The .abf backup is useful for disaster recovery, but it is not a migration document. It is a bundled configuration state file. You can pull strings out of it, but you will quickly end up spelunking through object references, internal IDs, and historical cruft.

A better approach is to use the Sophos UTM WebAdmin API to export the object graph into JSON, then process that into readable files.

That gives you something much more useful for migration planning:

  • Network objects
  • Host objects
  • Network groups
  • Service definitions
  • Service groups
  • Firewall rules
  • NAT rules
  • Masquerading rules
  • Static routes
  • Policy routes
  • WAF entries
  • Mail protection leftovers
  • VPN objects
  • OSPF/BGP routing objects
  • References between rules and objects

Once exported, you can review what is actually in use, identify stale rules, and start mapping Sophos concepts into your next firewall platform, whether that is OPNsense, pfSense, VyOS, FortiGate, Juniper SRX, MikroTik, Linux nftables, or something completely custom.

Why not just use the backup file?

Sophos UTM backups are great if your goal is restoring Sophos UTM.

They are much less helpful if your goal is migration.

The backup contains the whole appliance configuration, but it is not shaped like a clean firewall policy export. You may be able to extract strings and object names, but it is not pleasant to answer questions like:

  • Which firewall rules reference this host?
  • Which NAT rules still matter?
  • Which service groups include TCP and UDP members?
  • Which rules reference deleted or stale objects?
  • Which WAF entries are still tied to services we no longer run?
  • Which aliases are genuinely reused versus one-off leftovers?
  • Which objects are safe to ignore?

The WebAdmin API gives you structured JSON. That is the difference between archaeology with a toothbrush and having a map.

Enabling the Sophos UTM WebAdmin API

In WebAdmin, enable the REST API under the WebAdmin/API settings.

The exact menu name can vary slightly by UTM version, but generally you need to:

  1. Enable the WebAdmin REST API.
  2. Create or identify an administrative account that can access the API.
  3. Make sure your export host is allowed to connect to WebAdmin.
  4. Confirm HTTPS access to the UTM WebAdmin port.

Most UTM appliances use a self-signed certificate, so your client will probably need to disable TLS verification for this one-off export, or trust the appliance certificate.

API export strategy

The Sophos API is object-reference heavy. A firewall rule often does not contain the full source, destination, and service definition directly. Instead, it references objects by REF IDs.

So the export process needs two layers:

  1. Export major collections such as firewall rules, NAT rules, network objects, and service objects.
  2. Resolve references later when generating human-readable output.

For example, a firewall rule may reference:

REF_NetHosDmzGeppetto
REF_ServiceTcpHttps
REF_NetGrpInternalNetworks

Those references are not useful by themselves. You need the object export too, so you can turn them into:

DMZ-Geppetto
HTTPS
Internal Networks

Example PHP API client

Here is a small PHP client that can connect to Sophos UTM and export a set of common API endpoints to JSON files.

It is deliberately simple. It is not a full SDK. Its job is to get the configuration out of the appliance and into a directory where you can inspect, process, diff, and transform it.

<?php

declare(strict_types=1);

final class SophosUtmClient
{
    private string $baseUrl;
    private string $username;
    private string $password;
    private bool $verifyTls;

    public function __construct(
        string $baseUrl,
        string $username,
        string $password,
        bool $verifyTls = false
    ) {
        $this->baseUrl = rtrim($baseUrl, '/');
        $this->username = $username;
        $this->password = $password;
        $this->verifyTls = $verifyTls;
    }

    public function get(string $path): array
    {
        $url = $this->baseUrl . '/' . ltrim($path, '/');

        $ch = curl_init($url);

        curl_setopt_array($ch, [
            CURLOPT_RETURNTRANSFER => true,
            CURLOPT_USERPWD => $this->username . ':' . $this->password,
            CURLOPT_HTTPAUTH => CURLAUTH_BASIC,
            CURLOPT_HTTPHEADER => [
                'Accept: application/json',
            ],
            CURLOPT_SSL_VERIFYPEER => $this->verifyTls,
            CURLOPT_SSL_VERIFYHOST => $this->verifyTls ? 2 : 0,
            CURLOPT_CONNECTTIMEOUT => 10,
            CURLOPT_TIMEOUT => 60,
        ]);

        $body = curl_exec($ch);

        if ($body === false) {
            $error = curl_error($ch);
            curl_close($ch);
            throw new RuntimeException("cURL error calling {$url}: {$error}");
        }

        $status = curl_getinfo($ch, CURLINFO_HTTP_CODE);
        curl_close($ch);

        if ($status < 200 || $status >= 300) {
            throw new RuntimeException("HTTP {$status} from {$url}: {$body}");
        }

        $decoded = json_decode($body, true);

        if (!is_array($decoded)) {
            throw new RuntimeException("Invalid JSON from {$url}: {$body}");
        }

        return $decoded;
    }
}

Export script

This script loops over a set of useful Sophos UTM endpoints and writes the responses to JSON files.

<?php

declare(strict_types=1);

require __DIR__ . '/SophosUtmClient.php';

$baseUrl = getenv('SOPHOS_URL') ?: '';
$username = getenv('SOPHOS_USERNAME') ?: '';
$password = getenv('SOPHOS_PASSWORD') ?: '';
$outputDir = getenv('SOPHOS_EXPORT_DIR') ?: __DIR__ . '/sophos-export';

if ($baseUrl === '' || $username === '' || $password === '') {
    fwrite(STDERR, "Required environment variables:\n");
    fwrite(STDERR, "  SOPHOS_URL=https://utm.example.com:4444/api\n");
    fwrite(STDERR, "  SOPHOS_USERNAME=admin\n");
    fwrite(STDERR, "  SOPHOS_PASSWORD=secret\n");
    fwrite(STDERR, "Optional:\n");
    fwrite(STDERR, "  SOPHOS_EXPORT_DIR=./sophos-export\n");
    exit(1);
}

$client = new SophosUtmClient(
    baseUrl: $baseUrl,
    username: $username,
    password: $password,
    verifyTls: false
);

$endpoints = [
    // Network objects
    '/objects/network/host',
    '/objects/network/network',
    '/objects/network/group',
    '/objects/network/interface_address',
    '/objects/network/dns_host',
    '/objects/network/dns_group',
    '/objects/network/availability_group',

    // Services
    '/objects/service/tcp',
    '/objects/service/udp',
    '/objects/service/tcpudp',
    '/objects/service/icmp',
    '/objects/service/group',

    // Firewall
    '/objects/packetfilter/packetfilter',
    '/objects/packetfilter/group',

    // NAT
    '/objects/nat/masquerading',
    '/objects/nat/dnat',
    '/objects/nat/snat',
    '/objects/nat/fullnat',

    // Routing
    '/objects/routing/static_gateway_route',
    '/objects/routing/static_interface_route',
    '/objects/routing/policy_route',

    // Interfaces
    '/objects/interface/ethernet',
    '/objects/interface/vlan',
    '/objects/interface/bridge',
    '/objects/interface/pppoe',

    // Web protection / WAF
    '/objects/reverse_proxy/frontend',
    '/objects/reverse_proxy/backend',
    '/objects/reverse_proxy/profile',

    // Mail protection
    '/objects/mail/smtp/profile',
    '/objects/mail/smtp/route',
    '/objects/mail/pop3/profile',

    // VPN-ish objects
    '/objects/ipsec/connection',
    '/objects/ipsec/remote_gateway',
    '/objects/openvpn/site_to_site',
    '/objects/openvpn/remote_access',

    // Routing daemons, if used
    '/objects/ospf/area',
    '/objects/ospf/interface',
    '/objects/bgp/system',
    '/objects/bgp/neighbor',
];

if (!is_dir($outputDir) && !mkdir($outputDir, 0775, true)) {
    throw new RuntimeException("Could not create output directory: {$outputDir}");
}

foreach ($endpoints as $endpoint) {
    $safeName = trim($endpoint, '/');
    $safeName = str_replace('/', '__', $safeName);
    $file = $outputDir . '/' . $safeName . '.json';

    echo "Exporting {$endpoint} -> {$file}\n";

    try {
        $data = $client->get($endpoint);

        file_put_contents(
            $file,
            json_encode($data, JSON_PRETTY_PRINT | JSON_UNESCAPED_SLASHES)
        );
    } catch (Throwable $e) {
        $errorFile = $outputDir . '/' . $safeName . '.error.txt';

        file_put_contents($errorFile, $e->getMessage() . PHP_EOL);

        echo "  ERROR: {$e->getMessage()}\n";
    }
}

Run it like this:

export SOPHOS_URL='https://utm.example.com:4444/api'
export SOPHOS_USERNAME='admin'
export SOPHOS_PASSWORD='your-password'
export SOPHOS_EXPORT_DIR='./sophos-export'

php export-sophos.php

After the export, you should have a directory full of files like:

objects__network__host.json
objects__network__network.json
objects__network__group.json
objects__service__tcp.json
objects__service__udp.json
objects__packetfilter__packetfilter.json
objects__nat__dnat.json
objects__nat__masquerading.json

Turning the export into a readable inventory

Raw JSON is better than a backup file, but it is still not migration-ready. The next step is to build lookup tables.

The basic idea is:

  1. Read every exported JSON file.
  2. Index every object by its Sophos REF.
  3. Use that index to resolve firewall rules, NAT rules, and groups.
  4. Emit readable CSV, Markdown, YAML, or JSON.

Here is a simplified object indexer:

<?php

declare(strict_types=1);

function loadJsonFiles(string $dir): array
{
    $objects = [];

    foreach (glob($dir . '/*.json') as $file) {
        $data = json_decode((string) file_get_contents($file), true);

        if (!is_array($data)) {
            continue;
        }

        foreach (normaliseSophosCollection($data) as $item) {
            if (!is_array($item)) {
                continue;
            }

            $ref = $item['_ref'] ?? $item['ref'] ?? null;

            if (is_string($ref) && $ref !== '') {
                $objects[$ref] = $item + [
                    '_source_file' => basename($file),
                ];
            }
        }
    }

    return $objects;
}

function normaliseSophosCollection(array $data): array
{
    if (array_is_list($data)) {
        return $data;
    }

    foreach (['objects', 'items', 'result', 'data'] as $key) {
        if (isset($data[$key]) && is_array($data[$key])) {
            return array_is_list($data[$key]) ? $data[$key] : array_values($data[$key]);
        }
    }

    return array_values($data);
}

function objectName(?string $ref, array $index): string
{
    if ($ref === null || $ref === '') {
        return '';
    }

    if (!isset($index[$ref])) {
        return $ref;
    }

    return $index[$ref]['name'] ?? $index[$ref]['_ref'] ?? $ref;
}

Then you can use it to make a readable firewall rule export.

<?php

declare(strict_types=1);

require __DIR__ . '/sophos-index.php';

$exportDir = $argv[1] ?? './sophos-export';

$index = loadJsonFiles($exportDir);

$rulesFile = $exportDir . '/objects__packetfilter__packetfilter.json';

if (!is_file($rulesFile)) {
    throw new RuntimeException("Missing firewall rules export: {$rulesFile}");
}

$rulesRaw = json_decode((string) file_get_contents($rulesFile), true);
$rules = normaliseSophosCollection($rulesRaw);

$out = fopen('php://output', 'w');

fputcsv($out, [
    'enabled',
    'position',
    'name',
    'source',
    'service',
    'destination',
    'action',
    'comment',
]);

foreach ($rules as $rule) {
    if (!is_array($rule)) {
        continue;
    }

    $sources = array_map(
        fn($ref) => objectName($ref, $index),
        (array)($rule['sources'] ?? $rule['source'] ?? [])
    );

    $services = array_map(
        fn($ref) => objectName($ref, $index),
        (array)($rule['services'] ?? $rule['service'] ?? [])
    );

    $destinations = array_map(
        fn($ref) => objectName($ref, $index),
        (array)($rule['destinations'] ?? $rule['destination'] ?? [])
    );

    fputcsv($out, [
        empty($rule['disabled']) ? 'yes' : 'no',
        $rule['position'] ?? '',
        $rule['name'] ?? '',
        implode(', ', $sources),
        implode(', ', $services),
        implode(', ', $destinations),
        $rule['action'] ?? '',
        $rule['comment'] ?? $rule['description'] ?? '',
    ]);
}

Run it:

php readable-firewall-rules.php ./sophos-export > firewall-rules.csv

Now you have something that humans can work with.

Important migration gotchas

Sophos groups do not always map cleanly

Sophos may allow a rule to refer to several objects directly. Your new platform may require a single alias, address group, or port group.

During one migration, I ran into this exact issue with OPNsense automation rules. A Sophos rule could effectively say:

source: Internal Networks, DNS Server 1, DNS Server 2
destination: Pi-hole
service: DNS

But the target firewall API wanted:

source: one alias
destination: one alias
service: one alias

So the migration process needed to create a new group alias such as:

DNS_ALLOWED_SOURCES:
  type: networkgroup
  content:
    - INT_INTERNAL_ADDRESSES
    - DNS_SERVER_1
    - DNS_SERVER_2

Then the rule could reference DNS_ALLOWED_SOURCES.

Service groups may need flattening

Sophos service groups can contain multiple service definitions, and those definitions may be TCP, UDP, or mixed.

For example:

AMP_GROUP1:
  protocol: tcp
  ports:
    - 2121:2281

AMP_GROUP10:
  protocol: tcpudp
  ports:
    - 30810:30900

For a target firewall, it may be cleaner to create one combined port alias:

AMP_PORTS:
  type: port
  content:
    - 2121:2281
    - 30810:30900

Then set the firewall rule protocol to tcp/udp.

Object names may be too long

Sophos object names can be verbose. Some target platforms have stricter alias length limits.

For example, a name like this may need shortening:

SOPHOS_OBJ_SYDNEY_VLAN102_MANAGEMENT

into something like:

so_sydney_vlan102_mgmt

If you do this, keep a mapping file:

SOPHOS_OBJ_SYDNEY_VLAN102_MANAGEMENT: so_sydney_vlan102_mgmt

That mapping file becomes extremely useful when troubleshooting after cutover.

“Any” cannot always be combined with other aliases

If a Sophos rule has something like:

source: Any, SomeSpecificAlias

the target platform may reject it.

Logically, Any + Something is just Any, so simplify it.

Special Sophos objects need manual mapping

Sophos has concepts like:

This Firewall
External WAN Address
Interface Address
SYNC net

Those may not export as normal aliases that your new firewall understands.

You may need to map them manually:

FW_ROUTING_VIP:
  type: host
  content:
    - 192.168.10.49

FW_DMZ_VIP:
  type: host
  content:
    - 103.235.52.81

Do not assume the target firewall will understand the literal string This Firewall.

Recommended output formats

For migration, I like producing several layers of output.

1. Raw JSON

Keep the original API exports untouched.

sophos-export/raw/*.json

This is your evidence locker.

2. Resolved JSON

Create a version where references are expanded.

{
  "name": "Allow DNS to Pi-hole",
  "source": [
    {
      "ref": "REF_NetGrpInternal",
      "name": "Internal Networks"
    }
  ],
  "service": [
    {
      "ref": "REF_SerDns",
      "name": "DNS",
      "protocols": ["tcp", "udp"],
      "ports": ["53"]
    }
  ],
  "destination": [
    {
      "ref": "REF_NetHostPiHole",
      "name": "DMZ Pi-hole"
    }
  ]
}

3. CSV for review

This is useful for non-automation review.

enabled, position, name, source, service, destination, action, comment

4. YAML for migration

This becomes the start of your Git-managed target firewall configuration.

allow-dns-to-pihole:
  enabled: true
  source_net: DNS_ALLOWED_SOURCES
  destination_net: DMZ_PI_HOLE
  destination_port: DNS
  protocol: tcp/udp
  action: pass

Suggested migration workflow

A practical migration workflow looks like this:

  1. Export the UTM API into raw JSON.
  2. Build an object reference index.
  3. Generate readable firewall/NAT/routing reports.
  4. Identify dead services, old VPNs, old mail protection, and stale WAF rules.
  5. Create target-platform aliases.
  6. Create target-platform service aliases.
  7. Convert high-value firewall rules first.
  8. Leave questionable imported rules disabled or marked for review.
  9. Manually validate NAT, routing, and special firewall-local rules.
  10. Test cutover using logs, packet captures, and known traffic flows.

Do not aim for a perfect one-to-one clone.

A firewall migration is a chance to delete years of sediment. Treat the Sophos export as a historical document, not a sacred scroll.

Lessons learned

The biggest lesson is that exporting the config is only step one.

The hard part is translating firewall semantics.

Sophos might allow constructs that your target platform rejects. Your target platform may have different ideas about aliases, port groups, NAT, interface matching, or firewall-local addresses.

In my case, the useful path was:

Sophos UTM API
  -> raw JSON export
  -> resolved object graph
  -> reviewable YAML
  -> target firewall rules
  -> manual cleanup and validation

The raw export gave me confidence. The readable YAML gave me control. The manual review kept me from blindly migrating old junk.

And that is the real goal: not just to move the firewall config, but to understand it well enough that the new firewall starts life cleaner than the old one ended.

SSD Caching – Actually doing it

So, caching is built right into LVM these days.  It’s quite neat.  I’m testing this on my OTHER caching box – a shallow 1RU box with space for two SSD’s and that’s about it.

First step is to mount an ISCSI target.  I’m just mounting a target I created on my fileserver, to save some latency (I can mount the main SAN from the DC, but there’s 15ms latency due to the EOIP tunnel over the ADSL here). There’s a much more detailed writeup of this Here

root@isci-cache01:~# iscsiadm -m discovery -t st -p 192.168.102.245 192.168.102.245:3260,1 iqn.2012-01.net.rendrag.fileserver:dedipi0
192.168.11.245:3260,1 iqn.2012-01.net.rendrag.fileserver:dedipi0

root@isci-cache01:~# iscsiadm -m node --targetname "iqn.2012-01.net.rendrag.fileserver:dedipi0" --portal "192.168.102.245:3260" --login
Logging in to [iface: default, target: iqn.2012-01.net.rendrag.fileserver:dedipi0, portal: 192.168.102.245,3260] (multiple)
Login to [iface: default, target: iqn.2012-01.net.rendrag.fileserver:dedipi0, portal: 192.168.102.245,3260] successful.

[ 193.182145] scsi6 : iSCSI Initiator over TCP/IP
[ 193.446401] scsi 6:0:0:0: Direct-Access IET VIRTUAL-DISK 0 PQ: 0 ANSI: 4
[ 193.456619] sd 6:0:0:0: Attached scsi generic sg1 type 0
[ 193.466849] sd 6:0:0:0: [sdb] 1048576000 512-byte logical blocks: (536 GB/500 GiB)
[ 193.469692] sd 6:0:0:0: [sdb] Write Protect is off
[ 193.469697] sd 6:0:0:0: [sdb] Mode Sense: 77 00 00 08
[ 193.476918] sd 6:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[ 193.514882] sdb: unknown partition table
[ 193.538467] sd 6:0:0:0: [sdb] Attached SCSI disk

root@isci-cache01:~# pvcreate /dev/sdb
root@isci-cache01:~# vgcreate vg_iscsi /dev/sdb

root@isci-cache01:~# pvdisplay
--- Physical volume ---
PV Name               /dev/sdb
VG Name               vg_iscsi
PV Size               500.00 GiB / not usable 4.00 MiB
Allocatable           yes
PE Size               4.00 MiB
Total PE              127999
Free PE               127999
Allocated PE          0
PV UUID               0v8SWY-2SSA-E2oL-iAdE-yeb4-owyG-gHXPQK
--- Physical volume ---
PV Name               /dev/sda5
VG Name               isci-cache01-vg
PV Size               238.24 GiB / not usable 0
Allocatable           yes
PE Size               4.00 MiB
Total PE              60988
Free PE               50784
Allocated PE          10204

PV UUID               Y3O48a-tep7-nYjx-gEck-bcwk-tJzP-2Sc2pP

root@isci-cache01:~# lvcreate -L 499G -n testiscsilv vg_iscsi
Logical volume "testiscsilv" created
root@isci-cache01:~# mkfs -t ext4 /dev/mapper/vg_iscsi-testiscsilv
mke2fs 1.42.12 (29-Aug-2014)
Creating filesystem with 130809856 4k blocks and 32702464 inodes
Filesystem UUID: 9aa5f499-902a-4935-bc67-61dd8930e014
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
102400000
Allocating group tables: done
Writing inode tables: done

Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

Now things get a little tricky, as I’d already installed my system with the ssd in one volume group..  I’ll be using a RAID array for the production box for PiCloud.
For now, we’ll just create an LV from the SSD VG, and then add it to the iscsi VG.

root@isci-cache01:~# lvcreate -L 150G -n iscsicaching isci-cache01-vg
Logical volume "iscsicaching" created
root@isci-cache01:~# vgextend vg_iscsi  /dev/mapper/isci--cache01--vg-iscsicaching
  Physical volume "/dev/isci-cache01-vg/iscsicaching" successfully created
  Volume group "vg_iscsi" successfully extended
 
root@isci-cache01:~# lvcreate -L 1G -n cache_meta_lv vg_iscsi /dev/isci-cache01-vg/iscsicaching
Logical volume "cache_meta_lv" created
root@isci-cache01:~# lvcreate -L 148G -n cache_lv vg_iscsi /dev/isci-cache01-vg/iscsicaching
  Logical volume "cache_lv" created
root@isci-cache01:~# lvs
LV            VG              Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
iscsicaching  isci-cache01-vg -wi-ao---- 150.00g
root          isci-cache01-vg -wi-ao----  30.18g
swap_1        isci-cache01-vg -wi-ao----   9.68g
cache_lv      vg_iscsi        -wi-a----- 148.00g
cache_meta_lv vg_iscsi        -wi-a-----   1.00g
  testiscsilv   vg_iscsi        -wi-a----- 499.00g
 
root@isci-cache01:~# pvs
PV                                VG              Fmt  Attr PSize   PFree
/dev/isci-cache01-vg/iscsicaching vg_iscsi        lvm2 a--  150.00g 1020.00m
/dev/sda5                         isci-cache01-vg lvm2 a--  238.23g   48.38g
  /dev/sdb                          vg_iscsi        lvm2 a--  500.00g 1020.00m
 
Now we want to convert these two new LV's into a 'cache pool'
root@isci-cache01:~# lvconvert --type cache-pool --poolmetadata vg_iscsi/cache_meta_lv vg_iscsi/cache_lv
WARNING: Converting logical volume vg_iscsi/cache_lv and vg_iscsi/cache_meta_lv to pool's data and metadata volumes.
THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
Do you really want to convert vg_iscsi/cache_lv and vg_iscsi/cache_meta_lv? [y/n]: y
Logical volume "lvol0" created
  Converted vg_iscsi/cache_lv to cache pool.

And now we want to attach this cache pool to our iscsi LV.

root@isci-cache01:~# lvconvert --type cache --cachepool vg_iscsi/cache_lv vg_iscsi/testiscsilv
  Logical volume vg_iscsi/testiscsilv is now cached.
 
 
root@isci-cache01:~# dd if=/dev/zero of=/export/test1 bs=1024k count=60
60+0 records in
60+0 records out
62914560 bytes (63 MB) copied, 0.0401375 s, 1.6 GB/s
root@isci-cache01:~# dd if=/dev/zero of=/export/test1 bs=1024k count=5000
^C2512+0 records in
2512+0 records out
2634022912 bytes (2.6 GB) copied, 7.321 s, 360 MB/sroot@isci-cache01:~# ls -l
total 0
root@isci-cache01:~# dd if=/export/test1 of=/dev/null
5144576+0 records in
5144576+0 records out
2634022912 bytes (2.6 GB) copied, 1.82355 s, 1.4 GB/s

Oh yeah!  Over a 15mbps network too!

Now we want to setup XFS quotas so we can have a quota per directory.

root@isci-cache01:/# echo "100001:/export/mounts/pi-01" >> /etc/projects
root@isci-cache01:/# echo "pi-01:10001" >> /etc/projid
root@isci-cache01:/# xfs_quota -x -c 'project -s pi-01' /export
root@isci-cache01:/# xfs_quota -x -c 'limit -p bhard=5g pi-01' /export

root@isci-cache01:/# xfs_quota -x -c report /export
Project quota on /export (/dev/mapper/vg_iscsi-testiscsilv)
Blocks
Project ID       Used       Soft       Hard    Warn/Grace
---------- --------------------------------------------------
pi-01         2473752          0    5242880     00 [--------]

Note: Need the thin-provisioning-tools package, and to ensure that your initramfs gets built with the proper modules included.

Sweet, so we CAN do this 🙂

Setting up ZFS on Debian in 10 minutes

We run a small Citrix XenServer cluster at work, for our internal servers, and we had been running just a simple raid-1 array on the backend server. However the idea of SSD cache peaked my interest, so I backed up our storage repo one weekend, and reinstalled the server.

Here’s how I installed it:

apt-get install build-essential gawk alien fakeroot linux-headers-$(uname -r) zlib1g-dev uuid-dev libblkid-dev libselinux-dev parted lsscsi

#Install SPL
wget http://github.com/downloads/zfsonlinux/spl/spl-0.6.0-rc11.tar.gz
tar -xzvf spl-0.6.0-rc11.tar.gz
cd spl-0.6.0-rc11/
./configure
make deb
dpkg -i *.deb

cd ..
# Install ZFS
wget http://github.com/downloads/zfsonlinux/zfs/zfs-0.6.0-rc11.tar.gz
tar -xzvf zfs-0.6.0-rc11.tar.gz
cd zfs-0.6.0-rc11/
./configure
make deb
dpkg –I *.deb

# have a look at /dev/disk/by-id, to get physical location mappings to drive names
ls -l /dev/disk/by-id/

# and shove them in here:
#e.g.

vi /etc/zfs/zdev.conf
1tb_1 pci-0000:03:06.0-scsi-0:0:0:0
1tb_2 pci-0000:03:06.0-scsi-1:0:0:0
1tb_3 pci-0000:03:06.0-scsi-2:0:0:0
ssd_1 pci-0000:00:11.0-scsi-2:0:0:0

zpool create storagepool raidz 1tb_1 1tb_2 1tb_3
zpool attach storagepool cache ssd_1

zfs create storagepool/pool

# Install iscsitarget to point our xenserver cluster at
apt-get install iscsitarget iscsitarget-dkms

# and create a 500gb backing volume
zfs create -V 500G storagepool/iscsivol01

vi /etc/iet/ietd.conf
Target iqn.2012-01.local.icongroup.icon-szfs01:iscsivol01
Alias iscsivol01
Lun 0 Path=/dev/storagepool/iscsivol01,Type=fileio,ScsiId=2012110201,ScsiSN=2012110201

vi /etc/default/iscsitarget
ISCSITARGET_ENABLE=true

# also remember to set targets.allow and initiators.allow as needed

/etc/init.d/iscsitarget restart

# all good to go!

MySQL Multi-Master Replication Setup

So we have a bunch of websites for different markets, running wordpress, which we would ideally like hosted in their home market. BUT, we want to be able to fail them over to a different country, should the servers in their country go down. Failover in a MySQL master-slave relationship is always a bit of a pain (as it is with any DB engine) – once you’ve failed, you really can’t ‘go back’ to the original master, until you’ve re-synced it all. Which isn’t overly easy when you only have a 1 hour per 24-hr-period maintenance window, across all the markets your company operates in.

Enter MySQL Multi-Master replication. Make a change on on server? It appears on the other. Make a change on the other server? It appears on the first!

The way this works, is that each MySQL server can be both a Master, AND a Slave. So Take two servers, A and B. Any changes made on A are played via logs to the B server. Similarly, any changes on the B server are pushed to the A server. Well, actually it’s a little more than that, as Server A will send the updates it receives from Server B, on to Server B. Why does it do this?

Well, we might have six Masters! Going Master A -> Master B -> Master C -> Master D -> Master E -> Master F. And Master F is feeding Master A. All a nice big circle. So when you make a change on Master B, it propagates to C, D, E, F, and then to A. AND back to B. But B knows not to replicate its own changes on again, and they stop there.

It’s easiest to set this all up with fresh, clean, servers.

I installed MySQL-server on two clean Debian VM’s, one in Australia, one in Ireland.

Configuration

Server A – /etc/my.cnf
Add the following:

[mysqld]
# ... other configuration, tuning, etc ...
server-id = 10
# Make sure this partition has space to log bin, relay and whatever else!
log-bin = /var/lib/mysql/bin.log
relay-log = /var/lib/mysql/slave-relay.log
relay-log-index = /var/lib/mysql/slave-relay-log.index
# Creating some room between pk ids, we can always manually insert if need be.
auto_increment_increment = 10
auto_increment_offset = 1
# This is the default, but let's be safe and ensure it's on
replicate-same-server-id = FALSE
# Want more slaves in the future with writes going to both masters?
log-slave-updates = TRUE
# If there's a reboot, let's not auto start replication. - we need to make sure of where we are, and start it manually..
skip-slave-start = TRUE

Server B – /etc/my.cnf

[mysqld]
# ... other configuration, tuning, etc ...
server-id = 11
log-bin = /var/lib/mysql/bin.log
relay-log = /var/lib/mysql/slave-relay.log
relay-log-index = /var/lib/mysql/slave-relay-log.index
auto_increment_increment = 10
auto_increment_offset = 2
replicate-same-server-id = FALSE
log-slave-updates = TRUE
skip-slave-start = TRUE

You could add multiple more servers here, just increment the server-id, and the auto_increment_offset.

Starting Replication

To start replication, we first need to create a replication user on both servers, then setup the replication attributes.

First, create the replication user on both servers:
Server A

# mysql -u root -p
mysql> GRANT REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO repluser@'serverb.ip.address' IDENTIFIED BY 'replpassword';

Server B

# mysql -u root -p
mysql> GRANT REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO repluser@'servera.ip.address' IDENTIFIED BY 'replpassword';

Find the master info on Server B:

mysql> show master status;
+------------+----------+--------------+------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+------------+----------+--------------+------------------+
| bin.000001 | 294 | | |
+------------+----------+--------------+------------------+
1 row in set (0.00 sec)

Now we want to start the replication on Server A, using the info from Server B:

mysql>CHANGE MASTER TO
MASTER_HOST='92.1.1.1',
MASTER_USER='repluser',
MASTER_PASSWORD='replpassword',
MASTER_LOG_FILE='bin.000001',
MASTER_LOG_POS=294;
mysql>start slave;
mysql>show slave status\G

You may need to run the show slave status\G a few times, before the slave drops into the standard ‘Waiting for master to send event’ state.

Once this is done, you can then work on repeating this process to start Server B slaving from Server A.

Find the master info on Server A:

mysql> show master status;
+------------+----------+--------------+------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+------------+----------+--------------+------------------+
| bin.000001 | 293 | | |
+------------+----------+--------------+------------------+
1 row in set (0.00 sec)

Now we want to start the replication on Server A, using the info from Server B:

mysql>CHANGE MASTER TO
MASTER_HOST='202.62.1.1',
MASTER_USER='repluser',
MASTER_PASSWORD='replpassword',
MASTER_LOG_FILE='bin.000001',
MASTER_LOG_POS=293;
mysql>start slave;
mysql>show slave status\G

Exporting/Importing the Data

Now you want to create any databases, users, grants, and then import any data you want. Keep an eye on the ‘show slave status\G’ on the server opposite to where you’re doing all this, to make sure it is replicating correctly 🙂

And you’re done!

Enabling SNMP on Xenserver 6

I spent an hour this afternoon cleaning up our OpManager install, making sure the new servers in the NZ office are monitored correctly.

And then I realised that we don’t have SNMP enabled on the XenServer hosts over there.

Turns out it’s not too hard though:

Edit /etc/sysconfig/iptables

Add in amongst the other allows:

-A RH-Firewall-1-INPUT -m state –state NEW -m udp -p udp –dport 161 -j ACCEPT

# service iptables restart

Edit /etc/snmp/snmpd.conf (aka change public to something more secure)

# service snmpd restart
# chkconfig snmpd on

Yup, that was easy 🙂