Playing with GlusterFS

Posted on: 2017-04-15

GlusterFS is a really cool software defined storage solution. It allows you to do RAID like setups using disks in different servers. It is MUCH cheaper, more flexible and sometimes more performant than the typical EMC or Netapp + shelves setup.

This post will setup two shares. A "RAID0" like distributed share where disk capacity from two servers is added together and represented as one big export with no redundancy and a RAID1 like setup where disks from two servers are mirrored. We will also go over how to use these shares on a client system.

Note that this stuff isn't even scratching the surface of this product. You can do mixed types of volume setups, replicate across data centers for DR, offer shares via samba and more. I encourage you to explore further.

Start by setting up 2 minimal CentOS 7 VMs with static IPs. Each VM should have 3 disks. vda for OS. vdb and vdc should be 10GB blanks used for gluster.

On both VMs: format the drives, install the software, disable firewalls, and configure /etc/hosts using your info

fdisk /dev/vdb
p (keep default)
1 (keep default)
2048 (keep default)
20971519 (keep default)

fdisk /dev/vdc
p (keep default)
1 (keep default)
2048 (keep default)
20971519 (keep default)

mkfs.xfs -i size=512 /dev/vdb1
mkfs.xfs -i size=512 /dev/vdc1
mkdir -p /data/brick1 /data/brick2
echo '/dev/vdb1 /data/brick1 xfs defaults 1 2' >> /etc/fstab
echo '/dev/vdc1 /data/brick2 xfs defaults 1 2' >> /etc/fstab
mount -a
df -h

mkdir /data/brick1/repl0
mkdir /data/brick2/dist0

yum -y update
yum -y install centos-release-gluster39 vim
yum -y install glusterfs-server
systemctl start glusterd 
systemctl enable glusterd

systemctl disable firewalld
systemctl stop firewalld

vim /etc/hosts gs1.lan gs2.lan

Now that the servers are configured we need them to talk to each other.

On gs1.lan:
gluster peer probe gs2.lan

On gs2.lan
gluster peer probe gs1.lan

Now lets create the replicated RAID1 like volume. On either VM.

gluster volume create repl0 replica 2 transport tcp gs1.lan:/data/brick1/repl0 gs2.lan:/data/brick1/repl0
gluster volume start repl0
gluster volume info repl0

Now lets create the distributed RAID0 like volume. On either VM.

gluster volume create dist0 gs1.lan:/data/brick2/dist0 gs2.lan:/data/brick2/dist0
gluster volume start dist0
gluster volume info dist0

Finally, lets pretend gs1.lan is a client and mount the shares there.

mkdir -p /mnt/dist0 /mnt/repl0
mount -t glusterfs gs1.lan:/dist0 /mnt/dist0
mount -t glusterfs gs1.lan:/repl0 /mnt/repl0

Checking it

df -h
for i in `seq -w 1 100`; do cp /etc/hosts /mnt/dist0/hosts-$i; done
for i in `seq -w 1 100`; do cp /etc/hosts /mnt/repl0/hosts-$i; done
#on both nodes do the following and look at file distribution
ls -lha /data/brick1/repl0 /data/brick2/dist0

Bonus: Mount a share via NFS

mkdir -p /mnt/dist0_nfs
gluster volume set dist0 nfs.disable false
mount -t nfs -o vers=3 gs1.lan:/dist0 /mnt/dist0_nfs
ls -lha /mnt/dist0_nfs