Partitioning!

Today I learned from the awesome bob how to add storage to a box, including partitioning, mounting, and other things I'd heard of but had never done myself.

We had to add some storage to the elasticsearch boxes. We generate a lot of logs and they all go to elasticsearch. By default when we provision servers we give them 32GB but this wasn't enough. So here's how to do it:

1. Add the extra disk through our virtualisation platform's control panel.
2. Reboot the box so that it sees the new disk.
3. Partition the new disk. There are many ways to partition a disk - we used cfdisk, which has a GUI.
   cfdisk /dev/sdb
   This opens the GUI and then you tab through it to select options. We just accepted all the defaults, so the only time we had to change it was at the end, to select [write]. We did consider making the type Linux LVM as this would have meant we could extend it later, but in this case we just accepted the default type of Linux.
   This means that as well as /dev/sdb we have /dev/sdb1. (Extra learning points here - disks are added alphabetically. So the existing disk was sda. If, subsequent to this, we add another disk, it will be sdc.)
4. Before we can mount the disk we need to give it a filesystem. We gave it ext4 because "everything uses ext4 these days" (bob). If you don't have a good reason for picking one of the others, then use ext4.
   mkfs.ext4 /dev/sdb1
5. Now we mount the new disk to a temporary directory:
   mkdir /mnt/tmp
   mount /dev/sdb1 /mnt/tmp
6. At this point we need to stop elasticsearch and also disable puppet (otherwise it will just start elasticsearch again on its next run):
   service elasticsearch-logging stop
   puppet agent --disable
7. Now we need to copy the elasticsearch indices to the new disk. Because the indices are just files you can rsync them, as long as elasticsearch has been stopped, which it has.
   rsync -av --stats --progress /mnt/elasticsearch/* /mnt/tmp
   The -v flag above is 'verbose', and in addition, the stats and progress options give some more noisy output while the rsync is going on. Often, when using rsync you would want to add -z, but since this is just from one disk to another and not going over the wire then there's no need here. Finally, -a is 'archive'. As the man page helpfully notes, this is "same as -rlptgoD (no -H)". I will leave that as an exercise for the reader.
   This stage takes some time.
8. When it's finished, we did two things, just as a (slightly paranoid) check to see if it worked:
   1. Run the same command again. If it has successfully rsynched everything, the two will be the same so it will output immediately.
   2. du -sh to check that /mnt/elasticsearch and /mnt/tmp are the same size.
9. Instead of mounting the new disk in the same way as step 5 above, we can edit /etc/fstab. We add this line:
   /dev/sdb1 /mnt/elasticsearch ext4 defaults 0 0
   We should have thought about using Puppet here, as doing it like this means there are parts of the infrastructure that are not code.
10. Next, we unmount sdb1 from /mnt/tmp:
    umount /mnt/tmp
11. Then, we move the old elastic search directory and create a new one:
    mv /mnt/elasticsearch /mnt/elasticsearch-old
    mkdir /mnt/elasticsearch
12. Now mount the sdb1 to the new elasticsearch. Because of step 9 above, we can pass either the mount directory or the directory to be mounted; we do not need both. This has the added advantage of testing our fstab setup.
    mount /mnt/elasticsearch
13. All done! Now we need to:
    1. Restart elastic search
    2. Re-enable puppet
    3. Delete /mnt/tmp and /mnt/elasticsearch-old