Monthly Archives: September 2015

Uncategorised, VMWare

Automated VMWare ESXI Backup

Leave a comment

Please note, this gives all the apearance of working then fails after 24 hours or so. It seems there are some funnies with the way / is handled and more investigation is needed.

ESXI has a very limited shell, some things are missing, others arent where they should be. It also seems that the SSH implementation is somehow broken. I’m not sure how so we are left with doing this as a two step backups. The ESXI server makes the backup and a remote host sucks it off (fnarr fnarr)

ESXI is a VERY tightly controlled environment, and resource use is critical. We will be keeping it simple for this procedure.

First up you will need to enable SSH, this is covered here

Start up a vSphere session to this server, select the server itself and click the configuration tab. Click security profile. You can also enable the SSH service here. Now click properties on the far right. In the list find SSH client and check the box. OK this and come out. You should now be able to SSH from the ESXI server.

Now to make our key. You wont find ssh-keygen in the path. We also have no home dir so this is a bit awkward too.

/usr/lib/vmware/openssh/bin/ssh-keygen

When prompted save the key to /.ssh/id_rsa.pub and enter for no password on both counts.

The command we used for the other backups will work here, it just needs some modification…

ssh <yourserverip> mkdir -p .ssh && cat /.ssh/id_rsa.pub | ssh <yourserverip> ‘cat >> .ssh/authorized_keys’ && ssh <yourserverip> chmod -R 700 .ssh

Test it, you should now have the ability to login with no password to your server. This only sadly works for root 🙁

Now, backups, this bit sucks. There are varying reports of this being possible/not possible without stopping the host. Turns out its entireley possible but you need the space to do it. It may actually be worth having a backup disk as an empty filestore for this. as we are going to have to essentiall clone the VM to do this.

Grab the script from here and get it onto your ESXI machine, you can SCP it over now. You’ll want to copy the rsync binary over now too which you can find here. Chuck it in /bin and make sure you rename it to rsync and chmod it. Same with the script and call it backupvm.sh

We are going to edit the script a little just to make our life easier. This means we have just one script to run and the CRON job can then take the VM name as an argument.

Edit lines 13 and 16 to fit your host, then change  line 29:

BACKUP_APPEND=$(date +“%Y%m%d-%H%M%S”)

to
BACKUP_APPEND=“”
and line 99:
tar –czpf “$BACKUP_ROOT/$MACHINE-$BACKUP_APPEND.tgz” “$BACKUP_PATH”

to
tar –czpf “$BACKUP_ROOT/$MACHINE.tgz” “$BACKUP_PATH”
Our backup is no longer a moving target, this makes our job easier. Run it and make sure all is well, you may want to create a small VM to test it. I had a Win98 vme hand and thus
backupvm win98
This is undoubtedly something were a faster server will help. Its definately something to run after hours. A G2030 takes about 2 mins to do a 4Gb VM however as another plus the resultant filesize is smaller than the raw VM.ls As an aside, the original script didnt remove its temporary folder, after the changes it now does.
Now, time to test rsync. I have a tgz file here called win98.tgz and I’ve a folder on my remote server of /backups/vm/win98.
rsync -avz -e “ssh” –progress win98.tgz root@<yourserver>:/backups/vm/win98
Should do the trick. Let it run and make sure the file has indeed gone over. Re-running the command should result in rsync coming back without doing an upload, the files are consistant. Edit the backup script again and add the following at line 32
BACKUPSERVER=”<yourserverip>”
REMOTEPATH=”<yourbackuppath>”
At what is now line 107 under  echo “removed temp files.” add
echo ” Starting server backup “
/bin/rsync -avz -e “ssh” –progress $BACKUP_ROOT/$MACHINE.tgz root@$BACKUPSERVER:$REMOTEPATH/$MACHINE/$MACHINE.tgz
                        if [ “$?” -eq “0” ]
                                then
                                        echo ” Sync done, deleting local file “
                                        rm $BACKUPROOT/$MACHINE.tgz
                                        logger Backup of $MACHINE to $BACKUPSERVER completed sucesfully
                                else
                                        logger Backup of $MACHINE to $BACKUPSERVER failed!
                                        echo ” Sync failed!     “
                                        echo ” local file NOT deleted”
                                fi
And test it again….
What we are doing now is using your new parameters and the existing one to build the rsync command line. So we have added to the end of the script another step that actually does the backup. All you need do is make sure that the server has a folder for the backup to drop in. If the sync fails the backup file will be left locally. Watch this as a failed sync could cause your storage to vanish as the local copies build up. you may want to just drop the file anyway.
Time for CRON, this should be easy…guess what? 🙂 You cant edit the crontab root file. Here is the official method from VMWare but it doesnt work, the file cant be written to no matter what you do, changes are also not persistent anyhow.
To keep things simple we are going to run our backups from a script. Create /backups.sh and pop the following in there
#!/bin/sh
#
# Backup script
#
/bin/logger Backups starting…
now add each vm you’ll be backing up, this will make them run sequentially
/bin/backupvm.sh <vmname>
Save the file and chmod it. test it if you feel the need. We will be calling this from cron. Now there is no need to do it this way, you could create individual cron jobs for each machine, batch them up, whatever you feel. This is justa  simple way to sequentially do it.
Its worth at this point, benchmarking your backups. Some larger VMs can take a LONG time to complete and transfer and you want to schedule things so you dont get overlaps. Using the method below with one cron job then a backup file will avoid this but its still worth doing so you can get an idea when to start them so they actually finish out of hours.
Now dealing with cron. The crontab lives in /var/spool/cron/crontabs however there are two issues. Firstly, this file is trashed on every boot. Secondly, its not even a real file, you cant actually edit it. You can copy it, edit it then copy it back though. So kludgey as it is, thats what we will do here.  Before we go further you need to know about how to schedule cron jobs.  This Link should help you out. We are going to run this job at 1am every other day from the second day of the month (even days). The schedule we need for cron is  0  1  2-30/2  *  * /backups.sh. Create a new script /addbackupjob.sh and pop in
#!/bin/sh
#
# Script to add the backup job
#
cp /var/spool/cron/crontabs/root /var/spool/cron/crontabs/root.tmp
cp /var/spool/cron/crontabs/root /var/spool/cron/crontabs/root.bak
 echo “0  1    2-30/2   *   *   /backups.sh” >> root.tmp
rm /var/spool/cron/crontabs/root
cp /var/spool/cron/crontabs/root.tmp /var/spool/cron/crontabs/root
 kill $(cat /var/run/crond.pid) && crond
rm /var/spool/cron/crontabs/root.temp
run this and cat /var/spool/cron/crontabs/root and make sure this has done as it should/ it’ll leave a backup file behind just in case.  This link details how to run a script an boot. We need to add /addbackup.sh to /etc/rc.local.d/local.sh. open the file and add it just above the last line, exit 0.
That *should* be it. The cron job will be readded on every reboot.
You can use /bin/logger in the script if you’d like to write to the syslog.
*** UPDATE ***
If you have multiple datastores, this isnt going to work for you. However with some changes it can be made to.
Edit your ./backupvm.sh and make the following changes:
Linux

Auto Remote Backups PFSense, FreeNAS, FreePBX – DIY Cloudy goodness

2 Comments

Well its been a while but it has been manic here. However after some faffing with getting some backups working I thought I’d pop this up here as it covers a fair bit.

A customer wants to sync their office server to one of our so their engineers can get to CAD drawings. Now the basics of getting everyone up using SCP et all are pretty simple so I wont go into those. However the sync was problematic. The server is a Windows 2012 server and we popped on cwRsync and Syncovery. No way could we get Rsync to play ball however in the end we made SCP play. This isnt ideal but we will come back to it.

Next up was to put this server to good use and backup various other things. We do have a large number of PFSense, freePBX and Freenas boxen about the place and I thought it would be nice to make these work.

Now, key to this is making SSH work without passwords. You’ll find pretty quickly this isnt exactly non trivial in fact its a royal PITA. Most of the gudes arent quite there or dont work quite right. After a lot of digging I got there with a really simple and elegant solution.

So we need to pair up our client (in this example a PF box but this works on FreeNAS too and *should* work on FreePBX).

Create a user for the backup, using root isnt a good idea at all, remeber this box will have direct access to your server and if you use root and your customer decides to tinker they could in theory do a LOT of damage. Set up a user to use for the backup, we are going to create a backup user called ‘companya’ so on your server do:

useradd companya
passwd companya

You can do the same on the system to be backed up. Its much nicer if it all matches. You can assign multiple keys to an account so you dont need to do this again on the server end. Setting up home dirs is covered elsewhere so I wont go into that. I’ve created a structure in the home dir called backups and in t here used hostnames to identify the device. If you are backing up multiple customers you really should make sure they cant get to eachothers files.

On PFSense you need to do this through the user interface. Make sure that the user’s permissions include shell access else the permissions and default shell for that user will get nuked every reboot.

Now this is where it tends to go a bit runny. We are going to use SSH so we need SSH to be able to log in with just an rsa key. This *should* be easy, however getting FreeNAS to play was a pig. Turns out there is a magic command. So, on the system to be backed up we need to become our new user and make a key…

su companya
ssh-keygen -t rsa

*NOTE: DO NOT SET A PASSWORD!* Now this key has to go into the list of authorised keys at the other end. There seem to be billions of ways to do this but they all gave varying amounts of success. Now provided you created a user on the server and accept the defaults for ssh-keygen the following will do the trick…

ssh <yourserver> mkdir -p .ssh && cat ~/.ssh/id_rsa.pub | ssh <yourserver> ‘cat >> .ssh/authorized_keys’ && ssh <yourserver> chmod -R 700 .ssh

You should be asked for the password for companya twice. This will make the .ssh dir, upload they key and make sure its CHMOD’ed correctly. Note I have seen references to chmoding the authorized_keys file to 600 if you see errors on your /var/log/auth or /var/log/security

Now if all has gone well. you can do:

ssh <serverip>

If you get asked for a password you’ll have to go chase down why. The log files above may help but the usual cause is permissions. Type exit to bring you out of the SU and you are done here.

So now, to backups….

FreeNAS

You can now enable Rsync on your server which is explained elsewhere then use the Rsync Tasks under system to create backup jobs. Make sure you use your new user to do this else nothing will work. Its that simple.

To back up the config requires something a little like the PFSense solution below. As we have Rsync already running over SSH we may as well use that.

Make yourself a directory on a visible share for your backup, you could use this to aggregate a number of backups at this point. As we will be using rsync only changes will be backed up so this may be a good point to put router backups etc. For this we have a volume called adminnas and I’ve created a folder called sysbackups.

Now we need to make a file to DO the backup. It seems sensible to drop this in the backup folder too so things are kept neat. This means less work if the main OS partition goes AWOL.
#!/bin/bash
# Freenas backup script by R.Inskip
# This maintains two copies of the FreeNAS backup file
#
# Change to suit your config
#
SOURCE=/data/freenas-v1.db
TARGETDIR=/mnt/adminnas/sysbackups
TARGETFILE=adminnas.db
BACKUPFILE=adminnas.old.db
#
# If there is an old backup, remove it

[ -f $TARGETDIR/$BACKUPFILE ] && rm $TARGETDIR/$BACKUPFILE
#
# if there is a backup here already rename it
#
[ -f $TARGETDIR/$TARGETFILE ] && mv $TARGETDIR/$TARGETFILE $TARGETDIR/$BACKUPFILE
cp $SOURCE $TARGETDIR/$TARGETFILE

make the changes to reflect your setup and save the script. Dont forget to CHMOD it!

And off to cron now.  Add a cron job, the timings are up to you. Your Rsync job should tie up with this however you could have this backup more often so you keep a local copy and then pickup the backups with Rsync less frequently, its totally up to you. Bear in mind that once your setup is stable FreeNAS tends to get left alone, frequent backups may be counter productive and you can always go and manually run your script if you are playing about. Your cron job needs to call the script you just saved. Set it to just ahead of the system clock to make sure all works.

All being well you can now setup your Rsync to drop it into the right place, et voila!

PFSense

First up make your backup script. We will be using SCP so make sure that your backup user can do passwordless logins.

Create a backup script, dropping this in root is just fine. We used:

#!/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
now=$(date +”%T”)
logger Automatic backup running at $now
/usr/bin/su <BACKUPUSER> -c “/usr/bin/scp /conf/config.xml <backupuser>@someserver.domain.com:/home/<backupuser>/backups/pfsense

The script above drops a file in the system log to say when the backup last ran. Make sure you change this to reflect your system and save it as backup.sh, chmod it and them make sure you can run it on the command line and it works.

You need to muck with Cron. The easiest way is to install the Beta CRON module and do it there. The command line you want to do the backup for cron is…

/root/backup.sh

Obviously you’ll need to change <youruser>,<yourserver> and <backupdir>. You can manually add it to /etc/cronttab by adding…

00      01      *       *       *       root    /root/backup.sh

Again make sure you correct to reflect your setup. This job is set to run daily at 01:00.

FreePBX

This one is is a little bit more convoluted. There is a backup UI in FreePBX but it doesnt handle the setup of the SSH side, expecting you to have done this. So assuming you’ve done the steps above on your FreePBX system you should be ok.

If you are going to use root or asterisk as your backup user thats all great and you can use the ui. However in this scenario I dont want that. I want customer backups to go to the right customer folder. This is a little harder than it needs to be but its not difficualt once you know what is going on. All scripts are run as the ‘asterisk’ user. This cant see the key we have made so we need to copy it. To really add to the fun FreePBX mucks with the permissions of asterisk’s .ssh directory on reboot so whil you may get this working first time, it wont after a reboot. Moving the key elsewhere fixes this. You will have to do this as root…

mkdir /home/asterisk
cp /home/backupuser/.ssh/id_rsa /home/asterisk/backup_rsa
chown -R asterisk /home/asterisk
chgrp -R asterisk /home/asterisk
chmod 700 /home/asterisk
chmod 600 /home/asterisk/backup_rsa

You should now be able to setup a backup server in the FreePBX backup/restore dialog. So for our example….

Untitled

again make sure you change paths etc where needed.