Category Archives: Linux

Linux related ramblings

Linux

Auto Remote Backups PFSense, FreeNAS, FreePBX – DIY Cloudy goodness

2 Comments

Well its been a while but it has been manic here. However after some faffing with getting some backups working I thought I’d pop this up here as it covers a fair bit.

A customer wants to sync their office server to one of our so their engineers can get to CAD drawings. Now the basics of getting everyone up using SCP et all are pretty simple so I wont go into those. However the sync was problematic. The server is a Windows 2012 server and we popped on cwRsync and Syncovery. No way could we get Rsync to play ball however in the end we made SCP play. This isnt ideal but we will come back to it.

Next up was to put this server to good use and backup various other things. We do have a large number of PFSense, freePBX and Freenas boxen about the place and I thought it would be nice to make these work.

Now, key to this is making SSH work without passwords. You’ll find pretty quickly this isnt exactly non trivial in fact its a royal PITA. Most of the gudes arent quite there or dont work quite right. After a lot of digging I got there with a really simple and elegant solution.

So we need to pair up our client (in this example a PF box but this works on FreeNAS too and *should* work on FreePBX).

Create a user for the backup, using root isnt a good idea at all, remeber this box will have direct access to your server and if you use root and your customer decides to tinker they could in theory do a LOT of damage. Set up a user to use for the backup, we are going to create a backup user called ‘companya’ so on your server do:

useradd companya
passwd companya

You can do the same on the system to be backed up. Its much nicer if it all matches. You can assign multiple keys to an account so you dont need to do this again on the server end. Setting up home dirs is covered elsewhere so I wont go into that. I’ve created a structure in the home dir called backups and in t here used hostnames to identify the device. If you are backing up multiple customers you really should make sure they cant get to eachothers files.

On PFSense you need to do this through the user interface. Make sure that the user’s permissions include shell access else the permissions and default shell for that user will get nuked every reboot.

Now this is where it tends to go a bit runny. We are going to use SSH so we need SSH to be able to log in with just an rsa key. This *should* be easy, however getting FreeNAS to play was a pig. Turns out there is a magic command. So, on the system to be backed up we need to become our new user and make a key…

su companya
ssh-keygen -t rsa

*NOTE: DO NOT SET A PASSWORD!* Now this key has to go into the list of authorised keys at the other end. There seem to be billions of ways to do this but they all gave varying amounts of success. Now provided you created a user on the server and accept the defaults for ssh-keygen the following will do the trick…

ssh <yourserver> mkdir -p .ssh && cat ~/.ssh/id_rsa.pub | ssh <yourserver> ‘cat >> .ssh/authorized_keys’ && ssh <yourserver> chmod -R 700 .ssh

You should be asked for the password for companya twice. This will make the .ssh dir, upload they key and make sure its CHMOD’ed correctly. Note I have seen references to chmoding the authorized_keys file to 600 if you see errors on your /var/log/auth or /var/log/security

Now if all has gone well. you can do:

ssh <serverip>

If you get asked for a password you’ll have to go chase down why. The log files above may help but the usual cause is permissions. Type exit to bring you out of the SU and you are done here.

So now, to backups….

FreeNAS

You can now enable Rsync on your server which is explained elsewhere then use the Rsync Tasks under system to create backup jobs. Make sure you use your new user to do this else nothing will work. Its that simple.

To back up the config requires something a little like the PFSense solution below. As we have Rsync already running over SSH we may as well use that.

Make yourself a directory on a visible share for your backup, you could use this to aggregate a number of backups at this point. As we will be using rsync only changes will be backed up so this may be a good point to put router backups etc. For this we have a volume called adminnas and I’ve created a folder called sysbackups.

Now we need to make a file to DO the backup. It seems sensible to drop this in the backup folder too so things are kept neat. This means less work if the main OS partition goes AWOL.
#!/bin/bash
# Freenas backup script by R.Inskip
# This maintains two copies of the FreeNAS backup file
#
# Change to suit your config
#
SOURCE=/data/freenas-v1.db
TARGETDIR=/mnt/adminnas/sysbackups
TARGETFILE=adminnas.db
BACKUPFILE=adminnas.old.db
#
# If there is an old backup, remove it

[ -f $TARGETDIR/$BACKUPFILE ] && rm $TARGETDIR/$BACKUPFILE
#
# if there is a backup here already rename it
#
[ -f $TARGETDIR/$TARGETFILE ] && mv $TARGETDIR/$TARGETFILE $TARGETDIR/$BACKUPFILE
cp $SOURCE $TARGETDIR/$TARGETFILE

make the changes to reflect your setup and save the script. Dont forget to CHMOD it!

And off to cron now.  Add a cron job, the timings are up to you. Your Rsync job should tie up with this however you could have this backup more often so you keep a local copy and then pickup the backups with Rsync less frequently, its totally up to you. Bear in mind that once your setup is stable FreeNAS tends to get left alone, frequent backups may be counter productive and you can always go and manually run your script if you are playing about. Your cron job needs to call the script you just saved. Set it to just ahead of the system clock to make sure all works.

All being well you can now setup your Rsync to drop it into the right place, et voila!

PFSense

First up make your backup script. We will be using SCP so make sure that your backup user can do passwordless logins.

Create a backup script, dropping this in root is just fine. We used:

#!/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
now=$(date +”%T”)
logger Automatic backup running at $now
/usr/bin/su <BACKUPUSER> -c “/usr/bin/scp /conf/config.xml <backupuser>@someserver.domain.com:/home/<backupuser>/backups/pfsense

The script above drops a file in the system log to say when the backup last ran. Make sure you change this to reflect your system and save it as backup.sh, chmod it and them make sure you can run it on the command line and it works.

You need to muck with Cron. The easiest way is to install the Beta CRON module and do it there. The command line you want to do the backup for cron is…

/root/backup.sh

Obviously you’ll need to change <youruser>,<yourserver> and <backupdir>. You can manually add it to /etc/cronttab by adding…

00      01      *       *       *       root    /root/backup.sh

Again make sure you correct to reflect your setup. This job is set to run daily at 01:00.

FreePBX

This one is is a little bit more convoluted. There is a backup UI in FreePBX but it doesnt handle the setup of the SSH side, expecting you to have done this. So assuming you’ve done the steps above on your FreePBX system you should be ok.

If you are going to use root or asterisk as your backup user thats all great and you can use the ui. However in this scenario I dont want that. I want customer backups to go to the right customer folder. This is a little harder than it needs to be but its not difficualt once you know what is going on. All scripts are run as the ‘asterisk’ user. This cant see the key we have made so we need to copy it. To really add to the fun FreePBX mucks with the permissions of asterisk’s .ssh directory on reboot so whil you may get this working first time, it wont after a reboot. Moving the key elsewhere fixes this. You will have to do this as root…

mkdir /home/asterisk
cp /home/backupuser/.ssh/id_rsa /home/asterisk/backup_rsa
chown -R asterisk /home/asterisk
chgrp -R asterisk /home/asterisk
chmod 700 /home/asterisk
chmod 600 /home/asterisk/backup_rsa

You should now be able to setup a backup server in the FreePBX backup/restore dialog. So for our example….

Untitled

again make sure you change paths etc where needed.

 

Linux, Personal Musings, Raspberry Pi

Security Through Obscurity…

Leave a comment

Time and time again we are told the above doesnt work and yet it still happens, why and why is it important.

In the process of our big tracker project I’ve been looking at packages that do the job already and also a number of control systems. Bearing in mins what these do, there is no security on most of it and token security on one of them. You may think that this isnt a big thing and in most cases it isnt, but there are good reasons it is. So lets take a look at two culprits…

Veichle control Systems:
I’ve looked at three of these in the last few days and one thing is common with all three. No encryption, no ability to detect errors and no authentication. For the best part this isnt a huge problem, in fact its a boon, I know how all three work and can talk to them. However the bar is set low and any of them can be subverted easilly. In the case of the first two I looked out you wouldnt gain a lot past the ability to turn things on and off but the potential is there. The third is a fully integrated system and is designed to control all aspects of the vehicle including battery management and its also designed to be linked to an onboard computer, now here there is scope for trouble. The system concerned could easilly be subverted by malware and then the fun begins. Further digging on this one turns up no encryption is used on the comms link to the office either, but it gets worse…

Of the dispatch and tracking systems looked at not one uses encryption, for the more basic ones this means simply you can rap the tracking data out of the air and at least one was fooled easilly to connect to a wifi network and forwd the unencrypted traffic over my network, patient details, addresses etc all in the clear. This means that this data can also be modified so this particular system couldbe subverted pretty easilly with a laptop and either data creamed off and used for nefarious means (ask someone how expensive ambulance kit is!) or worse. Tracking data can be falsified, fed back to the base meaning the veichle doesnt show as being where it belongs and uh-oh, the SOS button isnt going to help if someone jumps the ‘bus’

There is no excuse for this. None at all and the scary thing is that the same can be found in Automotive systems from the factory and has recently been found in Avionics systems too!

Serial channels can simply and effectively be secured. Even our most basic bus system, TRNet uses authentication and device tracking mechanisms. If a device doesnt authenticate, its ignored, even if it does it must be recognised on the bus else the system, will lock the bus out. This is done by a few dozen lines of code and although it could be subverted with a moderate amoun of effort you’d need access to a lot of information to get started. This also allows us to give a mechanism for making sure the bus is reliable. Scarilly all but one of the three systems we looked at just kept sending the same data over and over again with no control path back to the master. Although it’ll work it means bit flips can happen in transmission and may be ignored (no CRC/Checksum used) and there is a wealth of data for someone to grab and decode.

Data channels running over a network should also be secured and only links trusted should be used. Again this is basic common sense and doesnt take a lot of code, for this application we are using a pretty hardcore authentication system based on AES but its neither difficult to do or expensive. With the system we have both ends of the chain must be compromised and in such a way that pe-encrypted data is grabbed, key shifts mean that just grabbing the encrypted stream and a session key wont help you, even if you do grab a key you need the previous key to use it so good luck.

There is still a strong sense in the embedded and bespoke software/hardware market that no one is interested in specialist applications, wont try and thus you dont need to bother about security above making it convoluted. This is fallacy, the value of the equipment on an ambulance is truly amazing to those not in the know so gaining the ability to send such a veichle to a false location of your choice, disabling its tracking  and ability to call for help is going to be of interest.

 

Linux, Personal Musings

Linus VS Win Server TCO – The reality

Leave a comment

While talking with a Collegue yesterday the issue of their server came up. The IT guy has, and I cant fault the logic, gone with Linux and Samba to power what is essentially a Windows Network. This would be an ideal situation except for a small number of flies in the ointment. At the most basic Level, drivers, are the first hurdle. Te Windows drivers include support for all manner of offload engines, acceleration technologies etc. The Linux drivers mostly lag way behind the Windows ones, but they are out there as a rule. Just to get the bare server up and running to snuff can take a while. So starting from a bare server we have a TCO at this point of £0 for Linux and £450 for Server 2012. Now if we run on from here assuming you just install and go you’ll already be having problems. You’ll be seeing performance hits on Network and RAID hardware, easilly fixed with drivers however we are now talking a good hour or two fiddling and faffing with the Manufacturer Linux drivers (if they exist) Bar just running a CD under Server 2012. Say your time is worth £50 an hour and the drivers arent quite right and thus a build from scratch is needed, two hours, we stand at £100/£450. From experienc 4/5 hours is closer to reality as there will be missing things you dont find. On top of that we now have to get Samaba working on this server, Domain config with Samaba isnt fun and the documentation is poor,  we can easilly have blown a day by now so thats 8 hours or £400 vs £450 TCO on the software side. Now heres the nasty bit. That Linux server could EASILLY take you up to a day to get working to a useable extent. The Windows server, from bare metal to running a domain, 2 hours tops. being far we will allow for that two hours so £400/£550. SO excellent you saved £100, probobly shaved a few days off your life too. Only its not done. As a rule AD and Windows *just* works at the basic level. You’ll now be looking at 5 mins per machine to connect the PCs.  Samaba, prepare for permission and authentication hell, possibly on a per PC basis. In short unless you are a Samba God you wont get it right first time and file permissions are the usual cause.  In short, the Windwos Side could be finished in 4/5 hours total from start to users loging in with Roaming profiles and you could Probobly have exchange going too in that time. With Samba you may end the day at ‘Almost works’ and I’ll fix the niggles tomorrow. This is a ‘bit of string’ situation but already there is a disparity here. The Windows server is done and working. I’ve gone home and am Reasonably happy about a productive day, the Samba installer is probobly not having such a good day.

Now lets look at he basics for what we might want to do with this server.

A small office I’m probobly going to want IIS for various tasks (It failry central to 2012 so not having it isnt an option TBH) SO I have a web server and a scripting engine out of the box. I have a pretty useable and simple to use user administrator and unless I install Server Core (and even then not neceserilly) I can manage users, groups, email routing, web setup, replication and indeed about 90% of basic office tasks and be done by the end of that first day.

Linux, well we’ve fixed Samba. I now have to Install Apache, almost certainly some flavour of PHP, probobly MySQL (We do on Windows anyway) and then start installing admin interfaces. Now most Distros do come with various bits and bobs there already, however you’ll find apps start asking for newer PHP versions, or extensions which need newewer (or older) versions, it can get knotty fast.

Its not unfair to say that the windows server would cost you about £800 and be done on day one. (I’m not factoring in Exchange cost, the reason being getting the same functionality for Linux is hell/impossible)

Linux wise we are at two days, so again £800 and will have a reduced level of functionality. Now theres another big trick there many miss. That extra day? Well theres your £400 but if this is a live or new setup you could be having to factor user downtime in there too so lets look at that.

Users cost less than consultants, how much yours cost is down to you but if we assume the following. You average user is on £25K There are about 250 working days per year, so thats anice easy £100 a day; for an 8 hour day thats  £12.50 and hour. Think of a small office of 10 users (Yes I’m keeping the maths simple) so thats costing you £125 an hour to pay them, now its about to go wrong…

On Monday you’ve scheduled a server upgrade, you are doing it yourself and lets say its a complete refresh and start over. Your users have been told no PC use (its a little contrived but it DOES happen in small business). So for Windows we were down for a day, thats £1000 in lost time. You know where this is going so we have £2000 on the other time. Its interesting to note that the extra £1000 would purchase a low end Windows server!

Its not hard to see the Linux Admin over a day behind. If thats so its all gone wrong now as the free server has cost heading on for twice the paid one.

Its not all gravy though, troubleshooting MS products can be an issue, debugging and logging with MS products can be lousy and that can slow down finding some issues. There is a very twisty mindset required sometimes, that said. MS will help you if you call (within defined limits)

Its all up and running and now we need to think about day to day use. Adding a new printer and sharing it can be done quite feasably in a few minutes on a Windows server and this includes making sure users get to it with no work from them, dirver issues etc. The whole thing can certainly be comfortably wrapped up pretty fast. Linux offers no such mechanism and indeed its even possible that there are no native drivers for you printer, with a USB device this could mean a lot of work.

So to sum up, think long an hard about what you want to do. If you have plenty of time and it has not associated value (You need to look at the way you use your time!), want to use older hardware, maybe only want the absolute basics of a Windows network  and have low blood pressure then go for it. There are situations it is exactly the right tool. If it *has* to work then it may be significntly cheaper to go Windows.

Now to stop the flames, we normally deploy both. In fact we often use Samba for slave/backup operations and most of our sites do have at least one dedicated Linux system. Some things *do* work much better, often the server will be providing a web backend or database services to Windows (Access can use MySQL). Right tools for the right job. I also know I’ve simplified a LOT in here, but the numbers do stack up if you sit down and work your project out on paper you’ll see similar results, at best the Linux option may well cost you as much and deployment methodology can make a big difference but its unlikeley the TCOs will ever be better than eaqal and the £0 TCO of Linux is a myth for business. Businesses rely on saving money as a rule, if Linux were a magic bullet Windows would have no market.