Here we go folks.  Complete, yet half-baked documentation on how to setup your very own home NAS using free Linux software.  I know this has been written about before, and in fact I have pulled from many fine sources (see my acknowledgement below).  This is documented here mainly so I don’t forget how I did this.  It’s not really for you – sorry internet.  Let’s get started:
More »

Was this post helpful?
  • Thanks 
  • Sort-of 
  • Not Helpful 

It happened over the weekend. I can’t precisely say when, but I know when I attempted to reboot my machine on Saturday morning, I was greeted with hard drive failure. The first sign that trouble was lurking was that my BIOS stalled for a moment, the screen went black and then printed “A problem has been detected with your Hard Drive. Press any key to continue.”. At this point, I was taken off guard and was still in denial. I pressed a key and the normal GRUB popped up and began the boot process. The familiar Ubuntu splash screen popped up and everything felt normal.

My confidence was almost restored, when the screen went to rapidly scrolling text with things like “Unable to find root inode”, etc. And so, it went. My OS was hosed. I downloaded the latest Western Digital Diagnostics CD, booted with this and discovered (through the SMART protocol) that my drive had finally given up. It had simply quit. As it turns out, as a drive begins to fail, it will attempt to remap the bad sectors to known good sectors on the disk that are allocated just for this purpose. But when you run out of good sectors to remap to, then catastrophic failures are eminent. My drive had reached this point… with really no warning whatsoever. Lesson learned – check the health of your drives regularly. Linux has some free tools to check the SMART attributes (smartmontools) that can even e-mail you when they detect a problem. And backup regularly.

On another subject, I do not “recycle” my hard drives. That is, I don’t ever re-distribute my old or broken hard drives back into the wild. Depending on the drive, I will not sell my old HD on e-bay or even give it away. There may be sensitive personal data on these drives that I don’t really want to get into the wrong hands. I have a “best-known-method” to dispose of my old and/or broken hard drives. Because I feel like sharing, I decided to film the process and post it here for your viewing pleasure. Enjoy.


UPDATE 12/8/2007: This might be a less destructive and less satisfying method.

Was this post helpful?
  • Thanks 
  • Sort-of 
  • Not Helpful 

Another techie post. For my regular readers, you can probably skip this one. Apologies, apologies in advance. I’m posting this, because I think this might be helpful for folks, just like me, who had to spend several hours on the internet looking for a tutorial on how to setup a simple RAID-1 array with mdadm in Ubuntu. Hours. Erg. And I thought it was going to be easy. I’m posting this a public service to those who want to set up a straight forward RAID-1 array AND as a reminder to myself so that I can repeat these steps in the future (if I have to).

My setup is Ubuntu 7.04 (Feisty Fawn) installed on a 500GB drive and two additional 250GB drives specifically purchased for RAID-1. RAID-1 offers me simple mirroring of my data, such that if one of my drives fails, I have a robust backup. This means that both 250GB drives will have an exact copy of one another. After reading, reading, and reading some more on the internets, I decided that using a simple software RAID-1 solution with mdadm was the best solution for me. For the newbie, this process is still a bit daunting, but it’s really not that bad.

Step 1: Install gPartEd (partition editor for gnome).

From the terminal prompt type:

sudo apt-get install gparted

Step 2: Format your new drives.

Launch gPartEd from System-> Administration menu.

Select one of your new drives from the drop down menu on the right. (mine were /dev/sda and /dev/sdc) Format both drives with ext3 (NOTE: You will lose all data on drive so be careful to back your data up).

After the format has been applied, you’ll need to right-click on the partition that you created (both drives) and click ‘Manage Flags’. Tick the ‘RAID’ option to make sure that this disk will be automatically included in the RAID array. Now close down gPartEd. There is an excellent pictorial for this operation on this site.

Use ‘sudo fdisk -l’ to list out the current partitions. You should see your freshly created partitions in the list.

Step 3: Install ‘mdadm’

From the terminal prompt type:

sudo apt-get install mdadm

There are some setup screens that pop-up on my version of mdadm install that may or may not happen for you. I haven’t seen any documentation on this on other websites. It basically asks if you want any of the drives in your RAID array to be detected early in boot. Since I’m using my two independent drives for backup purposes, and not for root or GRUB then this does not apply to me. I typed ‘none’ and continued.

Step 4: Create your RAID array with mdadm

From the terminal prompt type:

mdadm –create –verbose /dev/md0 –level=1 –raid-devices=2 /dev/sda /dev/sdc

Step 5: Check the status of the array while it synchs

From the terminal prompt type:

sudo watch cat /proc/mdstat

Press CTRL-C to exit this
or

sudo mdadm –detail /dev/md0

This will take a long time, most likely, so go make yourself a delicious sandwich. Eat that sandwich and then go and watch some TV, surf the internet or whatever you like to do to pass the time. My initial synch took around 60+ minutes.

Step 6: Add an ext3 file system to the Array

From the terminal prompt type:

mke2fs -j /dev/md0

This may seem like a repeat, but during my research it seemed necessary. The -j option creates journals, which is a new feature of the ext3 file system. This may take some time. Grab a beverage of your choice.

Step 7: Mount the array

From the terminal prompt type:

sudo mkdir /mnt/RAID
sudo mount /dev/md0 /mnt/RAID

I originally set this up to mount in ‘/usr/RAID’ but for some reason ubuntu wouldn’t mount to this location. So I switched up to ‘/mnt/RAID’ which worked like a charm. Not sure why this is. If someone can enlighten me, that’d be great.

Step 8: Make the array automatically mount on boot-up

Add the following line to the end of your ‘/etc/fstab’ file:

/dev/md0 /mnt/RAID auto defaults 0 0

If you’re not familiar with the format of the fstab file, here’s what wikipedia told me:

The columns are as follows:

1. The device name or other means of locating the partition or data source.
2. The mount point, where the data is to be attached to the file system.
3. The file system type, or the algorithm used to interpret the file system.
4. Options, including if the file system should be mounted at boot.
5. Adjusts the archiving schedule for the partition (used by dump). I don’t use this.
6. Indicates the order in which the fsck utility will scan the partitions for errors when the computer powers on. When in doubt, leave 0.

A value of zero in either of the last 2 columns disables the corresponding feature.

UPDATE (11/19/2007): After a recent catastrophic hard drive crash (ironically not on my RAID array), I had to re-install my OS and re-assemble my RAID array. This is pretty simple to do, using the existing drives you can just type the following to get it up and running again:

sudo mdadm -A /dev/md0 /dev/sda /dev/sdc

…and to confirm that it’s running and healthy:

sudo mdadm -D /dev/md0

After doing this you just need to add the line above to your /etc/fstab file, but don’t forget to update your /etc/mdadm/mdadm.conf file! After I completed the above, the file was regenerated, but I still had to add a few lines to the end by typing the following command while in the /etc/mdadm/ directory:

cp mdadm.conf mdadm.conf.`date +%y%m%d`
echo “DEVICE partitions” > mdadm.conf
echo “MAILADDR root” >> mdadm.conf
mdadm -D -s >> mdadm.conf

Step 9: Make the RAID Array R/W (this is where a lot of guides are lacking for us noobs)

From the terminal prompt type:

chmod 777 /mnt/RAID

This sets permissions for all users to write to the device. You may want to be a bit more stringent with your security settings. (775 may also be acceptable)

Automated Backups

Now to do something useful with my RAID array. Of course, my original plan was to use this for backup, so I searched for an application that could simply synch the contents of my Pictures, Videos and Music folders with my RAID backup. I stumbled upon rsync, which already comes with Ubuntu 7.04 out of the box. This piece of software actually synch’s the data between two directories (even remote ones!), and has loads of options to do so. It only copies over the new/changed files to the backup which saves a huge amount of backup time on a day-to-day basis. And it couldn’t be more simple to use.

Step 1: Using rsync

From the terminal prompt type:

sudo rsync -r -t –delete /home/ben/pictures /mnt/RAID/pictures

This will copy all of the files from my home/pictures directory to a backup on the RAID array. The -r recurses sub-directories. And if you’re really tight on space, you can even use the -z option to compress everything in the process. The –delete option will delete any files that are in the destination but not in the source. For my MP3 directory this is pretty useful since I tend to move things around often. NOTE: DO NOT FORGET THE -t option. I found that if I did not include this option, rsync will copy all the files every time. The -t option will preserve the ‘Modified Time’. This way, if no change was made, we can skip the file. Pretty neat!

I created a script in my ‘home/bin’ directory called ‘synchback’ that contains several rsync commands to backup my photos, videos, music and documents. This way, I can manually run this script whenever I want to back things up. I added the ‘–stats’ option and piped the output to a log file so that I can periodically check my backups to make sure things are running smoothly.

Step 2: Scheduling a Backup

Of course you may want to do this backup on a daily, or weekly schedule so let’s explore how to setup a cron job to make this happen. I personally want to backup on a daily basis, but only late at night when I’m not going to be bothered. So I setup a new cron by doing the following:

From the terminal prompt type:

sudo crontab -e

This opens an editor so you can edit your crontable. The cron job will be saved after you save and close the editor. Cron will load the crontab file and restart the schedule. The following shows the edit I made to the crontab file to add my scheduled task:

# m h dom mon dow command
01 04 * * * /home/ben/bin/synchback

This sets up the my script to run at 4:01AM each and every day. The table is formatted in Minutes (m), hours (h), day of the month (dom), month (mon), day of the week (dow). Using a star means that the job should occur on all instances.

Voila! Automated backup/synch to my RAID-1 array.

What if my RAID-1 array has a disk failure?

Simple, rebuild the array with mdadm. First you’ll want to type the following to get the status of your drives:

sudo mdadm –detail /dev/md0

Now, this should kick out a report and show you the failed devices. At this point, you should be able to identify the failed device and remove it from the array. Use the following sequence of commands to mark the device faulty and then remove it from the array (where sdx = your failing device):

sudo mdadm /dev/md0 -f /dev/sdx -r /dev/sdx

Go get yourself a new drive, format it and add it back to the array. NOTE: You may be able to reuse the failed drive, if you’re brave, but I wouldn’t recommend it. You can hot-add a device into the array with the following command:

sudo mdadm /dev/md0 -a /dev/sdx

That should be it! Since I haven’t really tested this in a true failure scenario, I would suggest some more poking around on the internet if this happens to you. Hope this helps someone!

While setting this project up and writing this article, I utilized (and blatantly stole from) the following, very helpful websites to get this going (they may be very helpful to you as well):

Was this post helpful?
  • Thanks 
  • Sort-of 
  • Not Helpful