HOME > MISC > LINUX > NETWORK BACKUP
Quick Search:

Return to Quick Linux Tips

Network Backup Using Large Hard Drive in External Enclosure (v0.3)
Last modified: Friday November 9, 2012

Just getting started on this. I'm going to slowly update this page as I have consistent success with my configuration.

Short summary:

  1. The ADS USB2.0/1394 case does work. I had another cheaper unit that was nothing but trouble. A friend brought his WinNT drive in a different enclosure over today and plugged it into the same Linux system and we had nothing but trouble with it, constant stream of I/O errors.

    This is the one I bought:
    http://store.yahoo.com/pcsound/840971.html

    The StorTecc PM-350U2-LCS also works and is more compact, but will only hold a regular sized 3.5" hard drive.

    This is the one I bought:
    http://www.dealsonic.com/stalidetousb.html

  2. Choose a good USB 2.0 card. I picked up a StarTech USB 2.0 from the local MicroCenter. I figured if it doesn't work I could just take it back and try another one. I've been told many times to be sure to stay away from VIA chipsets. This is how this card comes up in lspci:
    00:0c.0 USB Controller: ALi Corporation USB 1.1 Controller (rev 03)
    00:0c.1 USB Controller: ALi Corporation USB 1.1 Controller (rev 03)
    00:0c.2 USB Controller: ALi Corporation USB 1.1 Controller (rev 03)
    00:0c.3 USB Controller: ALi Corporation USB 2.0 Controller (rev 01)
    
    Strangely I had another USB2.0/1394 card from OrangeMicro (also recommended by other Linux users) that I had a lot trouble with.
  3. Use Fedora or anything with at least a 2.4.22 kernel. The system is FAR more stable with 2.4.22 than it ever was with RH9's 2.4.20 kernel. With USB 2.0 it would occasionally lock the machine hard when backing up a fast machine over the network. For a while I went back to the motherboard's USB 1.1 ports to slow down the backup. That at least kept it from crashing but the load average would get very high. I'm using USB 2.0 reliably now with hardly any noticeable load on the backup machine.
  4. I'm using this all currently with a 250G drive. I have several large drives (160G, 200G and now 250G) that are in easy swap carriers that go into the ADS case.
  5. Turn on the enclosure and plug it into your system. Something resembling the following should appear in your /var/log/messages :
    ... kernel: hub.c: new USB device 00:0c.3-3, assigned address 2
    ... kernel: scsi2 : SCSI emulation for USB Mass Storage devices
    ... kernel:   Vendor: WDC WD25  Model: 00JB-32FUA0       Rev:  0 0
    ... kernel:   Type:   Direct-Access                      ANSI SCSI revision: 02
    ... kernel: Attached scsi disk sdd at scsi2, channel 0, id 0, lun 0
    ... kernel: SCSI device sdd: 488397168 512-byte hdwr sectors (250059 MB)
    ... kernel:  sdd: sdd2
    
    In this example you can see the hard drive is already formatted because it shows my existing partition sdd2 . If your drive isn't partitioned use fdisk /dev/sdX  to partition the drive in whatever way makes sense to you.
  6. In my case these drives are being used for backups where there will only be multi-gigabyte files on the drive so I created the ext3 file system with the the command line:
    mkfs.ext3 -j -m 0 -T largefile4 /dev/sdd2
    
    The meaning of all the options are:
    -j- Create the filesystem with the ext3 journal
    -m 0- Specify the percentage of the filesystem blocks reserved for the super-user. I set this to 0 because only root will write to this filesystem.
    -T largefile4- Use one inode per 4 megabytes, more efficient for very large files.
    Please note: There is a bug in the Fedora Core 1 version of mkfs.ext3  which causes it to core dump with the -T largefile4  option. There is a fixed e2fsprogs  RPM at http://people.redhat.com/laroche/. which fixes the problem. More info about this bug is in Bugzilla Bug# 113150.
  7. One thing to remember when hot swapping drives is that there is a problem with the Mass Storage driver not clearing out partitioning information from within the kernel when swapping different geometry drives. This doesn't always make itself evident right away but will result in flaky operation or what look like drive errors. The solution to the problem is after removing one drive be sure to tell the kernel to eject the drive before installing a different drive. This also applies to Compact Flash memory devices. For example, if your device comes up on /dev/sdd  do this:
    # umount /dev/sdd1
    # eject /dev/sdd
    # mount /dev/sdd1 /mnt/XXXXX
    

Warnings

This has been a much more difficult project than I expected. There are various issues with USB hardware compatibility. It takes a little work to get the "sweet spot" combination of kernel and hardware. For now the Fedora 2.4.22 kernel looks ok as long as you don't use VIA USB hardware.

Other Useful Resources

I'll add some as I come up with them.

Comments From People Like You!
Network Backup Using Large Hard Drive in External Enclosure
Add a Comment add a comment
Shawn
07-Dec-2005 03:14
Good Day
I have a very wierd scenaria in that I have a 300GB external USB drive for backups. When I plug in the USB Drive the External drive Icon appears on the KDE desktop and when i click on it I can access it.
Now when I run my backup script to the /media/usbdisk4 directory it writes these files to the /media/usbdisk4 directory and not to the USB drive,even though this is where the USB drive is "mounted",therefore the main HDD is filling up.I am really stuck here as I thought the USB drive automatically mounted to the /media/usbdisk4 (as can be seen on the desktop??)

Thanks
Watch out for vfat!!
24-Sep-2004 16:48
So I decided to not "completely" follow the "manual" on this page and backup to vfat (fat32). I thought this might be useful if I needed to get my files restored if I ever went back to Windows.

I tried a couple of tests and each one created a tar file that appears to be corrupt. I was unable to restore from it.

But luckily I am aware of the first rule of disaster recovery. Make sure you test your scenario.

I converted the drive to ext3 and it seems to be working fine now. I don't plan to go back to Windows at home now so I'm sure to be Ok. I can always boot off my System Rescue CD to copy over files when Microsoft releases their product as Open Source. Although I'm not sure I want to use an Open Source MS Product.
Gordon McCague
19-Sep-2004 09:29
I have an external 80GB drive that I can attach to my system via USB2 or FireWire. I really appreciate the shell script providing the commands to backup my data. Especially since it covers a few items I would not have thought of.

Do you have a recommended command for backing up a postgresql database? I can extrapolate from the command provided for mysql but want to make sure I have what I need. I will check out the postgresql documentation for recommendations but it looks like you have given backup some thought and might be able to help me avoid rookie issues.

Not that I'm a backup rookie. I just finally (and completely) switched to Fedora 2 on my workstation at home but still suffer from WinDoze.

Thank you!
Andy Stewart
02-Mar-2004 15:39
Nice article, and very useful - especially highlighting the mkfs bug in fedora, I have been banging my head against a brickwall with that one for a while.  I have also used Troy's excellent tip and as he suggests this does make swapping usb drives somewhat easier.

As an input to his article, I am now using a Maxtor 120Gb USB drive to backup my Fedora Core 1 system, and thought I would share my script for doing that.  I wanted the drive to hold as much data as possible and also handle the cycling of versions to maximise the disk space.  I do a full backup each day, cos I can,and only delete a previous backup if the space is needed;

Works very well, I have /dev/maxtor_usb_120 mounting to /mnt/backup, and holds currently  8 x 18Gb .tar.gz files which it rotates very well.  Hope this may help someone;

#!/bin/bash
#
#script to backup my data folder, web folder, mysql databases,
#and important configuration files.
#
#last modified : 19.February.2004
#author        : Andy Stewart
#
DATA="/home /root /data /opt"
CONFIG="/etc /var/named /var/lib /maintenance /usr/local/apache"
LOGDIR="/maintenance/logs"
BACKUPDIR="/mnt/backup"
MYDATE=`date +%F`
LASTFILESIZE=0
LASTFILENAME=""

echo "backup started : `date`" > $LOGDIR/backup_$MYDATE.log
echo "----------------------------------------------------" >> $LOGDIR/backup_$MYDATE.log
mount $BACKUPDIR

#find the last file in the backup folder
#and store information about it.
for FILE in $BACKUPDIR/*; do
       if [ -f $FILE ]; then
               FILEINFO=(`ls -lk $FILE`)
               LASTFILENAME=${FILEINFO[8]}
               LASTFILESIZE=${FILEINFO[4]}
       fi
done

#delete files until there is enough space to hold a file
#the same size as the last backup.
for FILE in $BACKUPDIR/*; do
       DISKINFO=(`df -k | grep $BACKUPDIR`)
       DISKFREE=${DISKINFO[3]}
       if [ -f $FILE ]; then
               if [[ "$LASTFILESIZE" -gt "$DISKFREE" ]]; then
                       rm -vf $FILE >> $LOGDIR/backup_$MYDATE.log
               fi
       fi
done


mysqldump -umysql_user -pmysql_pwd --quote-names --opt --all-databases > /data/backup/mysql_$MYDATE.sql
tar -cvvzf $BACKUPDIR/backup_$MYDATE.tar.gz $DATA $CONFIG >> $LOGDIR/backup_$MYDATE.log

umount $BACKUPDIR

echo "-----------------------------------------------------" >> $LOGDIR/backup_$MYDATE.log
echo "backup finished `date`" >> $LOGDIR/backup_$MYDATE.log
Troy
18-Dec-2003 23:58
OK, here is a cool thing that isn't too well known, but has been around since Redhat 9: Devlabel.  In a nutshell, devlabel consistently associates storage devices with device nodes.  What does that mean?  Well, consider the situation where you plug two digital memory reader into two USB ports.  The first (reader A) shows up as /dev/sda1 and the second (reader B) as /dev/sda2.  However, if you plug them in in reverse order next time, reader A will show up as /dev/sda2 and reader B is now /dev/sda1.  This is annoying if you have put entries in /etc/fstab to allow you to easily mount USB storage devices.

Devlabel to the rescue.

Sorry I don't have time to explain this fully, but here is an example of the process I go through to use devlabel with removable storage devices:

 make a devlabel for Belkin USB 2.0 external drive enclosure
 (enclosing Seagate 80GB drive):
   plug in to USB 2.0 port; power-up
   locate drive (examine dmesg output)
     (likely /dev/sda for first attached removable device)
   identify desired device for mounting:
     fdisk -l /dev/sda
   identify drive UUID (informative, not required):
     devlabel printid -d /dev/sda1
   create devlabel:
     devlabel add -d /dev/sda1 -s /dev/external_drive --automount
     devlabel status
       (informative, not required)
   create mount point:
     mkdir /mnt/external_drive
   update /etc/fstab:
     /dev/external_drive /mnt/external_drive auto defaults 0 0

[sorry, my clever formatting is lost when this stuff is posted]

Now, no matter when I plug in my external drive, it will always get associated with /dev/external_drive.  Better yet, as soon as I plug the drive in, the system will automatically mount it at /mnt/externa_drive.

Right now, you are either reading this and thinking "huh?" or else you are thinking "this may be the coolest thing I have ever heard of."  Those of you thinking the latter are probably like me and have about 18 USB devices, and have just about pulled your hair out trying to get things to work easily and reliably in linux.

Now, someone make my day: How can you get an icon to appear automatically after you plug in a devlabel device?  I am using Gnome, and I want to plug in my drive and have a drive icon automatically appear on my desktop.  Can someone get this to work and tell me how they did it?

Please E-mail Me with any questions, comments or corrections.
All pictures and text are the property of Gregory Gulik unless otherwise specified. Please contact site owner if you would like permission to use these pictures for your own purposes or to make comments about anything you see here.