Comments From People Like You!
Managing RAID and LVM with Linux |  add a comment |
DB
10-Oct-2010 10:19 |
Hi all,
I have a Hammer NAS device set up with 2 drives that I believe are set up as RAID LVM. The reason that I think this is Ghost says LVM, and Clonezilla says Linux RAID when I try to image the drives. I'm not sure what exactly these are except what I've read online.
I have 2 questions: 1) How do I tell what exactly the setup is so I can read more about it, and, 2) is it possible to recover my data if 1 of these drives has failed?
Thanks in advance
|
Rejean
24-Mar-2010 18:14 |
This is very interesting. I like reading pertinent information on similar topic I experienced. If I may also suggest some reading that I wrote about similar issue.
It's an howto on how to Create a Raid5 under Linux RHEL5.4 using md, lvm and ext4 filesystem. (look at http://panoramicsolution.com/blog/?p=92 )
and I also wrote the experience on Testing for a Raid5 failure (with LVM and MD) on Linux RHEL (http://panoramicsolution.com/blog/?p=118)
Rejean
|
Jon
26-Jul-2009 13:59 |
This is a great guide, I've showed it to a couple of friends who were interested in software RAID but they complained about not having a GUI. After a couple of days of searching I've found a great tutorial on howto setup a RAID array using Fedora's installer, if anybody is interested the tutorial is at http://www.optimiz3.com/installing-fedora-11-and-setting-up-a-raid-0-1-5-6-or-10-array/ . It's a nice tutorial for the Linux n00bs out there.
|
James Cassell
15-Sep-2008 21:48 |
With regard to /etc/mdadm.conf -- you can regenerate it by running:
mdadm --detail --scan > /etc/mdadm.conf
this generates the file using UUIDs, as sascha (below) advises
|
sascha
04-Jul-2008 00:18 |
@Joe - regarding /etc/mdadm.conf I find it much easier in the long run to specify "DEVICE partitions" and dynamically scan for the array based on the UUID.
Makes things a lot easier if you ever have to move things around as it doesn't matter where your devices end up (is it is on sda or sde?) I can literally turn off the server, swap the drives around in any order, boot up and be running with no reconfiguration.
eg:
------------BEGIN /etc/mdadm.conf--------------------
DEVICE partitions
ARRAY /dev/md0 level=raid1 num-devices=2 UUID=93eda5e8:13b3c1b7:72db9882:353cc100
ARRAY /dev/md1 level=raid6 num-devices=4 UUID=d02e45eb:27fdf574:771fb23c:4aa16637
------------END /etc/mdadm.conf--------------------
|
Joe
06-Jun-2008 15:34 |
This is a wonderful tutorial. It saved me hours of work.
I have only one comment to add:
For any raid array that you want to have auto-detected, you need to add some information to /etc/mdadm.conf. Below is a copy of mine. I used raid 1 for device /dev/md0 on /dev/sdb1 and /dev/sdc1. mdadm --detail will give you the UUID. My system is FC8.
------------BEGIN /etc/mdadm.conf--------------------
DEVICE /dev/sd[bc]1
ARRAY /dev/md0 UUID=fb04f6e9:559396b2:7db8d3db:abcb5107
------------END /etc/mdadm.conf--------------------
I created a new initrd with mkinitrd adding the --force-raid-probe --force-lvm-probe options. The drives auto-detect and auto-mount on boot-up.
Thanks for your help.
|
Bill
03-Jun-2008 11:12 |
I've been using WinXP and mounting drives in folders to accommodate my storage requirements, but admittedly, I've been looking for alternatives.
One option is Windows Home Server, as it's *extremely* easy to add storage to, and you can choose what "shares" are stored redundantly, and which ones aren't. Storing files twice isn't as efficient as using parity, but it's a simple solution nonetheless.
Looking this over though, I put together a diagram indicating multiple options, and the last one seems best - when adding storage, add 3 drives of equal size, using R5 on them, and adding them to the logical volume.
You can view the diagram here - http://images35.fotki.com/v1169/photos/8/847763/4046789/HomeRAIDSetup-vi.png
What are your thoughts?
|
mark
28-May-2008 21:58 |
Thanks for this great how to article. Finally, someone who can write an easy to understand guide to do something on Linux. What a breath of fresh air!
|
Dwain Blazej
29-Feb-2008 14:35 |
The guide suggests increasing the physical extent size (-s) to because "The maximum number of physical extents is approximately 65k". That is only true if you are still using the LVM 1 meta data format.
|
Chris
17-Oct-2007 02:23 |
You should also point out that when replacing a raid array by swapping in bigger drives and using grow that you should make the partitions on the bigger drives the size you want in the end raid array. i.e if you are replacing 300 GB drives with 750 GB drives and want the whole drive being used, make the partition on the 750 GB drive 750 GB with a tiny bit of space at the end. I just wasted 10 hours by making the partitions 300 GB... lame.
|
Adam Butler
29-Sep-2007 18:20 |
From what I read elsewhere it is now possible to add a disk to a raid array!
http://scotgate.org/?p=107
|
Tim B
09-Sep-2007 02:10 |
Very useful guide. Have run in on my Fedora 6 Server and found some errors(?).
lvextend -l 57235 lvm-raid -n lvm0 did not work but lvextend -l +57235 /dev/lvmraid/lvm0 did.
Also
ext2online seems to have been replaced with resize2fs.
Apart from these easy to follow.
|
xerxes
15-Aug-2007 13:21 |
Tried the pure lvm method as mentioned earlier. It's not as robust under linux. It has no ability as far as I know to rebuild the array, and if a drive fails, it system will not come back up cleanly because the PV within the LV is missing. I like mdadm.
|
Ketil
24-Jul-2007 04:41 |
I have a problem. I set up a RAID-1 using Fedora Core 4. Then i tried to update the installation to Fedora7, but it failed. I ended up installing Fedora7 on my main disk, and just left the raid disks alone during the installation. Is there a way I can get the raid up and running again with the data that is on them?
|
sudhakar
15-May-2007 00:36 |
Excellent and superb tutorial ..
Really helped me learn and work on LVM
thanks ..
Sudhi
|
Me
20-Apr-2007 07:33 |
Oh, I should also mention that when doing any steps that involve modifying the raid array (mdadm --create, mdadm --grow, or mdadm -a), you must WAIT even though the command returned right away. Rest assured, the raid subsystem is churning in the background.
Use mdadm --detail /dev/md0 to see the progress:
Rebuild Status : 82% complete
|
Me
19-Apr-2007 16:26 |
Just to recap the info thus far:
If you have LVM on top of Raid5, you can easily add another volume to the array by using the various resize functions.
Here's a working example you can run as root. Start in a fresh directory somewhere.
Steps:
1. Create the initial array using three 100MB loopback devices
2. Create an LVM volume
3. Format it with reiserfs, mount it, then add a file.
4. Create a new 100MB loopback device and expand the raid array with it.
5. Resize raid, lvm, and reiserfs (note that this is done LIVE, while the device is still mounted)
6. Tear down everything since this is just for practice.
Be sure to step through each line individually so you understand how it works. Especially take care with lvcreate and lvresize. Be sure to use vgdisplay first to see how many extents you have available. In this example, you start with 49 and later add another 25.
Steps 1-5: Build the array+lvm, then resize them.
mkdir mnt
dd if=/dev/zero of=raid-0 bs=10240 count=10240
cp raid-0 raid-1
cp raid-0 raid-2
losetup /dev/loop0 raid-0
losetup /dev/loop1 raid-1
losetup /dev/loop2 raid-2
mdadm --create --verbose /dev/md0 -l5 -n3 /dev/loop0 /dev/loop1 /dev/loop2
pvcreate /dev/md0
vgcreate lvm-test /dev/md0
vgdisplay lvm-test
lvcreate -l 49 lvm-test -n lvm0
mkreiserfs /dev/lvm-test/lvm0
mount /dev/lvm-test/lvm0 mnt
dd if=/dev/zero of=mnt/testfile.bin bs=1024 count=10240
mdadm --grow /dev/md0 -n4 --backup-file=raidbackup
dd if=/dev/zero of=raid-3 bs=10240 count=10240
losetup /dev/loop3 raid-3
mdadm -a /dev/md0 /dev/loop3
pvresize /dev/md0
vgdisplay lvm-test
lvresize -l +25 /dev/lvm-test/lvm0
resize_reiserfs -s 296M /dev/lvm-test/lvm0
Step 6: Teardown and cleanup (destroys the array and cleans up everything you did):
umount mnt
lvremove -f lvm-test
vgremove lvm-test
mdadm --stop /dev/md0
mdadm --remove /dev/md0
losetup -d /dev/loop0
losetup -d /dev/loop1
losetup -d /dev/loop2
losetup -d /dev/loop3
rm raid-0 raid-1 raid-2 raid-3
rmdir mnt
|
SkaDood1
23-Mar-2007 01:12 |
I agree with Nevyn sorta, but technically I think that Splinter/slashdots below method would work, and keep data fault-tolerant. This is because he is putting each of the 10x RAID arrays across 3 disks. If one disk goes down, all 10 RAIDs go down, but they are each recoverable if you did them right.
This then leads to the question of, is there a means to this end? Is it really worth it? With Spinters method, there is a LOT of points of confusion, it has to be done exactly correct, you have to fully understand what you are doing and deeply plan it out, you will have to write a shell script to do it (so you have to have mastered scripting already), you have to wait a LONG time for the whole thing to run and upgrade, and if a drive fails, it may take you many many hours of agravating work to get all 10+ arrays back.
I am thinking it is not worth it to me. Leave Splinter/slashdots method to unix gurus....
|
Nevyn
28-Feb-2007 07:10 |
I'm a tad concerned by people talking about partitioning off their hard drives. This would make the redundancy pointless wouldn't it? I.e. if 1 drive goes down, suddenly you've got a loss of several partitions when Raid 5 only has tolerance for 1 drive (/ partition) to go down.
In which case, you might as well just use raid-0 and not have any redundancy thus increasing the amount of usuable disk space and still not being able to recover in case of a problem.
|
taber
02-Dec-2006 02:39 |
Having alot of experience with mdadm (even raid 10 and 50 over 16 disks) I have thought about doing this but am going to experiment with a straight LVM "raid" and not use mdadm underneath as LVM supports multiple disks and striping, although what it may lack is the ability to rebuild (see: http://tldp.org/HOWTO/LVM-HOWTO/recipethreescsistripe.html).
I would appreciate your thoughts and input.
|
Justcim
07-Aug-2006 09:25 |
For those now reading this, you can actually expand a raid 5 array with mdadm for software based raid in linux. At least as of kernel 2.6.17 and mdadm 2.5.2 it is possible. One easy way to stest this is to create 4 files with dd, I created 4 100MB files named raidfile[1-4]. Next, use losetup to set them up as loop back devices, I used loop[0-3]. To create the initial array do:
mdadm --create --verbose /dev/md0 -l5 -n3 /dev/loop0 /dev/loop1 /dev/loop2
The array should now be initializing, when it is finished, increase the number of raid devices in the array by issuing the command:
mdadm --grow /dev/md0 -n4 --backup-file=/tmp/raidbackup
This part requires a backup file in case power is lost during the reordering of data, this backup file has to be used to assemble the raid array in that case.
Next, to add the other drive to the array use:
mdadm -a /dev/md0 /dev/loop4
the new drive will then be added to the array with no data lose hopefully. During my testing it has worked flawlessly so far, but as for using it on a production box its probably not recommended unless you have backups. You do keep backups correct?
|
Stuart Gathman
07-Aug-2006 01:23 |
You can have mixed drive sizes with raid 5. The trick is to partition the drives into smaller partitions that can be matched in threes (or pairs for raid-1). When adding a drive to expand space, it may be necessary to move 1 or more partitions to the new drive. Simply add a same size partition on the new drive as a hot spare to the md device, then 'fail' the partition to be moved. When the sync is done, remove the "failed" partition from that md - it is now free to use with another md.
I use partitions of around 40G for this. Things work out most easily if all partitions are the same size. AIX LVM automates this and takes it to the extreme, dividing each physical drive into thousands of partitions, each of which is moved, mirrored, raid5ed independently. The Linux LVM also divides partitions into smalller 4M or so chunks, but alas doesn't integrate with software raid, so you have to do the partitioning manually.
|
hondaman
11-Jun-2006 01:39 |
I have a 3-ware 9550sx controller that has online capactity expansion capabilites. I also am using it in a raid-5 configuration. I stumbled upon your page looking for answers on how to make linux see the new hard drive I just added to the array, and you said it cant be done :(
Not to say youre wrong, but man, there HAS to be a way, somehow, to use the space I just added to my array!
|
Michal
08-Jun-2006 12:53 |
Thanks Fake Rake, so to truley take advantage of a RAID-5 you have to have seperate drives for each part, but the limiting part is expansion correct? If I start with 300gb HD I can only add 300gb HDs and to add it to the RAID array with out rebuilding it is to use EVMS correct?
I guess im trying to find a soultion to be able to add different size HD that will have all the HD under one grouping (so one drive letter vs 12) and the backup of RAID-5, any suggestions?
|
Fake Rake
08-Jun-2006 01:15 |
Michael,
That won't give you any benefit from the RAID array. If you set up a RAID made up only of partitions on the new drive, then you will lose data when that drive fails. Each element in the RAID array needs to be on a separate physical drive, so that one drive failing doesn't lead to any lost data.
|
Michal
07-Jun-2006 12:11 |
This is in regards to Splinters splitting the HDs up in to smaller bits and then putting them together to expand on the raid.
I noticed he split up the partitons in to equal parts because RAID-5 requires that but why split it up in to such small partitions. From this tutorial (not splinters) I can just as easily exapnd on the volume group by adding another seperate RAID array. Currently I have 3 300gb Seagate HDs where I was initally going to put those into there own RAID array. After I fill those up I was simply going to buy another HD (Does size matter?) lets say 400gb, split that up in to 4 100gb partitions create a RAID array out of that and then add it to the volume group that the original 3x300gb is in, now from my understanding this will work correct?
Im still a little lost on what the benifits are to going with splinters idea, but that also could be that im really confused to moving, removing, rebuilding and adding by his method, if possible could someone simplify this?
|
Tom K.
04-Jun-2006 17:47 |
Adding a small comment for the pvmove info. Kind of for my own benefit, as I come back to the page to remind myself how to do things with raid/lvm. Anyhow...
For raid1, trying to do the pvmove gave me the error: "mirror: Required device-mapper target(s) not detected in your kernel". It seems you need the dm-mirror kernel module. So, the process for replacing a raid1 disk in an lvm goes like this.
modprobe dm-mirror
vgextend vg0 /dev/md4
pvmove -v /dev/md1 /dev/md4
vgreduce vg0 /dev/md1
|
bobdazzla
30-May-2006 13:58 |
Excellent guide, excellent explanations of each step. Thank you for providing this resource.
|
splinter
27-May-2006 02:19 |
Actually, there is a cool trick to be able to extend a raid/lvm scheme (I got this from slashdot):
"If you're using Linux software RAID, carve your drives into multiple partitions, build RAID arrays over those, then use LVM to weld them into a larger pool of storage. It may seem silly to break the drives up into paritions, just to put them back together again, but it buys you a great deal of flexibility down the road.
Suppose, for example, that you had three 500GB drives in a RAID-5 configuration, no hot spare. That gives you 1TB of usable storage. Now suppose you're just about out of space, and you want to add another drive. How do you do it? In order to construct a new, four-disk array, you have to destroy the current array. That means you need to back up your data so that you can restore it to the new array. If there were a cheap and convenient backup solution for storing nearly a terabyte, this topic wouldn't even come up.
If, instead, you had cut each 500GB drive into ten 50GB partitions, created ten RAID-5 arrays (each of three 50GB partitions) and then used LVM to place them all into a single volume group, when it comes time to upgrade, you will have another option. As long as you have *free space at least equal in size to one of the individual RAID arrays*, you can use 'pvmove' to instruct LVM to migrate all of the data off of one array, then take that array down, rebuild it with a fourth partition from the new disk, then add it back into the volume group. Do that for each array in turn and at the end of the process you'll have 1.5TB, and not only will all of your data be safely intact, your storage will have been fully available for reading and writing the whole time!
Note that this process isn't particularly fast. I did it when I added a fifth 200GB disk to my file server, and it took nearly a week to complete. A backup and restore would have been faster (assuing I had something to back up to!). But it only took about 30 minutes of my time to write the script that performed the process and then I just let it run, checking on it occasionally. And my kids could watch movies the whole time.
For anyone who's interested in trying it, the basic steps to reconstruct an array are as follows. This example will assume we're rebuilding /dev/md3, which is composed of /dev/hda3, /dev/hdc3 and /dev/hde3 and will be augmented with /dev/hdg3
* pvmove /dev/md3 # Move all data off of /dev/md3
* vgreduce vg /dev/md3 # Remove /dev/md3 from the volume group
* pvremove /dev/md3 # Remove the LVM signature from /dev/md3
* mdadm --stop /dev/md3 # Stop the array
* mdadm --zero-superblock /dev/md3 # Remove the md signature from the disk
* mdadm --create /dev/md3 --level=5 --raid-devices=4 /dev/hda3 /dev/hdc3 /dev/hde3 /dev/hdg3 # Create the new array
* pvcreate /dev/md3 # Prepare /dev/md3 for LVM use
* vgextend vg /dev/md3 # Add /dev/md3 into the array
In order to make this easy, you want to make sure that you have at least one array's worth of space not only unused, but unassigned to any logical volumes. I find it's a good idea to keep about about 1.5 times that much unallocated. Then, when I run out of room in some volume, I just add the 0.5 to the logical volume, and then set about getting more storage to add in [ this is to ensure that you ALWAYS have 1 array's worth of freespace (don't accidentally go over)]."
I'm currently running raid 5 on 4 320GB hdds. I've broken them up into 10 32GB chunks and consequently have 10 raid arrays (md1 - md11).
|
James Cook
24-May-2006 01:28 |
To expand on what an earlier commenter said, you can actually expand your RAID-5 provided you use EVMS to do it. Midway down the EVMS FAQ page, they have a somewhat hackish way of doing so. The relevant Q&A is:
Q: I have a RAID-0 or RAID-5 volume with a JFS or XFS filesystem. I would like to expand my RAID volume by adding another disk. EVMS says the RAID volume must be inactive in order to expand, but it also says that JFS and XFS must be mounted in order to expand. How can I expand my RAID volume?
A: This is definitely an unfortunate catch-22. Luckily there's a pretty simple workaround. All you need to do is fool EVMS into thinking there isn't an XFS or JFS filesystem on your RAID-0 or RAID-5 volume just during the time that you want to expand it. To do this, move the appropriate plugin library (XFS and/or JFS) out of /lib/evms/x.y.z/ to some temporary location. Then run the EVMS UI, and since the XFS/JFS plugin isn't loaded, it won't detect the filesystem. Then you'll be able to expand the RAID volume. After the RAID expand is complete, you'll just need to manually expand the XFS or JFS filesystem (after mounting it). For XFS, use the xfs_growfs command. For JFS, you simply remount the filesystem using the command mount -o remount,resize /mnt/point. After this, you can move the XFS/JFS plugins back to the /lib/evms/x.y.z/ directory.
Taken from: http://evms.sourceforge.net/faq.html
|
Curt
09-Apr-2006 12:50 |
Excellent tutorial! *REALLY* helped. But I must agree with the previous comment that keeping Logical Volume and Logical Group terms straight is important and in building LVs on my test system (using your tutorial) I have learned to name Volume Groups vg01 or vg01 and Logical Volumes lv01 or lv02 so that there is no confusion with the commands. But THANK YOU for taking the time to build this page! WONDERFUL stuff!!!
|
Tobias
26-Jan-2006 09:02 |
@author:
Fine article.
Keep it uptodate and it's a good source in the net.
Please, don't mix up Volume Group and Logical Volume.
> The second command vgcrate will then create the Logical Volume, in my case I called it lvm-raid:
lvm-raid is the Volume Group.
An important point is:
You could (and want) to create more than one Logical Volume. LVs are the guys that you want to mount later, so I want to keep system data and user data apart, e.g.
VG: myhost 120 GB
LV: system 30 GB
LV: user 50 GB
then mount /dev/myhost/system to /
and /dev/myhost/user to /home
Afterwards you can still extend "user" to the maximum of 90 GB and expand the filesystem, or you can make a new LV: mp3 40 GB and mount this under /home/shared/mp3
that's kind of cool flexibility.
@David
RAID makes it more difficult. Adding more discs would mean adding always a even number (for RAID 1) of discs to maintain mirroring....e.g. in fact adding always an independent RAID to the host.
@John Dehls
Try smaller partitions on the raid (i.e. several partitions on a disc, each having a RAID with the corresponding partitions on the other driver), you can then add all the /dev/mdX devices with vgextend/vgcreate to a volume group...
|
Lars Tobias B??rsting
15-Jan-2006 17:47 |
> The answer to how to expand a RAID-5 array is very simple: You can't.
You can if you use EVMS.
|
John Dehls
03-Jan-2006 03:50 |
Everything has gone fine for me until the creation of the file system. I have setup a RAID 5 array with four 400Gb disks, sda, sdb, sdc, sdd. FC4 is running of sdf. The logical volume gets created and looks fine. When I try mkfs.ext3, it runs until its about 25% finished, and then the whole system crashes. After I hit the reset button and reboot, the system spends about 3 hours resyncing md0. I have tried this now about 5 times. How do I go about debugging? Should can I make a smaller logical volume first and then expand it?
|
David
13-Dec-2005 22:43 |
every thing went great. killer notes. suggestions
how to add drives to make the disk space grow. for example started out with 3 120 drives and want to bring it up to 4 120 drives.
That's something I'm working on, just don't have the time lately to experiment and get it right.
|
Dave
23-Oct-2005 15:51 |
Great tutorial, simple and effective. Thanks.
|
Torsten K??hnel
22-Oct-2005 07:20 |
Thanks again! Your infos are very readable and all steps to the point. Its probaly exactly the amount of information you need to set it up the first time, and it has the instructions how to handle disk-failures. The man-pages i've seen are missing that.
So hopefully everyone who is looking for how to set it up finds it !
|
DB
Rejean
Jon
James Cassell
sascha
Joe
Bill
mark
Dwain Blazej
Chris
Adam Butler
Tim B
xerxes
Ketil
sudhakar
Me
SkaDood1
Nevyn
taber
Justcim
Stuart Gathman
hondaman
Michal
Fake Rake
Tom K.
splinter
James Cook
Curt
Lars Tobias B??rsting
John Dehls
David
Dave
Torsten K??hnel