Notes on Veritas Volume Manager

This is a supplement to the Veritas documentation. That documentation is accurate as far as it goes, but there are some things missing. In particular, a big chunk of how to really do RAID5 seems to have been elided in favor of the vxassist method, but vxassist is not capable of really creating a RAID5 volume out of scattered pieces of subdisk.

Q: How do I replace a failed drive?

A: To begin with, remove the failed plexes from their mirrors. Hey wait! I've got RAID5, not mirrors! Sorry, I haven't encountered that yet myself. Let me know what you did. Also, I suspect that if you really knew what you were doing, this step could be skipped, or might happen automatically when you do some of the below, but I don't know that, so I'm reporting what worked for me. The command to dissociate a plex from a volume is vxplex dis plex-name. Repeat for each plex containing part of the failed drive.

First, use vxprint to see which disk you'll be replacing. Note the disk media (dm) name and the disk group name. Then invoke vxdiskadm. Select item 4 (Remove a disk for replacement), and enter the disk name when prompted. If you're like me, you don't have any hot spares to replace the disk with, so if you're prompted to name the replacement drive, respond none.

Do what you need to do to replace the physical drive at this point.

Next, get the new drive into the system: run vxdiskadd. Enter the disk group name when prompted. You'll be prompted to use the default disk name; respond n. You'll be prompted to make it a spare disk; respond n. When prompted, confirm your choices. If prompted to encapsulate, respond n. About now, the disk will be initialized. Afterward, you're prompted for the disk media name: instead just quit; we've accomplished initialization of the drive.

Finally, invoke vxdiskadm. Select 5 (Replace a failed or removed disk). You'll be prompted for the disk media name; choose appropriately; you'll be prompted for the access name (c#t#d#s#); choose appropriately; confirm your choice; it says it succeeded; tell it you don't want to do another; and you're finished.

Did I say ``Finally''? There's the small matter of recovering the mirrored or RAID 5 volumes that the failed disk sent to the edge. As noted above, I had dissociated the plexes first. To put them back, for each plex do vxplex att volume-name plex-name.
Q: How do I create a RAID5 volume?
A: First, identify the subdisks that are going to participate. Count the number of physical disks that will be participating. That's the number of columns you'll have in the plex. Find a common denominator for the sizes of the subdisks (the factor program is your friend). Create the plex, like this:
```
vxmake plex raid5plex-a stwidth=8 ncolumn=12
```
where stwidth is one of those common denominators in the size of your subdisks, and ncolumn is, you guessed it, the number of columns in the plex. Put all the subdisks from an actual physical disks into the same column of the plex, being careful to specify the offsets for the subdisks in the columns so that you start at 0 and so that subdisks beyond the first in the column have an offset that neither overlaps the previous subdisk sectors nor leaves a hole. The command to add a subdisk to your plex goes like this:
```
vxsd -l 10/1927208 assoc raid5plex-a disk1-2
```
Where 10 is the column number (counting from 1), 1927208 is the offset in secotors from the beginning of the column (other subdisks having completely covered the preceding sectors) raid5plex-a is the name of our plex, and disk1-2 is the name of our subdisk. Your shortest plex column is going to be the multiplier by (the number of columns minus one) to give the amount of space in your RAID5 volume, minus a little overhead. Since the space in the longer columns is wasted; you might contemplate redefining your subdisks so that you can use that space for something else. Now you have to attach the plex to a volume; that's done like so:
```
vxmake -Uraid5 vol raid5vol-a plex=raid5plex-a
```
where raid5vol-a is your volume name and raid5plex-a is your plex name. Things get weird at this point: the manual says that your next step is to start the volume with vxrecover -s raid5vol-a, and that actually worked for me -- once. But then I decided to reclaim some of that wasted space I just mentioned, so I stopped the volume, vxedit -rf rm raid5vol-a, and rebuilt things. But this time the vxrecover didn't appear to do anything. Instead, the following seems to have worked:
```
 
#root# vxvol init active raid5vol-a
#root# vxvol resync raid5vol-a
```
The business of defining subdisks and attaching them to plexes is tedious and error prone. My solution is to use some simple perl scripts, with which I process the result of a vxprint free. Here's some typical output, names changed to protect the guilty:
```
DISK         DEVICE       TAG          OFFSET    LENGTH    FLAGS
disk101      c1t1d0s2     c1t1d0       2106264   702088    -
disk101      c1t1d0s2     c1t1d0       27381432  4212528   -
disk101      c1t1d0s2     c1t1d0       32291336  4712      -
disk102      c1t2d0s2     c1t2d0       2106264   702088    -
disk102      c1t2d0s2     c1t2d0       27381432  4212528   -
disk102      c1t2d0s2     c1t2d0       32291336  4712      -
```
So to find how much space I've got on each disk, I might use this:
```
#!/usr/local/bin/perl
# input line:
# disk1    c2t4d0s2     c2t4d0       25275168  3510440   -
$j=-1;
while(<>) {
  next unless /^disk1/; # don't process the title line
  @F=split;
  if ($F[0] eq $prevdsk) {
    $s += $prevsiz;
  } else {
    print "$prevdsk size is: ", $s, "\n" if $prevdsk;
    $s=0;
    $prevdsk=$F[0];
  }
  $prevsiz = $F[4];
}
print "$prevdsk size is: ", $s, "\n" if $prevdsk;
```
This is gross overkill on this data, but if you've got lots of disks of different sizes with irregular chunks available, it can be quite a time saver.

Next, we want to turn those into subdisks. So how about something like this:
```
#!/usr/local/bin/perl
# input line:
# disk1    c2t4d0s2     c2t4d0       25275168  3510440   -
while(<>) {
  next unless /^disk1/; # skip the title line
  @F=split;
  if ($F[0] eq $prev) {
    $i++;
  } else {
    $i=1;
    $prev=$F[0];
  }
  print "vxmake sd ", $F[0], "-", $i, " ", $F[0], ",", $F[3], ",", $F[4], "\n";
}
```
which, when fed the vxprint free output you saved, generates text like this:
```
vxmake sd disk1-1 disk1 2106264 702088
```
Pipe that to /bin/sh and you're finished making subdisks! But be careful! You don't want to set this broom to fetching water without eyeballing what it's about to do!

The other tedious piece is getting all those subdisks into the right place in the plex. So here's a possible time saver:
```
#!/usr/local/bin/perl
# input line:
# disk1    c2t4d0s2     c2t4d0       25275168  3510440   -
$j=-1;
while(<>) {
  next unless /^disk1/;
  @F=split;
  if ($F[0] eq $prevdsk) {
    $i++;
    $s += $prevsiz;
  } else {
    $j++;
    $s=0;
    $i=1;
    $prevdsk=$F[0];
  }
  $prevsiz = $F[4];
  print "vxsd -l ", $j ,"/", $s, " assoc raid5plex-a ", $F[0], "-", $i, "\n";
}
```
When fed with that vxprint free output you'd tucked away, that should produce lines like
```
vxsd -l 1/0 assoc raid5plex-a disk1-1
vxsd -l 1/702088 assoc raid5plex-a disk1-2
```
Again, check that broom before you fly away on it!