How to setup SVM (Sun Volume Manager) using RAID 1 mirroring one disk to another with Solaris 10. Derek Crudgington - dacrud@gmail.com CONTENTS: INTRODUCTION PART I: DISK LAYOUT PART II: METADB SETUP - FREEING SWAP SPACE FOR A NEW SLICE PART III: VTOC COPY AND METADB INSTALLATION PART IV: CONFIGURING OBP AND TESTING ------------------------------------------------------------------------------------------ INTRODUCTION This file is an attempt to explain how to setup RAID 1 on a Solaris 10 machine using Sun Volume Manager. This was done on a Sun E4500 attatched by fiber to a StorEDGE A5100 array running Solaris 10 release 3/05, but should work on any Solaris system. I tried to make this as easy to read as possible because all of the documentation I found was hard to comprehend at first. ------------------------------------------------------------------------------------------ PART I: DISK LAYOUT The disks I will be using are c0t096d0 c0t097d0 for the Solaris OS and c0t098d0 c0t099d0 for the mirrors. Here's a layout of how I have c0t096d0: 0 root wm 293 - 514 389.26MB (222/0/0) 797202 1 usr wm 515 - 970 799.56MB (456/0/0) 1637496 2 backup wm 0 - 4923 8.43GB (4924/0/0) 17682084 3 var wm 971 - 1156 326.14MB (186/0/0) 667926 4 swap wu 0 - 256 450.63MB (257/0/0) 922887 5 unassigned wm 1157 - 2720 2.68GB (1564/0/0) 5616324 6 usr wm 2721 - 4923 3.77GB (2203/0/0) 7910973 7 unassigned wm 257 - 292 63.12MB (36/0/0) 129276 Here's a layout of how I have c0t097d0: 0 unassigned wm 0 0 (0/0/0) 0 1 unassigned wm 0 0 (0/0/0) 0 2 backup wm 0 - 4923 8.43GB (4924/0/0) 17682084 3 unassigned wm 0 0 (0/0/0) 0 4 unassigned wm 0 0 (0/0/0) 0 5 unassigned wm 0 0 (0/0/0) 0 6 unassigned wm 0 0 (0/0/0) 0 7 home wm 0 - 4923 8.43GB (4924/0/0) 17682084 c0t096d0 is using all of the slices and c0t097d0 is filled completey with slice 7 (/export/home). This was automatically setup in the Solaris 10 install. ------------------------------------------------------------------------------------------ PART II: METADB SETUP - FREEING SWAP SPACE FOR A NEW SLICE If you already have a free slice and you don't need to do this, skip down to PART II. The first part in setting up a mirror is the metadb's. These are databases placed on slices that hold the information for the mirrors. It's good to have these on their own slice and its good to have atleast two of these on each disk. After installing Solaris I realized I didnt have any free slices to use for the metadb's so I had to shrink my swap to allocate space for a new slice and then add swap back. This is how I did it: First Boot into single user mode. {5} ok boot -s Once it's booted, list our swap status: # swap -l swapfile dev swaplo blocks free /dev/dsk/c0t96d0s4 118,76 16 1052144 1052144 This tells us it is on slice 4 (c0t96d0s4), we want to delete it: # swap -d /dev/dsk/c0t96d0s4 /dev/dsk/c0t96d0s4 was dump device -- invoking dumpadm(1M) -d swap to select new dump device dumpadm: no swap devices are available List the swaps again to make sure it's gone: # swap -l No swap devices configured Enter the format utility (this is where you manage the disks): # format Searching for disks...done AVAILABLE DISK SELECTIONS: 0. c0t96d0 /sbus@2,0/SUNW,socal@d,10000/sf@1,0/ssd@w210000203723015c,0 1. c0t97d0 /sbus@2,0/SUNW,socal@d,10000/sf@1,0/ssd@w21000020370ede66,0 2. c0t98d0 /sbus@2,0/SUNW,socal@d,10000/sf@1,0/ssd@w210000203714ef36,0 3. c0t99d0 /sbus@2,0/SUNW,socal@d,10000/sf@1,0/ssd@w21000020370bbfd9,0 4. c0t100d0 /sbus@2,0/SUNW,socal@d,10000/sf@1,0/ssd@w210000203716830d,0 5. c0t101d0 /sbus@2,0/SUNW,socal@d,10000/sf@1,0/ssd@w2100002037146891,0 6. c0t102d0 /sbus@2,0/SUNW,socal@d,10000/sf@1,0/ssd@w21000020370cf8dc,0 7. c0t112d0 /sbus@2,0/SUNW,socal@d,10000/sf@1,0/ssd@w2100002037139837,0 8. c0t113d0 /sbus@2,0/SUNW,socal@d,10000/sf@1,0/ssd@w210000203713bb84,0 9. c0t114d0 /sbus@2,0/SUNW,socal@d,10000/sf@1,0/ssd@w21000020371413ba,0 10. c0t115d0 /sbus@2,0/SUNW,socal@d,10000/sf@1,0/ssd@w2100002037141b20,0 Specify disk (enter its number): 0 selecting c0t96d0 [disk formatted] Warning: Current Disk has mounted partitions. partition> print Current partition table (original): Total disk cylinders available: 4924 + 2 (reserved cylinders) Part Tag Flag Cylinders Size Blocks 0 root wm 293 - 514 389.26MB (222/0/0) 797202 1 usr wm 515 - 970 799.56MB (456/0/0) 1637496 2 backup wm 0 - 4923 8.43GB (4924/0/0) 17682084 3 var wm 971 - 1156 326.14MB (186/0/0) 667926 4 swap wu 0 - 292 513.75MB (293/0/0) 1052163 5 unassigned wm 1157 - 2720 2.68GB (1564/0/0) 5616324 6 usr wm 2721 - 4923 3.77GB (2203/0/0) 7910973 7 unassigned wm 0 0 (0/0/0) 0 The swap partition is going from cylinders 0-292 so I want to fix that to be smaller. Choose the swap to modify it. partition> 4 Part Tag Flag Cylinders Size Blocks 4 swap wu 0 - 292 513.75MB (293/0/0) 1052163 Enter partition id tag[swap]: Enter partition permission flags[wu]: Enter new starting cyl[0]: Enter partition size[1052163b, 293c, 292e, 513.75mb, 0.50gb]: 450mb partition> print Current partition table (unnamed): Total disk cylinders available: 4924 + 2 (reserved cylinders) Part Tag Flag Cylinders Size Blocks 0 root wm 293 - 514 389.26MB (222/0/0) 797202 1 usr wm 515 - 970 799.56MB (456/0/0) 1637496 2 backup wm 0 - 4923 8.43GB (4924/0/0) 17682084 3 var wm 971 - 1156 326.14MB (186/0/0) 667926 4 swap wu 0 - 256 450.63MB (257/0/0) 922887 5 unassigned wm 1157 - 2720 2.68GB (1564/0/0) 5616324 6 usr wm 2721 - 4923 3.77GB (2203/0/0) 7910973 7 unassigned wm 0 0 (0/0/0) 0 This made it goto cylinder 0-256 instead of 0-292 so now I have 32 cylinders for the unassigned slice we are about to create on number 7. partition> 7 Part Tag Flag Cylinders Size Blocks 7 unassigned wm 0 512.00MB (292/0/0) 1048572 Enter partition id tag[unassigned]: Enter partition permission flags[wm]: Enter new starting cyl[257]: Enter partition size[1048572b, 292c, 548e, 512.00mb, 0.50gb]: 292e partition> print Current partition table (unnamed): Total disk cylinders available: 4924 + 2 (reserved cylinders) Part Tag Flag Cylinders Size Blocks 0 root wm 293 - 514 389.26MB (222/0/0) 797202 1 usr wm 515 - 970 799.56MB (456/0/0) 1637496 2 backup wm 0 - 4923 8.43GB (4924/0/0) 17682084 3 var wm 971 - 1156 326.14MB (186/0/0) 667926 4 swap wu 0 - 256 450.63MB (257/0/0) 922887 5 unassigned wm 1157 - 2720 2.68GB (1564/0/0) 5616324 6 usr wm 2721 - 4923 3.77GB (2203/0/0) 7910973 7 unassigned wm 257 - 292 63.12MB (36/0/0) 129276 partition> label Ready to label disk, continue? y partition> quit # reboot After it reboots, check the slices again with format make sure they are the same. ------------------------------------------------------------------------------------------ PART III - VTOC COPY AND METADB INSTALLATION Now that we have the slices setup how we want them, we need to make them the exact same on the mirror drive (c0t98d0). There is an easy command to do this: # prtvtoc /dev/rdsk/c0t96d0s2 | fmthard -s - /dev/rdsk/c0t98d0s2 fmthard: New volume table of contents now in place. After the slices are all setup in place on both disks, we are now ready to create the metadb's on both disks: Remember above we allocated space from swap to use slice 7 for the metadbs so this is where we will specify to use it. Here we are telling it to create two (-c 2) metadbs on each disk. # metadb -f -a -c 2 c0t96d0s7 c0t98d0s7 Create concat/stripe of slice 0 (c0t96d0s0). # metainit -f d10 1 1 c0t96d0s0 d10: Concat/Stripe is setup Create concat/stripe of slice 0 on mirror disk (c0t98d0s0). # metainit -f d20 1 1 c0t98d0s0 d20: Concat/Stripe is setup Create d0 and attatch submirror d10. # metainit d0 -m d10 d0: Mirror is setup For ONLY the root slice, you can use metaroot command to update the vfstab. For other slices, you have to update vfstab by hand. # metaroot d0 Reboot the machine. You have to reboot after running the metaroot command before attatching the second submirror. # reboot Attatch the second sub-mirror (d20) to the volume (d0) which causes a mirror resync. # metattach d0 d20 d0: submirror d20 is attached To check progress of the attatchment: # metastat | grep progress Resync in progress: 12 % done After this, the root slice is now setup to mirror and this can be done for all other slices. To double check your configuration type metastat and it should look similiar to this: # metastat d0: Mirror Submirror 0: d10 State: Okay Submirror 1: d20 State: Okay Pass: 1 Read option: roundrobin (default) Write option: parallel (default) Size: 797202 blocks (389 MB) d10: Submirror of d0 State: Okay Size: 797202 blocks (389 MB) Stripe 0: Device Start Block Dbase State Reloc Hot Spare c0t96d0s0 0 No Okay Yes d20: Submirror of d0 State: Okay Size: 797202 blocks (389 MB) Stripe 0: Device Start Block Dbase State Reloc Hot Spare c0t98d0s0 0 No Okay Yes Device Relocation Information: Device Reloc Device ID c0t98d0 Yes id1,ssd@n200000203714ef36 c0t96d0 Yes id1,ssd@n200000203723015c ------------------------------------------------------------------------------------------ PART IV - CONFIGURING OBP AND TESTING After we have the root slice mirrored now, we can setup the OBP to boot from the mirror root slice if the main one fails. First get the device path for the mirror slice: Write this down, we will need this for the OBP configuration. # ls -al /dev/dsk/c0t98d0s0 lrwxrwxrwx 1 root root 74 May 11 16:05 /dev/dsk/c0t98d0s0 -> ../../devices/sbus@2,0/SUNW,socal@d,10000/sf@1,0/ssd@w 210000203714ef36,0:a Reboot and go into the OBP. # reboot Now we're in the OBP, we setup the alias called "fcalbackup" to point to our root mirror slice that we wrote down. {0} ok nvalias fcalbackup /sbus@2,0/SUNW,socal@d,10000/sf@1,0/ssd@w210000203714ef36,0:a Show the boot devices {0} ok printenv boot-device boot-device = disk diskbrd diskisp disksoc net Now we set the boot device to boot from fcal then fcalbackup if it fails, then net. {0} ok setenv boot-device fcal fcalbackup net boot-device = fcal fcalbackup net I also set the diag-device because my diags are always set to max. {0} ok printenv diag-device diag-device = fcal {0} ok setenv diag-device fcal fcalbackup diag-device = fcal fcalbackup NOTE: If you mess up when you create a nvalias and have to delete a nvalias you must reboot for it to show up when you issue devalias command. If you want to test to see if it works, just do "boot fcalbackup" to boot from the mirror.