 Veritas
Volume Manager Recovery Features
Ryan Matteson
Veritas Volume Manager (VxVM) has become the standard Logical
Volume Manager (LVM) in many enterprises for its robust feature
set, its ability to run on multiple operating systems (e.g., HP-UX,
Linux, Solaris, Windows), and the numerous scalability, availability,
and recoverability features that come with the base product. The
recoverability features help to ensure that data is protected when
hardware platforms fail and to ease the process required to restore
systems to an operational state.
This article will provide an introduction to two important and
often overlooked recovery features: failure notifications and configuration
database backups. The article will also provide two disaster-recovery
case studies to show how these recovery features can be used to
aide in recovering from disasters when they strike. A basic knowledge
of Veritas Volume Manager will be assumed. If you are new to Veritas
Volume Manager, consult the vxintro(1m) man page for an introduction
to terminology and basic usage.
Veritas Disk Headers
When a device is initialized or encapsulated with Veritas Volume
Manager, a disk header is written to the device's private region.
The disk header contains a diskid to uniquely identify the device,
a disk group identifier to indicate which disk group the device
is associated with, a set of flags to indicate the status and intended
use (e.g., hot spare) of the device, and a hostid to indicate which
host (if any) has the device imported. The disk header can be printed
by passing the Veritas device name to vxdisk(1m)'s "list"
option:
$ vxdisk list c1t1d0
Device: c1t1d0s2
devicetag: c1t1d0
type: auto
hostid: pooh
disk: name=c1t1d0 id=1123602295.10.pooh
group: name=oradg id=1123603158.13.pooh
info: format=cdsdisk,privoffset=256,pubslice=2,privslice=2
flags: online ready private autoconfig autoimport imported
pubpaths: block=/dev/vx/dmp/c1t1d0s2 char=/dev/vx/rdmp/c1t1d0s2
version: 3.1
iosize: min=512 (bytes) max=2048 (blocks)
public: slice=2 offset=2304 len=35365968 disk_offset=0
private: slice=2 offset=256 len=2048 disk_offset=0
update: time=1123603160 seqno=0.6
ssb: actual_seqno=0.0
headers: 0 240
configs: count=1 len=1280
logs: count=1 len=192
Defined regions:
config priv 000048-000239[000192]: copy=01 offset=000000 enabled
config priv 000256-001343[001088]: copy=01 offset=000192 enabled
log priv 001344-001535[000192]: copy=01 offset=000000 enabled
lockrgn priv 001536-001679[000144]: part=00 offset=000000
Multipathing information:
numpaths: 1
c1t1d0s2 state=enabled
Due to the critical nature of the configuration information stored
in the disk headers, it is important to periodically back up this
information. As I will discuss in the section Configuration Backups,
Veritas introduced new features in Veritas Volume Manager 4.X to ease
the backup process.
Veritas Configuration Database
When new objects (e.g., subdisks, plexes, volumes) are created
with the Veritas CLI or GUI, Veritas will write the configuration
data to describe these objects to a configuration database. The
configuration database is stored in the private region of several
devices in each disk group for redundancy. To list the devices in
a disk group with a copy of the configuration database, the vxdg(1m)
utility can be invoked with the "list" option and a disk group to
query:
$ vxdg list oradg | egrep "config disk.*clean online"
config disk c1t1d0s2 copy 1 len=1280 state=clean online
config disk c1t2d0s2 copy 1 len=1280 state=clean online
config disk c1t3d0s2 copy 1 len=1280 state=clean online
config disk c1t4d0s2 copy 1 len=1280 state=clean online
config disk c1t5d0s2 copy 1 len=1280 state=clean online
To print the contents of the configuration database, the vxprint(1m)
utility can be used. The following example uses the vxprint(1m)
"-h" (list record hierarchies) and "-t" (use one-line
format tailored for each record type) options to display the configuration
database with an informational header and descriptive records:
$ vxprint -ht
DG NAME NCONFIG NLOG MINORS GROUP-ID
ST NAME STATE DM_CNT SPARE_CNT APPVOL_CNT
DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE
RV NAME RLINK_CNT KSTATE STATE PRIMARY DATAVOLS SRL
RL NAME RVG KSTATE STATE REM_HOST REM_DG REM_RLNK
CO NAME CACHEVOL KSTATE STATE
VT NAME NVOLUME KSTATE STATE
V NAME RVG/VSET/CO KSTATE STATE LENGTH READPOL PREFPLEX UTYPE
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE
SC NAME PLEX CACHE DISKOFFS LENGTH [COL/]OFF DEVICE MODE
DC NAME PARENTVOL LOGVOL
SP NAME SNAPVOL DCO
dg oradg default default 46000 1123603158.13.pooh
dm c1t1d0 c1t1d0s2 auto 2048 35365968 -
dm c1t2d0 c1t2d0s2 auto 2048 35521408 -
dm c1t3d0 c1t3d0s2 auto 2048 35521408 -
dm c1t4d0 c1t4d0s2 auto 2048 35521408 -
dm c1t5d0 c1t5d0s2 auto 2048 35365968 -
dm c1t6d0 c1t6d0s2 auto 2048 35521408 -
v oravol01 - ENABLED ACTIVE 41943040 SELECT oravol01-03 fsgen
pl oravol01-03 oravol01 ENABLED ACTIVE 41943168 STRIPE 3/128 RW
sv oravol01-S01 oravol01-03 oravol01-L01 1 13981056 0/0 2/2 ENA
sv oravol01-S02 oravol01-03 oravol01-L02 1 13981056 1/0 2/2 ENA
sv oravol01-S03 oravol01-03 oravol01-L03 1 13981056 2/0 2/2 ENA
v oravol01-L01 - ENABLED ACTIVE 13981056 SELECT - fsgen
pl oravol01-P01 oravol01-L01 ENABLED ACTIVE 13981056 CONCAT - RW
sd c1t1d0-02 oravol01-P01 c1t1d0 0 13981056 0 c1t1d0 ENA
pl oravol01-P02 oravol01-L01 ENABLED ACTIVE 13981056 CONCAT - RW
sd c1t4d0-02 oravol01-P02 c1t4d0 0 13981056 0 c1t4d0 ENA
v oravol01-L02 - ENABLED ACTIVE 13981056 SELECT - fsgen
pl oravol01-P03 oravol01-L02 ENABLED ACTIVE 13981056 CONCAT - RW
sd c1t2d0-02 oravol01-P03 c1t2d0 0 13981056 0 c1t2d0 ENA
pl oravol01-P04 oravol01-L02 ENABLED ACTIVE 13981056 CONCAT - RW
sd c1t5d0-02 oravol01-P04 c1t5d0 0 13981056 0 c1t5d0 ENA
v oravol01-L03 - ENABLED ACTIVE 13981056 SELECT - fsgen
pl oravol01-P05 oravol01-L03 ENABLED ACTIVE 13981056 CONCAT - RW
sd c1t3d0-02 oravol01-P05 c1t3d0 0 13981056 0 c1t3d0 ENA
pl oravol01-P06 oravol01-L03 ENABLED ACTIVE 13981056 CONCAT - RW
sd c1t6d0-02 oravol01-P06 c1t6d0 0 13981056 0 c1t6d0 ENA
The object types (e.g., subdisks, disk groups, plexes, volumes, etc.),
block offsets, object names, and locations can be useful for understanding
the Veritas Volume Manager layout and reassembling the configuration
layout during a disaster. Due to the importance of this information,
backing up the configuration database is imperative! As you will see
in the next section, Veritas Volume Manager makes backing up the configuration
information a snap.
Configuration Backups
The vxconfigbackupd(1m) daemon was introduced in Veritas
Volume Manager 4.X to provide an automated way to back up disk headers
and the configuration database. Vxconfigbackupd(1m) is started
automatically at system boot time and actively monitors the configuration
database and disk headers. When vxconfigbackupd(1m) detects
a change to the active configuration, vxconfigbackupd(1m)
will write the configuration to a set of files in the /etc/vx/cbr/bk/<dgname>.<dgid>
directory. ("<dgname>" identifies the disk group, and "<dgid>"
identifies the disk group id.) The vxconfigbackupd(1m) man
page provides the following descriptions for the files in the /etc/vx/cbr/bk/<dgname>.<dgid>
directory:
/etc/vx/cbr/bk/<dgname>.<dgid>/<dgid>.diskinfo
Location of backup file for disk attributes.
/etc/vx/cbr/bk/<dgname>.<dgid>/<dgid>.binconfig
Location of backup file for binary configuration copy.
/etc/vx/cbr/bk/<dgname>.<dgid>/<dgid>.cfgrec
Location of backup file for configuration records
in vxprint -m format.
In addition to the automated backups that are performed by vxconfigbackupd(1m),
manual backups can be performed with the vxconfigbackup(1m)
utility. This is valuable for backing up specific versions of the
configuration and archiving the configuration to a user-supplied destination.
The following example uses the vxconfigbackup(1m) "-l"
(location to store backup) to back up the current configuration to
the /etc/dgbackups directory:
$ /usr/lib/vxvm/bin/vxconfigbackup -l /etc/dgbackups
Start backing up diskgroup oradg to \
/etc/dgbackups/oradg.1127240283.19.winnie ...
VxVM NOTICE V-5-2-3100 Backup complete for diskgroup: oradg
If you are using a version of Veritas Volume Manager prior to 4.X,
you will be unable to use the vxconfigbackupd(1m) and vxconfigbackup(1m)
utilities to manage backups. Don't fear -- the sample shell script
in Listing 1 can be used to archive the configuration to a text file,
which can be passed to vxmake(1m)'s "-d" option to re-create
the configuration if disaster were to strike.
Restoring Configuration Data
When configuration information is lost due to a system outage
or a mistake at the keyboard, the configuration can often be recovered
to a point in time (preferably to a point in time when a known working
backup was taken with vxconfigbackup(1m)) with the vxconfigrestore(1m)
utility. The following example shows how to restore the contents
on the oradg disk group using the configuration backup in the /etc/dgbackups
directory:
$ /usr/lib/vxvm/bin/vxconfigrestore -l /etc/dgbackups oradg
Diskgroup oradg configuration restoration started ......
Installing volume manager disk header for c1t1d0s2 ...
Installing volume manager disk header for c1t2d0s2 ...
Installing volume manager disk header for c1t3d0s2 ...
Installing volume manager disk header for c1t4d0s2 ...
Installing volume manager disk header for c1t5d0s2 ...
Installing volume manager disk header for c1t6d0s2 ...
-
oradg's diskgroup configuration is restored (in a precommitted state).
Diskgroup can be accessed in read only and can be examined using
vxprint(1m) in this state.
Run:
vxconfigrestore -l /etc/dgbackups/ -c oradg ==> to commit the \
restoration.
vxconfigrestore -l /etc/dgbackups/ -d oradg ==> to abort the \
restoration.
When vxconfigrestore(1m) is invoked, the configuration will
be staged to a transient location, allowing the administrator to validate
the configuration with the vxprint(1m) utility. If the configuration
looks sound, the configuration can be committed to the disk headers
and the configuration database by invoking vxconfigrestore(1m)
with the "-c" (commit) option:
$ /usr/lib/vxvm/bin/vxconfigrestore -l /etc/dgbackups/ -c oradg
Committing configuration restoration for diskgroup oradg ....
oradg's diskgroup configuration restoration is committed.
Once the configuration is restored, the volumes can be started, and
the file systems that reside on those volume can be mounted.
Getting Notified When Failures Occur
When Veritas volume manager is installed with the default options,
the vxrelocd(1m) failure event detection and subdisk relocation
daemon is started at boot time. This daemon is responsible for detecting
device failures, relocating subdisks on failed devices to alternate
devices, and notifying personnel that a device or Veritas object
failed. The notifications are sent to the root user by default and
look similar to the following:
To: root@system
Subject: Volume Manager failures on host winnie
Content-Length: 240
Failures have been detected by the VERITAS Volume Manager:
failed disks:
c1t4d0
failed plexes:
raid5vol-P08
Since email addressed to the root user is directed to the local mail
spool by default, it is advantageous to forward all messages addressed
to root to an actual email address. This can be accomplished by enabling
email forwarding and adding the following alias to /etc/aliases:
root: admin@mycompany.com
If you would prefer to use a separate account for notifications, you
can append the account name to the line that starts vxrelocd(1m)
in the vxvm-recover init script:
$ grep vxrelocd /etc/init.d/S95vxvm-recover
vxrelocd root storageproblems &
Configuring the notification facility to route messages to a valid
account is crucial and ensures timely notifications when failures
are detected.
Case Study #1: Recovering from a Power Failure
While sitting at my desk on a Friday afternoon (around 6pm), my
email client chimed to alert me to the following message:
To: root@system
Subject: Volume Manager failures on host winnie
Content-Length: 240
Failures have been detected by the VERITAS Volume Manager:
failed volumes:
oravol01
I immediately logged into winnie and received the following error
when trying to list the directory that was on the failed volume oravol01:
$ ls -la /u01
.: I/O error
A quick check of the system logs revealed numerous SCSI error messages:
$ tail /var/adm/messages
Jul 26 17:49:37 winnie scsi: [ID 107833 kern.warning] \
WARNING: /pci@1f,0/pci@1/scsi@4/sd@1,0 (sd1):
Jul 26 17:49:37 winnie SCSI transport failed: \
reason 'incomplete': retrying command
Jul 26 17:49:37 winnie scsi: [ID 107833 kern.warning] \
WARNING: /pci@1f,0/pci@1/scsi@4/sd@4,0 (sd5):
Jul 26 17:49:37 winnie SCSI transport failed: \
reason 'incomplete': retrying command
Jul 26 17:49:40 winnie scsi: [ID 107833 kern.warning] \
WARNING: /pci@1f,0/pci@1/scsi@4/sd@1,0 (sd1):
Jul 26 17:49:40 winnie disk not responding to selection
Jul 26 17:49:40 winnie scsi: [ID 107833 kern.warning] \
WARNING: /pci@1f,0/pci@1/scsi@4/sd@4,0 (sd5):
Jul 26 17:49:40 winnie disk not responding to selection
Jul 26 17:49:42 winnie scsi: [ID 107833 kern.warning] \
WARNING: /pci@1f,0/pci@1/scsi@4/sd@1,0 (sd1):
Jul 26 17:49:42 winnie disk not responding to selection
Jul 26 17:49:44 winnie scsi: [ID 107833 kern.warning] \
WARNING: /pci@1f,0/pci@1/scsi@4/sd@1,0 (sd1):
Jul 26 17:49:44 winnie disk not responding to selection
The Veritas vxprint(1m) utility was also unable to display
the disk group configuration (since the configuration database was
unavailable):
$ vxprint -g oradg -ht
$
I raced to the machine room to check the status of the hardware and
started to speculate that a SCSI cable or the storage array had failed.
When I checked the hardware, I noticed that the storage array was
powered off, and the equipment in the surrounding racks was working
correctly. Based on my findings, I thought that a power cord failed.
Once I replaced the cord, the storage array came back to life, but
the oradg disk group I needed to access was in the disabled state:
$ vxdg list
NAME STATE ID
oradg disabled 1123603158.13.winnie
A quick check of the disks showed that they were online and associated
with the disabled disk group:
$ vxdisk list
DEVICE TYPE DISK GROUP STATUS
c0t0d0s2 auto:none - - online invalid
c0t1d0s2 auto:none - - online invalid
c1t1d0s2 auto:cdsdisk c1t1d0 oradg online dgdisabled
c1t2d0s2 auto:cdsdisk c1t2d0 oradg online dgdisabled
c1t3d0s2 auto:cdsdisk c1t3d0 oradg online dgdisabled
c1t4d0s2 auto:cdsdisk c1t4d0 oradg online dgdisabled
c1t5d0s2 auto:cdsdisk c1t5d0 oradg online dgdisabled
c1t6d0s2 auto:cdsdisk c1t6d0 oradg online dgdisabled
In some situations, Veritas may report offline devices as "failed
was: cXtXdXs2". When this happens, the vxreattach(1m) command
can reconnect Veritas Volume Manager to "lost" devices. Luckily, in
our case, Veritas was able to reconnect to the devices so I unmounted
the file system, deported the disk group, and imported the disk group
to enable the oradg disk group:
$ umount /u01
$ vxdg deport oradg
$ vxdg import oradg
The deport and import operations are required to fix a disabled disk
group and will validate that the disk group configuration records
are consistent. Once the disk group was imported, I ran vxinfo(1m)
to view the volume and plex status:
$ vxinfo -g oradg -p
vol oravol01 fsgen Startable
plex oravol01-03 ACTIVE
vol oravol01-L01 fsgen Startable
plex oravol01-P01 ACTIVE
plex oravol01-P02 ACTIVE
vol oravol01-L02 fsgen Startable
plex oravol01-P03 ACTIVE
plex oravol01-P04 ACTIVE
vol oravol01-L03 fsgen Startable
plex oravol01-P05 ACTIVE
plex oravol01-P06 ACTIVE
The "Startable" flag indicates that the volume and plexes are in a
startable state, so I executed the vxvol(1m) utility to start
the volume:
$ vxvol -g oradg start oravol01
Once the volume came online, I ran fsck(1m) to replay the transactions
in the VxFS journal:
$ fsck -F vxfs /dev/vx/dsk/oof/oravol01
log replay in progress
replay complete - marking super-block as CLEAN
After fsck(1m) finished the consistency check, I mounted the
file system, applied the archive logs, and was able to bring the database
back up to an operational state. Due to the recovery features built
into Veritas volume manager, Veritas file system, and Oracle, we were
able to avoid a full file system restore! Since I received the failure
notification immediately after Veritas detected a problem with the
volume, I was able to reduce the time it took to recover the faulted
system.
Case Study #2: Recovering a Deleted Volume
While debugging a C program on a Friday afternoon (around 6pm),
one of my colleagues came to my desk and told me he had accidentally
destroyed the production disk group instead of the test disk group
he thought he was working on. A quick check of the Veritas configuration
verified this:
$ vxprint -hft
$
Since we used vxconfigbackup(1m) to take regular backups of
the disk group configuration, we located the last known good working
backup and passed this location to vxconfigrestore(1m)'s "-l"
(location) option:
$ /usr/lib/vxvm/bin/vxconfigrestore -l /etc/dgbackups/oradg_0612205 oradg
Diskgroup oradg configuration restoration started ......
Installing volume manager disk header for c1t1d0s2 ...
Installing volume manager disk header for c1t2d0s2 ...
Installing volume manager disk header for c1t3d0s2 ...
Installing volume manager disk header for c1t4d0s2 ...
Installing volume manager disk header for c1t5d0s2 ...
Installing volume manager disk header for c1t6d0s2 ...
\
oradg's diskgroup configuration is restored (in precommitted state).
Diskgroup can be accessed in read only and can be examined using
vxprint(1m) in this state.
Run:
vxconfigrestore -l /etc/dgbackups/oradg_0612205 \
-c oradg ==> to commit the restoration.
vxconfigrestore -l /etc/dgbackups/oradg_0612205 \
-d oradg ==> to abort the restoration.
We verified the configuration was consistent by running vxprint(1m),
and committed the configuration to the disk headers and configuration
database by invoking vxconfigrestore(1m) with the "-c"
option:
$ /usr/lib/vxvm/bin/vxconfigrestore -c -l /var/tmp/back oradg
Committing configuration restoration for diskgroup oradg ....
oradg's diskgroup configuration restoration is committed.
Once we committed the configuration, the layout matched our last known
good configuration backup:
$ vxprint -ht
Disk group: oradg
DG NAME NCONFIG NLOG MINORS GROUP-ID
ST NAME STATE DM_CNT SPARE_CNT APPVOL_CNT
DM NAME DEVICE TYPE PRIVLEN PUBLEN STATE
RV NAME RLINK_CNT KSTATE STATE PRIMARY DATAVOLS SRL
RL NAME RVG KSTATE STATE REM_HOST REM_DG REM_RLNK
CO NAME CACHEVOL KSTATE STATE
VT NAME NVOLUME KSTATE STATE
V NAME RVG/VSET/CO KSTATE STATE LENGTH READPOL PREFPLEX UTYPE
PL NAME VOLUME KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME PLEX DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME PLEX VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE
SC NAME PLEX CACHE DISKOFFS LENGTH [COL/]OFF DEVICE MODE
DC NAME PARENTVOL LOGVOL
SP NAME SNAPVOL DCO
dg oradg default default 10000 1127240283.19.winnie
v oravol01 - ENABLED ACTIVE 41943040 SELECT oravol01-03 fsgen
pl oravol01-03 oravol01 ENABLED ACTIVE 41943168 STRIPE 3/128 RW
sv oravol01-S01 oravol01-03 oravol01-L01 1 13981056 0/0 2/2 ENA
sv oravol01-S02 oravol01-03 oravol01-L02 1 13981056 1/0 2/2 ENA
sv oravol01-S03 oravol01-03 oravol01-L03 1 13981056 2/0 2/2 ENA
v oravol01-L01 - ENABLED SYNC 13981056 SELECT - fsgen
pl oravol01-P01 oravol01-L01 ENABLED ACTIVE 13981056 CONCAT - RW
sd c1t1d0-02 oravol01-P01 c1t1d0 0 13981056 0 c1t1d0 ENA
pl oravol01-P02 oravol01-L01 ENABLED ACTIVE 13981056 CONCAT - RW
sd c1t4d0-02 oravol01-P02 c1t4d0 0 13981056 0 c1t4d0 ENA
v oravol01-L02 - ENABLED SYNC 13981056 SELECT - fsgen
pl oravol01-P03 oravol01-L02 ENABLED ACTIVE 13981056 CONCAT - RW
sd c1t2d0-02 oravol01-P03 c1t2d0 0 13981056 0 c1t2d0 ENA
pl oravol01-P04 oravol01-L02 ENABLED ACTIVE 13981056 CONCAT - RW
sd c1t5d0-02 oravol01-P04 c1t5d0 0 13981056 0 c1t5d0 ENA
v oravol01-L03 - ENABLED SYNC 13981056 SELECT - fsgen
pl oravol01-P05 oravol01-L03 ENABLED ACTIVE 13981056 CONCAT - RW
sd c1t3d0-02 oravol01-P05 c1t3d0 0 13981056 0 c1t3d0 ENA
pl oravol01-P06 oravol01-L03 ENABLED ACTIVE 13981056 CONCAT - RW
sd c1t6d0-02 oravol01-P06 c1t6d0 0 13981056 0 c1t6d0 ENA
To ensure that no damage had occurred to the file system, we used
fsck(1m) to consistency check the file system:
$ fsck -F vxfs /dev/vx/rdsk/oradg/oravol01
and got our Oracle DBA to verify the integrity of each data file with
Oracle's database verification utilities. Because accidents and disasters
can strike at any time, it is important to perform regular backups
of the Veritas configuration and store this data in a safe location
(having a configuration database backup saved us from having to restore
the database or re-create the Veritas objects by hand).
Conclusion
This article provided an introduction to two important Veritas
Volume Manager recovery techniques and discussed ways to ensure
recoverability after a disaster. When dealing with difficult Veritas
situations, it is imperative to know what function a command performs
before executing it. It is also helpful to engage Veritas support
or the folks on the Veritas mailing lists when major disasters occur.
This will ensure that a second set of eyes is available to review
the problem, and will ensure the quickest and safest methods are
used to recover a faulted system. I tested all commands on a Solaris
10 server running Veritas Volume Manager 4.1, and I highly recommend
testing all configuration changes on non-production systems.
References
Veritas Documentation -- http://www.veritas.com
SSA Helpful Hints -- http://www.eng.auburn.edu/pub/mail-lists/ssastuff/
Veritas Volume Manager Mailing List Archives -- http://mailman.eng.auburn.edu/search.html
Acknowledgements
I thank the individuals who maintain the SSA helpful hints site,
as well as the members of the Veritas Volume Manager list.
Ryan Matteson works as a systems engineer, and specializes
in Web technologies, SANs, and the OpenBSD, Linux, and Solaris operating
systems. When Ryan isn't busy working, he enjoys playing guitar
and maintaining his BLOG at daemons.net/~matty. Questions
and comments about this article can be addressed to matty@daemons.net. |