[Bug 1190295] Re: latest 10.04 kernel update breaks software RAID + LVM
bl8n8r
1190295 at bugs.launchpad.net
Thu Jun 13 13:25:12 UTC 2013
This is the state of the disk subsystem after rebooting from 10.04 live
media an bringing up /dev/md0 on one disk (to make backups) and then re-
adding the second disk to rebuild the array. Notice /dev/md0p1 is Gone
and pvs says it's using /dev/md0 now!
[08:06 06/13/13]
[root at usb-live /mnt/slash]
# mdadm --readwrite /dev/md0
[08:07 06/13/13]
[root at usb-live /mnt/slash]
# mdadm --add /dev/md0 /dev/sdc1
mdadm: re-added /dev/sdc1
[08:07 06/13/13]
[root at usb-live /mnt/slash]
# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sdc1[2] sdb1[0]
58615552 blocks [2/1] [U_]
[>....................] recovery = 0.3% (198592/58615552) finish=14.7min speed=66197K/sec
unused devices: <none>
[08:07 06/13/13]
[root at usb-live /mnt/slash]
# pvs
PV VG Fmt Attr PSize PFree
/dev/md0 vgssd lvm2 a- 55.90g 20.90g
/dev/sda3 vgsata lvm2 a- 913.29g 824.29g
[08:07 06/13/13]
[root at usb-live /mnt/slash]
# lvs
LV VG Attr LSize Origin Snap% Move Log Copy% Convert
home vgsata -wi-a- 4.00g
mysql.bak vgsata -wi-a- 10.00g
sysbak vgsata -wi-a- 50.00g
usr vgsata -wi-ao 5.00g
var vgsata -wi-ao 20.00g
mysqldump vgssd -wi-a- 10.00g
usr vgssd -wi-ao 5.00g
var vgssd -wi-ao 20.00g
[08:07 06/13/13]
[root at usb-live /mnt/slash]
# vgs
VG #PV #LV #SN Attr VSize VFree
vgsata 1 5 0 wz--n- 913.29g 824.29g
vgssd 1 3 0 wz--n- 55.90g 20.90g
[08:07 06/13/13]
[root at usb-live /mnt/slash]
# mount
aufs on / type aufs (rw)
none on /proc type proc (rw,noexec,nosuid,nodev)
none on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /dev type devtmpfs (rw,mode=0755)
none on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
/dev/sdd1 on /cdrom type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=cp437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
/dev/loop0 on /rofs type squashfs (ro,noatime)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
none on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev)
none on /var/run type tmpfs (rw,nosuid,mode=0755)
none on /var/lock type tmpfs (rw,noexec,nosuid,nodev)
none on /lib/init/rw type tmpfs (rw,nosuid,mode=0755)
/dev/sda1 on /mnt/slash type xfs (rw)
/dev/mapper/vgssd-var on /mnt/slash/var type ext4 (ro)
/dev/mapper/vgsata-usr on /mnt/usr type xfs (rw)
/dev/mapper/vgsata-var on /mnt/var type xfs (rw)
/dev/mapper/vgssd-usr on /mnt/slash/usr type ext4 (ro)
[08:07 06/13/13]
[root at usb-live /mnt/slash]
# fdisk -l
Disk /dev/sda: 1000.0 GB, 999989182464 bytes
255 heads, 63 sectors/track, 121575 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000c3b74
Device Boot Start End Blocks Id System
/dev/sda1 * 1 262 2104483+ 83 Linux
/dev/sda2 263 2352 16787925 82 Linux swap / Solaris
/dev/sda3 2353 121575 957658747+ 8e Linux LVM
Disk /dev/sdb: 60.0 GB, 60022480896 bytes
32 heads, 32 sectors/track, 114483 cylinders
Units = cylinders of 1024 * 512 = 524288 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000cb60c
Device Boot Start End Blocks Id System
/dev/sdb1 1 114484 58615640 fd Linux raid autodetect
Disk /dev/sdc: 60.0 GB, 60022480896 bytes
32 heads, 32 sectors/track, 114483 cylinders
Units = cylinders of 1024 * 512 = 524288 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000da964
Device Boot Start End Blocks Id System
/dev/sdc1 1 114484 58615640 fd Linux raid autodetect
Disk /dev/sdd: 8011 MB, 8011120640 bytes
41 heads, 41 sectors/track, 9307 cylinders
Units = cylinders of 1681 * 512 = 860672 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000a7237
Device Boot Start End Blocks Id System
/dev/sdd1 * 5 9308 7819328 c W95 FAT32 (LBA)
Disk /dev/md0: 60.0 GB, 60022325248 bytes
2 heads, 4 sectors/track, 14653888 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/md0 doesn't contain a valid partition table
[08:07 06/13/13]
[root at usb-live /mnt/slash]
# pvs
PV VG Fmt Attr PSize PFree
/dev/md0 vgssd lvm2 a- 55.90g 20.90g
/dev/sda3 vgsata lvm2 a- 913.29g 824.29g
[08:07 06/13/13]
[root at usb-live /mnt/slash]
# uname -a
Linux usb-live 2.6.32-45-server #102-Ubuntu SMP Wed Jan 2 22:53:00 UTC 2013 x86_64 GNU/Linux
[08:07 06/13/13]
[root at usb-live /mnt/slash]
# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sdc1[1] sdb1[0]
58615552 blocks [2/2] [UU]
unused devices: <none>
[08:15 06/13/13]
[root at usb-live /mnt/slash]
# fdisk -l
Disk /dev/sda: 1000.0 GB, 999989182464 bytes
255 heads, 63 sectors/track, 121575 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000c3b74
Device Boot Start End Blocks Id System
/dev/sda1 * 1 262 2104483+ 83 Linux
/dev/sda2 263 2352 16787925 82 Linux swap / Solaris
/dev/sda3 2353 121575 957658747+ 8e Linux LVM
Disk /dev/sdb: 60.0 GB, 60022480896 bytes
32 heads, 32 sectors/track, 114483 cylinders
Units = cylinders of 1024 * 512 = 524288 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000cb60c
Device Boot Start End Blocks Id System
/dev/sdb1 1 114484 58615640 fd Linux raid autodetect
Disk /dev/sdc: 60.0 GB, 60022480896 bytes
32 heads, 32 sectors/track, 114483 cylinders
Units = cylinders of 1024 * 512 = 524288 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000da964
Device Boot Start End Blocks Id System
/dev/sdc1 1 114484 58615640 fd Linux raid autodetect
Disk /dev/sdd: 8011 MB, 8011120640 bytes
41 heads, 41 sectors/track, 9307 cylinders
Units = cylinders of 1681 * 512 = 860672 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000a7237
Device Boot Start End Blocks Id System
/dev/sdd1 * 5 9308 7819328 c W95 FAT32 (LBA)
Disk /dev/md0: 60.0 GB, 60022325248 bytes
2 heads, 4 sectors/track, 14653888 cylinders
Units = cylinders of 8 * 512 = 4096 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000
Disk /dev/md0 doesn't contain a valid partition table
[08:15 06/13/13]
[root at usb-live /mnt/slash]
# pvs
PV VG Fmt Attr PSize PFree
/dev/md0 vgssd lvm2 a- 55.90g 20.90g
/dev/sda3 vgsata lvm2 a- 913.29g 824.29g
[08:17 06/13/13]
[root at usb-live /mnt/slash]
# lvs
LV VG Attr LSize Origin Snap% Move Log Copy% Convert
home vgsata -wi-a- 4.00g
mysql.bak vgsata -wi-a- 10.00g
sysbak vgsata -wi-a- 50.00g
usr vgsata -wi-ao 5.00g
var vgsata -wi-ao 20.00g
mysqldump vgssd -wi-a- 10.00g
usr vgssd -wi-ao 5.00g
var vgssd -wi-ao 20.00g
[08:17 06/13/13]
[root at usb-live /mnt/slash]
# cat /etc/lvm/
archive/ backup/ cache/ lvm.conf
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to mdadm in Ubuntu.
https://bugs.launchpad.net/bugs/1190295
Title:
latest 10.04 kernel update breaks software RAID + LVM
Status in “mdadm” package in Ubuntu:
New
Bug description:
Been running 10.04 LTS on 8 similar AMD Opteron x86_64 servers for
several years. The servers have been kept up-to-date with patches as
they come out. These servers have been running 2.6.x kernels. Each
server has some form of Linux software RAID running on it as well as
3Ware hardware RAID card using SATA disks. Software RAID is
configured as RAID1 for all but one server running software RAID10.
All servers had software raid configured to use single partitions on
each disk of types of 0xFD (Linux Software Raid Autodetect). All
servers were configured with LVM over the top of /dev/md0.
In past year, mysterious problems have been happening with software
RAID after applying system patches. Upon reboot, server is unable to
mount LVM partitions on Linux software RAID and boot is interrupted
with "Continue to wait; or Press S to skip mounting or M for manual
recovery" requiring intervention from an operator.
Upon pressing 'M' and logging in as root, the LVM slices on the
software RAID partition are not mounted and sometimes appear to be
missing from LVM. Oftentimes pvs, vgs and lvs will complain about
"leaking memory". Germane to the issue, LVM will sometimes show the
problem partitions as "Active" while other times during the login,
they will simply be gone. With LVM and /dev/md0 unstable, there is
no way to discern the true state of the partitons in question.
Starting the system from alternate boot media such as CDROM or USB
drive, sometimes shows the software RAID and LVM in proper state which
leads to suspicion of a kernel update on the afflicted system.
Historically and subjectively, best practice in this instance seems to
be booting from live media and starting the array degraded mode, and
backing up the array.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/1190295/+subscriptions
More information about the foundations-bugs
mailing list