[Bug 1190295] Re: latest 10.04 kernel update breaks software RAID + LVM
bl8n8r
1190295 at bugs.launchpad.net
Thu Jun 13 13:16:56 UTC 2013
I was expecting the RAID system to blow up so I saved the scroll-back of
the terminal. This is the state of the disk subsystem and RAID info
prior to rebooting. Of particular note, /dev/md0p1 is marked as Linux
Raid Autodetect but it's supposed to be an LVM partition!
$ ssh root at newbox
Linux newbox 3.0.0-32-server #51~lucid1-Ubuntu SMP Fri Mar 22 17:53:04 UTC 2013 x86_64 GNU/Linux
Ubuntu 10.04.4 LTS
Welcome to the Ubuntu Server!
* Documentation: http://www.ubuntu.com/server/doc
System information as of Wed Jun 12 10:08:23 CDT 2013
System load: 0.15 Processes: 171
Usage of /home: 0.8% of 3.99GB Users logged in: 0
Memory usage: 0% IP address for eth0: 10.140.136.41
Swap usage: 0%
Graph this data and manage this system at
https://landscape.canonical.com/
18 packages can be updated.
11 updates are security updates.
New release 'precise' available.
Run 'do-release-upgrade' to upgrade to it.
Last login: Wed Jun 12 10:07:11 2013
# screen -x
[detached from 1306.tty1.newbox]
# fdisk -l
Disk /dev/sda: 60.0 GB, 60022480896 bytes
32 heads, 32 sectors/track, 114483 cylinders
Units = cylinders of 1024 * 512 = 524288 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000cb60c
Device Boot Start End Blocks Id System
/dev/sda1 1 114484 58615640 fd Linux raid autodetect
Disk /dev/sdb: 60.0 GB, 60022480896 bytes
32 heads, 32 sectors/track, 114483 cylinders
Units = cylinders of 1024 * 512 = 524288 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000da964
Device Boot Start End Blocks Id System
/dev/sdb1 1 114484 58615640 fd Linux raid autodetect
Disk /dev/sdc: 1000.0 GB, 999989182464 bytes
255 heads, 63 sectors/track, 121575 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000c3b74
Device Boot Start End Blocks Id System
/dev/sdc1 * 1 262 2104483+ 83 Linux
/dev/sdc2 263 2352 16787925 82 Linux swap / Solaris
/dev/sdc3 2353 121575 957658747+ 8e Linux LVM
Disk /dev/md0: 60.0 GB, 60022325248 bytes
32 heads, 32 sectors/track, 114483 cylinders
Units = cylinders of 1024 * 512 = 524288 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000cb60c
Device Boot Start End Blocks Id System
/dev/md0p1 1 114484 58615640 fd Linux raid autodetect
# pvs
PV VG Fmt Attr PSize PFree
/dev/md0p1 vgssd lvm2 a- 55.90g 20.90g
/dev/sdc3 vgsata lvm2 a- 913.29g 849.29g
# lvs
LV VG Attr LSize Origin Snap% Move Log Copy% Convert
home vgsata -wi-ao 4.00g
mysql.bak vgsata -wi-ao 10.00g
sysbak vgsata -wi-a- 50.00g
mysqldump vgssd -wi-ao 10.00g
usr vgssd -wi-ao 5.00g
var vgssd -wi-ao 20.00g
# mdadm -E --scan
ARRAY /dev/md0 level=raid1 num-devices=2 UUID=8035e4aa:c77c9788:e368bf24:bd0fce41
# cat /etc/mdadm/mdadm.conf
DEVICE partitions
CREATE owner=root group=disk mode=0660 auto=yes
HOMEHOST <system>
MAILADDR root
# This file was auto-generated on Mon, 29 Apr 2013 15:25:47 -0500
# by mkconf $Id$
ARRAY /dev/md0 level=raid1 num-devices=2 UUID=8035e4aa:c77c9788:e368bf24:bd0fce41
# mdadm --examine /dev/sda1
/dev/sda1:
Magic : a92b4efc
Version : 00.90.00
UUID : 8035e4aa:c77c9788:e368bf24:bd0fce41
Creation Time : Thu May 9 09:21:34 2013
Raid Level : raid1
Used Dev Size : 58615552 (55.90 GiB 60.02 GB)
Array Size : 58615552 (55.90 GiB 60.02 GB)
Raid Devices : 2
Total Devices : 2
Preferred Minor : 0
Update Time : Wed Jun 12 10:10:19 2013
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Checksum : 381900d2 - correct
Events : 41
Number Major Minor RaidDevice State
this 0 8 0 0 active sync /dev/sda
0 0 8 0 0 active sync /dev/sda
1 1 8 16 1 active sync /dev/sdb
# mdadm --examine /dev/sdb1
/dev/sdb1:
Magic : a92b4efc
Version : 00.90.00
UUID : 8035e4aa:c77c9788:e368bf24:bd0fce41
Creation Time : Thu May 9 09:21:34 2013
Raid Level : raid1
Used Dev Size : 58615552 (55.90 GiB 60.02 GB)
Array Size : 58615552 (55.90 GiB 60.02 GB)
Raid Devices : 2
Total Devices : 2
Preferred Minor : 0
Update Time : Wed Jun 12 10:10:19 2013
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Checksum : 381900e4 - correct
Events : 41
Number Major Minor RaidDevice State
this 1 8 16 1 active sync /dev/sdb
0 0 8 0 0 active sync /dev/sda
1 1 8 16 1 active sync /dev/sdb
# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sdb[1] sda[0]
58615552 blocks [2/2] [UU]
unused devices: <none>
# reboot
** Package changed: partman-base (Ubuntu) => mdadm (Ubuntu)
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to mdadm in Ubuntu.
https://bugs.launchpad.net/bugs/1190295
Title:
latest 10.04 kernel update breaks software RAID + LVM
Status in “mdadm” package in Ubuntu:
New
Bug description:
Been running 10.04 LTS on 8 similar AMD Opteron x86_64 servers for
several years. The servers have been kept up-to-date with patches as
they come out. These servers have been running 2.6.x kernels. Each
server has some form of Linux software RAID running on it as well as
3Ware hardware RAID card using SATA disks. Software RAID is
configured as RAID1 for all but one server running software RAID10.
All servers had software raid configured to use single partitions on
each disk of types of 0xFD (Linux Software Raid Autodetect). All
servers were configured with LVM over the top of /dev/md0.
In past year, mysterious problems have been happening with software
RAID after applying system patches. Upon reboot, server is unable to
mount LVM partitions on Linux software RAID and boot is interrupted
with "Continue to wait; or Press S to skip mounting or M for manual
recovery" requiring intervention from an operator.
Upon pressing 'M' and logging in as root, the LVM slices on the
software RAID partition are not mounted and sometimes appear to be
missing from LVM. Oftentimes pvs, vgs and lvs will complain about
"leaking memory". Germane to the issue, LVM will sometimes show the
problem partitions as "Active" while other times during the login,
they will simply be gone. With LVM and /dev/md0 unstable, there is
no way to discern the true state of the partitons in question.
Starting the system from alternate boot media such as CDROM or USB
drive, sometimes shows the software RAID and LVM in proper state which
leads to suspicion of a kernel update on the afflicted system.
Historically and subjectively, best practice in this instance seems to
be booting from live media and starting the array degraded mode, and
backing up the array.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/1190295/+subscriptions
More information about the foundations-bugs
mailing list