[Bug 1345682] Re: fsck on 24TB ext4 keeps crashing
DD Park
deedee6905 at gmail.com
Sun Mar 27 03:25:40 UTC 2016
Hello, I need your help. This bug seemed to have been placed offline due to
inactivity. It is still a problem as been working on moving things around
to get a testing platform. I've been getting new hardware, and started
another build process to get me to a point of testing. I'm plan on doing a
little more testing before going into production based on the thought that
this problem was fixed, but initial testing shows I'm stilll having some
similar problems. I've built a 18TB file system raid5 ext4, and I was
crossing my fingers that it would be stable, but I'm seeing all kinds of
corruptions and doing fsck early I see that the file system doesn't stay
clean for long. I've built 3 systems so far. Two of them have gone into
production and I've limited my ext4 to 16TB. I built another system with
18TB and I once I start copying large amounts of files onto the system, I
start seeing some warning messages indicating some forms of corruption, and
I stop the copy, run fsck, and I find I do not have a clean file system.
I'm running ubuntu 14.04.04 LTS on this test system. I've got another near
identical setup with ubuntu-14.04 and 16TB or less and works fine(this was
the original system that I saw my corruption. After downsizing, I'm good).
I've got another with 18TB, but split into a 16TB partition and 2TB
partition, on a ubuntu-15.04 system and that is working fine. I go back to
an hybrid system I built to do this test. It is running ubuntu14.04.04 and
built this one with 18TB. This was an older file server that did not have
problems that I decomissioned recently so I could do this testing. I
started my burn in tests and started seeing corruption of the file system.
As expected the only thing I can determine is that it doesn't seem to like
>16TB. Please let me know how I can help get this debugged.
On Wed, Nov 18, 2015 at 6:38 PM, Phillip Susi <psusi at ubuntu.com> wrote:
> To jump in for Ted here; I'm pretty sure that "very large" is in the
> millions range. Whatever the cause though, the best way for us to be
> able to reproduce, diagnose, and fix it is with an image of the
> filesystem, even if it is with the file names scrambled ( the names
> don't really matter anyhow, just the number and possibly placement of
> them ).
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1345682
>
> Title:
> fsck on 24TB ext4 keeps crashing
>
> Status in e2fsprogs package in Ubuntu:
> Incomplete
>
> Bug description:
> date; fsck -vy /dev/mapper/raid61p1 ;date
> Thu Jul 17 11:54:21 PDT 2014
> fsck from util-linux 2.20.1
> I can't get my ext4 24TB to fsck clean. there came a whole bunch of file
> stat problems after a clean reboot.
> I tried 3 times so far, and each does something like segfault. I looked
> twice, and the errors were different,
> but was able to capture one output. The system has 6GB ram, and it used
> up all the memory. ended up adding 25GB
> of swap as fsck seems to use up a huge amount of memory. After a long
> while, usually after mem use is >90%,
> I come back to see the fsck has crashed and file system is still not
> clean.
>
> The reboot was clean and should not have caused any corruption. the
> system is using ubuntu-14.04
>
> # uname -a
> Linux gigabyte133 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15 03:51:08
> UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
> root at gigabyte133:~#
>
> I had to use the 64 bit version since the 32 bit version I couldn't get
> more than 15TB file system built and read
> that there were compatibility problems between 32bit version with the -O
> 64bit and the 64 bit version so just
> built the whole thing in 64 bit ubuntu..
>
>
> e2fsck 1.42.9 (4-Feb-2014)
>
>
> /dev/mapper/raid61p1 contains a file system with errors, check forced.
> Pass 1: Checking inodes, blocks, and sizes
> Inode 203167820 has compression flag set on filesystem without
> compression support. Clear? yes
>
> Signal (11) SIGSEGV si_code=SI_KERNEL fault addr=(nil)
> fsck.ext4[0x4266f1]
> /lib/x86_64-linux-gnu/libc.so.6(+0x36ff0)[0x7fd38d087ff0]
> /lib/x86_64-linux-gnu/libext2fs.so.2(ext2fs_rb_next+0x23)[0x7fd38dc7bc43]
> /lib/x86_64-linux-gnu/libext2fs.so.2(+0x10670)[0x7fd38dc5e670]
> fsck.ext4[0x4100f6]# lsb_release -va
> No LSB modules are available.
> Distributor ID: Ubuntu
> Description: Ubuntu 14.04 LTS
> Release: 14.04
> Codename: trusty
>
> /lib/x86_64-linux-gnu/libext2fs.so.2(+0x11c3d)[0x7fd38dc5fc3d]
> /lib/x86_64-linux-gnu/libext2fs.so.2(+0x11f78)[0x7fd38dc5ff78]
>
> /lib/x86_64-linux-gnu/libext2fs.so.2(ext2fs_block_iterate3+0xa13)[0x7fd38dc60b13]
> fsck.ext4[0x4115c4]
> fsck.ext4[0x412699]
> fsck.ext4[0x412761]
>
> /lib/x86_64-linux-gnu/libext2fs.so.2(ext2fs_get_next_inode_full+0x59)[0x7fd38dc6c9a9]
> fsck.ext4(e2fsck_pass1+0x8d8)[0x4130c8]
> fsck.ext4(e2fsck_run+0x52)[0x40deb2]
> fsck.ext4(main+0xd27)[0x40a0e7]
> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7fd38d072ec5]
> fsck.ext4[0x40bde6]
> Thu Jul 17 15:55:01 PDT 2014
>
> # lsb_release -va
> No LSB modules are available.
> Distributor ID: Ubuntu
> Description: Ubuntu 14.04 LTS
> Release: 14.04
> Codename: trusty
>
> ProblemType: Bug
> DistroRelease: Ubuntu 14.04
> Package: util-linux 2.20.1-5.1ubuntu20.1
> ProcVersionSignature: Ubuntu 3.13.0-32.57-generic 3.13.11.4
> Uname: Linux 3.13.0-32-generic x86_64
> NonfreeKernelModules: zfs zunicode zavl zcommon znvpair
> ApportVersion: 2.14.1-0ubuntu3.2
> Architecture: amd64
> Date: Sat Jul 19 18:49:32 2014
> InstallationDate: Installed on 2014-07-11 (8 days ago)
> InstallationMedia: Ubuntu 14.04 LTS "Trusty Tahr" - Release amd64
> (20140417)
> ProcEnviron:
> TERM=xterm
> PATH=(custom, no user)
> LANG=en_US.UTF-8
> SHELL=/bin/bash
> SourcePackage: util-linux
> UpgradeStatus: No upgrade log present (probably fresh install)
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/ubuntu/+source/e2fsprogs/+bug/1345682/+subscriptions
>
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to e2fsprogs in Ubuntu.
https://bugs.launchpad.net/bugs/1345682
Title:
fsck on 24TB ext4 keeps crashing
Status in e2fsprogs package in Ubuntu:
Expired
Bug description:
date; fsck -vy /dev/mapper/raid61p1 ;date
Thu Jul 17 11:54:21 PDT 2014
fsck from util-linux 2.20.1
I can't get my ext4 24TB to fsck clean. there came a whole bunch of file stat problems after a clean reboot.
I tried 3 times so far, and each does something like segfault. I looked twice, and the errors were different,
but was able to capture one output. The system has 6GB ram, and it used up all the memory. ended up adding 25GB
of swap as fsck seems to use up a huge amount of memory. After a long while, usually after mem use is >90%,
I come back to see the fsck has crashed and file system is still not clean.
The reboot was clean and should not have caused any corruption. the
system is using ubuntu-14.04
# uname -a
Linux gigabyte133 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15 03:51:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
root at gigabyte133:~#
I had to use the 64 bit version since the 32 bit version I couldn't get more than 15TB file system built and read
that there were compatibility problems between 32bit version with the -O 64bit and the 64 bit version so just
built the whole thing in 64 bit ubuntu..
e2fsck 1.42.9 (4-Feb-2014)
/dev/mapper/raid61p1 contains a file system with errors, check forced.
Pass 1: Checking inodes, blocks, and sizes
Inode 203167820 has compression flag set on filesystem without compression support. Clear? yes
Signal (11) SIGSEGV si_code=SI_KERNEL fault addr=(nil)
fsck.ext4[0x4266f1]
/lib/x86_64-linux-gnu/libc.so.6(+0x36ff0)[0x7fd38d087ff0]
/lib/x86_64-linux-gnu/libext2fs.so.2(ext2fs_rb_next+0x23)[0x7fd38dc7bc43]
/lib/x86_64-linux-gnu/libext2fs.so.2(+0x10670)[0x7fd38dc5e670]
fsck.ext4[0x4100f6]# lsb_release -va
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04 LTS
Release: 14.04
Codename: trusty
/lib/x86_64-linux-gnu/libext2fs.so.2(+0x11c3d)[0x7fd38dc5fc3d]
/lib/x86_64-linux-gnu/libext2fs.so.2(+0x11f78)[0x7fd38dc5ff78]
/lib/x86_64-linux-gnu/libext2fs.so.2(ext2fs_block_iterate3+0xa13)[0x7fd38dc60b13]
fsck.ext4[0x4115c4]
fsck.ext4[0x412699]
fsck.ext4[0x412761]
/lib/x86_64-linux-gnu/libext2fs.so.2(ext2fs_get_next_inode_full+0x59)[0x7fd38dc6c9a9]
fsck.ext4(e2fsck_pass1+0x8d8)[0x4130c8]
fsck.ext4(e2fsck_run+0x52)[0x40deb2]
fsck.ext4(main+0xd27)[0x40a0e7]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7fd38d072ec5]
fsck.ext4[0x40bde6]
Thu Jul 17 15:55:01 PDT 2014
# lsb_release -va
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04 LTS
Release: 14.04
Codename: trusty
ProblemType: Bug
DistroRelease: Ubuntu 14.04
Package: util-linux 2.20.1-5.1ubuntu20.1
ProcVersionSignature: Ubuntu 3.13.0-32.57-generic 3.13.11.4
Uname: Linux 3.13.0-32-generic x86_64
NonfreeKernelModules: zfs zunicode zavl zcommon znvpair
ApportVersion: 2.14.1-0ubuntu3.2
Architecture: amd64
Date: Sat Jul 19 18:49:32 2014
InstallationDate: Installed on 2014-07-11 (8 days ago)
InstallationMedia: Ubuntu 14.04 LTS "Trusty Tahr" - Release amd64 (20140417)
ProcEnviron:
TERM=xterm
PATH=(custom, no user)
LANG=en_US.UTF-8
SHELL=/bin/bash
SourcePackage: util-linux
UpgradeStatus: No upgrade log present (probably fresh install)
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/e2fsprogs/+bug/1345682/+subscriptions
More information about the foundations-bugs
mailing list