[Bug 1089195] Re: linux-headers will eat your inodes on LTS.

Mon Feb 8 04:44:24 UTC 2016

@pabouk Hi. Thanks so much for the help!

In fact I wasn't able to use apt-get remove, because it tried to do apt-
get update first, which failed because of the inode exhaustion.

I wasn't comfortable removing anything manually because I didn't know
which files were safe to remove. All but one of  the surplus packages in
/usr/src were for a linux headers higher than my current linux version,
so for all I know the system might depend on them in some way when it
tries to upgrade itself from the current version. I erased the one,
older package, but this did not free enough inodes to allow any
operations to complete.

All of the linux headers were piling up in /usr/src. In the end I
resolved the process by doing the following:

1. attaching a much larger new volume, formating it, and mounting it on /mnt
2. cp -a /usr/src/* /mnt   # to copy all the linux headers onto the other volume
3. rm -rf /usr/src/*            # to remove them from the root volume
3. unmount the new volume mounted at /mnt   
4. remount the volume at /usr/src

By this sequence of operations, I exactly preserved the structure of
files and folders on the filesystem, while changing how the inode usage
was distributed over partitions. So that relieved the inode exhaustion.

Then I was able to use apt-get update, and apt-get autoremove, etc.. to
completed the pending upgrades and remove the unused packages.

My big takeaway from this is that I was naive to think unattended-
upgrades could run for years unattended, like a router.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to update-manager in Ubuntu.
https://bugs.launchpad.net/bugs/1089195

Title:
  linux-headers will eat your inodes on LTS.

Status in update-manager package in Ubuntu:
  Confirmed

Bug description:
  Hello all,

  Summary
  -----------------
  Both linux-image-* and linux-headers-* are installed every time you upgrade the kernel. However, they are never removed by any maintenance process.

  Every linux-headers-*-generic-pae package has approx. 6,700 files on
  it. Regular headers packages have even more files: around 11,700 files
  each.

  Although these packages' files won't occupy much space, after some
  years (think LTS installation), they will "eat" all the inodes on your
  root partition.

  The first effect you'll encounter will be the you are unable to
  upgrade your system.

  This is a situation that will affect all users, however it will be a
  greater problem regular non-technical user.

  There's no simple, high-level tool to solve this problem.

  Case Study
  ---------------
  I have a 2-year-and-8-months old 10.04 installation. I have a ~10GB root partition, which is double of the minimum recommend (5GB).

      $ df -h --type=ext4
      Filesystem            Size  Used Avail Use% Mounted on
      /dev/sda5             9.4G  6.2G  2.8G  70% /
      /dev/sda6              94G   45G   45G  51% /home

  It looks that I had plenty of space left to upgrade, but I got this
  error while upgrading:

      unable to create `/usr/src/linux-headers-2.6.32-45/arch/s390/include/asm/nmi.h.dpkg-new'
      (while processing `./usr/src/linux-headers-2.6.32-45/arch/s390/include/asm/nmi.h'): No space left on device

  It was because of I was running out of inodes:

      $ df -i /
      Filesystem            Inodes   IUsed   IFree IUse% Mounted on
      /dev/sda5             625856  618015    7841   99% /

  Why? Because the huge amount of linux-headers files:

      $ find /usr/src -type f | wc -l
      355112

  That is more than three hundred and fifty thousand files!

  These are the packages that I removed:

  Purg linux-headers-2.6.32-21 [2.6.32-21.32]
  Purg linux-headers-2.6.32-22-generic-pae [2.6.32-22.36]
  Purg linux-headers-2.6.32-22 [2.6.32-22.36]
  Purg linux-headers-2.6.32-23-generic-pae [2.6.32-23.37]
  Purg linux-headers-2.6.32-23 [2.6.32-23.37]
  Purg linux-headers-2.6.32-24-generic-pae [2.6.32-24.43]
  Purg linux-headers-2.6.32-24 [2.6.32-24.43]
  Purg linux-headers-2.6.32-25-generic-pae [2.6.32-25.45]
  Purg linux-headers-2.6.32-25 [2.6.32-25.45]
  Purg linux-headers-2.6.32-26-generic-pae [2.6.32-26.48]
  Purg linux-headers-2.6.32-26 [2.6.32-26.48]
  Purg linux-headers-2.6.32-27-generic-pae [2.6.32-27.49]
  Purg linux-headers-2.6.32-27 [2.6.32-27.49]
  Purg linux-headers-2.6.32-28-generic-pae [2.6.32-28.55]
  Purg linux-headers-2.6.32-28 [2.6.32-28.55]
  Purg linux-headers-2.6.32-29-generic-pae [2.6.32-29.58]
  Purg linux-headers-2.6.32-29 [2.6.32-29.58]
  Purg linux-headers-2.6.32-30-generic-pae [2.6.32-30.59]
  Purg linux-headers-2.6.32-30 [2.6.32-30.59]
  Purg linux-headers-2.6.32-31-generic-pae [2.6.32-31.61]
  Purg linux-headers-2.6.32-31 [2.6.32-31.61]
  Purg linux-headers-2.6.32-32-generic-pae [2.6.32-32.62]
  Purg linux-headers-2.6.32-32 [2.6.32-32.62]
  Purg linux-headers-2.6.32-33-generic-pae [2.6.32-33.72]
  Purg linux-headers-2.6.32-33 [2.6.32-33.72]
  Purg linux-headers-2.6.32-34-generic-pae [2.6.32-34.77]
  Purg linux-headers-2.6.32-34 [2.6.32-34.77]
  Purg linux-headers-2.6.32-35-generic-pae [2.6.32-35.78]
  Purg linux-headers-2.6.32-35 [2.6.32-35.78]
  Purg linux-headers-2.6.32-36-generic-pae [2.6.32-36.79]
  Purg linux-headers-2.6.32-36 [2.6.32-36.79]
  Purg linux-headers-2.6.32-37-generic-pae [2.6.32-37.81]
  Purg linux-headers-2.6.32-37 [2.6.32-37.81]
  Purg linux-headers-2.6.32-38-generic-pae [2.6.32-38.83]
  Purg linux-headers-2.6.32-38 [2.6.32-38.83]
  Purg linux-headers-2.6.32-39-generic-pae [2.6.32-39.86]
  Purg linux-headers-2.6.32-39 [2.6.32-39.86]
  Purg linux-headers-2.6.32-40-generic-pae [2.6.32-40.87]
  Purg linux-headers-2.6.32-40 [2.6.32-40.87]
  Purg linux-headers-2.6.32-41-generic-pae [2.6.32-41.94]
  Purg linux-headers-2.6.32-41 [2.6.32-41.94]
  Purg linux-headers-2.6.32-42-generic-pae [2.6.32-42.96]
  Purg linux-headers-2.6.32-42 [2.6.32-42.96]
  Purg linux-headers-2.6.32-43-generic-pae [2.6.32-43.97]
  Purg linux-headers-2.6.32-43 [2.6.32-43.97]
  Purg linux-headers-generic-pae [2.6.32.45.52]
  Purg linux-image-2.6.32-42-generic-pae [2.6.32-42.96]
  Purg linux-image-2.6.32-43-generic-pae [2.6.32-43.97]

  Then, problem solved:

      $ find /usr/src/ -type f | wc -l
      28276

      $ ls /usr/src/
      linux-headers-2.6.32-44              linux-headers-2.6.32-45-generic-pae
      linux-headers-2.6.32-44-generic-pae  nvidia-current-195.36.24
      linux-headers-2.6.32-45              virtualbox-ose-3.1.6

      $ df -i --type=ext4
      Filesystem            Inodes   IUsed   IFree IUse% Mounted on
      /dev/sda5             625856  192891  432965   31% /
      /dev/sda6            6225920   95025 6130895    2% /home

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/update-manager/+bug/1089195/+subscriptions