[Bug 1751675] [NEW] NTFS-3G Major Issue: Improper clearing of memory buffer/cache resulting in OOM Kills

[Ben Roten]

Public bug reported:

I am running a modern gaming rig with Kubuntu 16.10 installed. 16GB ram,
32GB swap. I have an 8TB external that can only be formatted in NTFS
(does not behave correctly when formatted any other way). When
transferring INDIVIDUAL files to the drive (and reading files), the disk
memory cache/buffer as according to the KInfoCenter slowly fills up and
is not cleared. After transferring/reading enough INDIVIDUAL files the
system runs out of memory and starts OOM killing every app that is
currently launched, including the desktop environment after a time (in
this case, Plasma). If I use the command

"sudo sync && sudo echo 3 | sudo tee /proc/sys/vm/drop_caches"

when the buffer/cache is nearly full, no crashes occur. SWAP is
essentially never used at any point, even if swappiness value is set
very high. Relaunching the apps does not clear the memory and typically
results in the apps immediately crashing as soon as more memory is
needed. Furthermore, I am on kernel version "4.4.0-62-generic" which is
necessary in pointing out because a previous kernel version had a bug
involving memory and overactive OOM killer.

***However, the distinction for individual files is important because if
many files are selected simultaneously and copied over, the memory
buffer/cache is cleared systematically as the transfers are completed,
preventing any crash. This is useful but still does not prevent the
eventual accruing of memory/buffer space when accessing many large files
on the drive over time.

As far as I can tell, there is no way to automatically clear the
buffer/cache without having a manual script that would have to be run
every time. Also worth noting, I have spent sometime trying to nail down
the issue and nearly positive that is has nothing to do with any other
aspect of how Debian/the kernel manages memory because every memory
tweak I have tried has had no effect, short of running the terminal
command above to clear the buffer/cache.

It seems that NTFS-3G needs to clear the memory/buffer after
transferring individual files or reading them or run a check to clear it
after a certain percentage is reached.

** Affects: ntfs-3g (Ubuntu)
     Importance: Undecided
         Status: Confirmed

** Description changed:

  I am running a modern gaming rig with Kubuntu 16.10 installed. 16GB ram,
  32GB swap. I have an 8TB external that can only be formatted in NTFS
  (does not behave correctly when formatted any other way). When
- transferring INDIVIDUAL files to (and reading files), the disk memory
- cache/buffer as according to the KInfoCenter slowly fills up is not
- cleared. After transferring/reading enough INDIVIDUAL files the system
- runs out of memory and starts OOM killing every app that is launched,
- including the desktop environment (in this case, Plasma). If I use the
- command
+ transferring INDIVIDUAL files to the drive (and reading files), the disk
+ memory cache/buffer as according to the KInfoCenter slowly fills up and
+ is not cleared. After transferring/reading enough INDIVIDUAL files the
+ system runs out of memory and starts OOM killing every app that is
+ currently launched, including the desktop environment after a time (in
+ this case, Plasma). If I use the command
  
  "sudo sync && sudo echo 3 | sudo tee /proc/sys/vm/drop_caches"
  
  when the buffer/cache is nearly full, no crashes occur. SWAP is
  essentially never used at any point, even if swappiness value is set
- very high.
+ very high. Relaunching the apps does not clear the memory and typically
+ results in the apps immediately crashing as soon as more memory is
+ needed.
  
  ***However, the distinction for individual files is important because if
  many files are selected simultaneously and copied over, the memory
- buffer/cache is cleared systematically as the transfer are completed,
- preventing a crash. This is useful but still does not prevent the
+ buffer/cache is cleared systematically as the transfers are completed,
+ preventing any crash. This is useful but still does not prevent the
  eventual accruing of memory/buffer space when accessing many large files
  on the drive over time.
  
  As far as I can tell, there is no way to automatically clear the
- buffer/cache without having a manual script that would have to be
- selected every time. Also worth noting, I have spent sometime trying to
- nail down the issue and nearly positive that is has nothing to do with
- any other aspect of how debian manages memory because every memory tweak
- I have tried has had no effect, short of running the terminal command
- above to clear the buffer/cache.
+ buffer/cache without having a manual script that would have to be run
+ every time. Also worth noting, I have spent sometime trying to nail down
+ the issue and nearly positive that is has nothing to do with any other
+ aspect of how Debian/the kernel manages memory because every memory
+ tweak I have tried has had no effect, short of running the terminal
+ command above to clear the buffer/cache.
  
  It seems that NTFS-3G needs to clear the memory/buffer after
- transferring individual files or reading them or have a safeguard to
- clear it after a certain percentage is reached.
+ transferring individual files or reading them or run a check to clear it
+ after a certain percentage is reached.

** Changed in: ntfs-3g (Ubuntu)
       Status: New => Confirmed

** Description changed:

  I am running a modern gaming rig with Kubuntu 16.10 installed. 16GB ram,
  32GB swap. I have an 8TB external that can only be formatted in NTFS
  (does not behave correctly when formatted any other way). When
  transferring INDIVIDUAL files to the drive (and reading files), the disk
  memory cache/buffer as according to the KInfoCenter slowly fills up and
  is not cleared. After transferring/reading enough INDIVIDUAL files the
  system runs out of memory and starts OOM killing every app that is
  currently launched, including the desktop environment after a time (in
  this case, Plasma). If I use the command
  
  "sudo sync && sudo echo 3 | sudo tee /proc/sys/vm/drop_caches"
  
  when the buffer/cache is nearly full, no crashes occur. SWAP is
  essentially never used at any point, even if swappiness value is set
  very high. Relaunching the apps does not clear the memory and typically
  results in the apps immediately crashing as soon as more memory is
- needed.
+ needed. Furthermore, I am on kernel version "4.4.0-62-generic" which is
+ necessary in pointing out because a previous kernel version had a bug
+ involving memory and overactive OOM killer.
  
  ***However, the distinction for individual files is important because if
  many files are selected simultaneously and copied over, the memory
  buffer/cache is cleared systematically as the transfers are completed,
  preventing any crash. This is useful but still does not prevent the
  eventual accruing of memory/buffer space when accessing many large files
  on the drive over time.
  
  As far as I can tell, there is no way to automatically clear the
  buffer/cache without having a manual script that would have to be run
  every time. Also worth noting, I have spent sometime trying to nail down
  the issue and nearly positive that is has nothing to do with any other
  aspect of how Debian/the kernel manages memory because every memory
  tweak I have tried has had no effect, short of running the terminal
  command above to clear the buffer/cache.
  
  It seems that NTFS-3G needs to clear the memory/buffer after
  transferring individual files or reading them or run a check to clear it
  after a certain percentage is reached.

-- 
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to ntfs-3g in Ubuntu.
https://bugs.launchpad.net/bugs/1751675

Title:
  NTFS-3G Major Issue: Improper clearing of memory buffer/cache
  resulting in OOM Kills

Status in ntfs-3g package in Ubuntu:
  Confirmed

Bug description:
  I am running a modern gaming rig with Kubuntu 16.10 installed. 16GB
  ram, 32GB swap. I have an 8TB external that can only be formatted in
  NTFS (does not behave correctly when formatted any other way). When
  transferring INDIVIDUAL files to the drive (and reading files), the
  disk memory cache/buffer as according to the KInfoCenter slowly fills
  up and is not cleared. After transferring/reading enough INDIVIDUAL
  files the system runs out of memory and starts OOM killing every app
  that is currently launched, including the desktop environment after a
  time (in this case, Plasma). If I use the command

  "sudo sync && sudo echo 3 | sudo tee /proc/sys/vm/drop_caches"

  when the buffer/cache is nearly full, no crashes occur. SWAP is
  essentially never used at any point, even if swappiness value is set
  very high. Relaunching the apps does not clear the memory and
  typically results in the apps immediately crashing as soon as more
  memory is needed. Furthermore, I am on kernel version
  "4.4.0-62-generic" which is necessary in pointing out because a
  previous kernel version had a bug involving memory and overactive OOM
  killer.

  ***However, the distinction for individual files is important because
  if many files are selected simultaneously and copied over, the memory
  buffer/cache is cleared systematically as the transfers are completed,
  preventing any crash. This is useful but still does not prevent the
  eventual accruing of memory/buffer space when accessing many large
  files on the drive over time.

  As far as I can tell, there is no way to automatically clear the
  buffer/cache without having a manual script that would have to be run
  every time. Also worth noting, I have spent sometime trying to nail
  down the issue and nearly positive that is has nothing to do with any
  other aspect of how Debian/the kernel manages memory because every
  memory tweak I have tried has had no effect, short of running the
  terminal command above to clear the buffer/cache.

  It seems that NTFS-3G needs to clear the memory/buffer after
  transferring individual files or reading them or run a check to clear
  it after a certain percentage is reached.

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/ntfs-3g/+bug/1751675/+subscriptions