[ubuntu-mono] [Bug 455884] Re: beagle package should modify fs.inotify.max_user_watches in /etc/sysctl.conf

[Jeremy Bicha]

Thank you for taking the time to report this bug. We are sorry that we
do not always have the capacity to look at all reported bugs in a timely
manner.

Beagle, the search software, is no longer included in Ubuntu. Unrelated
genetic analysis software named beagle has now been uploaded to Ubuntu's
archives for the next version of Ubuntu. Therefore, to avoid confusion,
I am closing this bug.

** Changed in: beagle (Ubuntu)
       Status: New => Invalid

-- 
You received this bug notification because you are a member of Ubuntu
CLI/Mono Uploaders, which is subscribed to beagle in Ubuntu.
https://bugs.launchpad.net/bugs/455884

Title:
  beagle package should modify fs.inotify.max_user_watches in
  /etc/sysctl.conf

Status in beagle package in Ubuntu:
  Invalid

Bug description:
  Binary package hint: beagle

  The need for Beagle users to increase the kernel's inotify watch limit
  has been discussed in many places.

  However, it is not really reasonable to ask ordinary users to edit
  /etc/sysctl.conf just because they use Beagle.  Especially because
  currently Beagle just logs obscure ioctl errors when the limit is
  exceeded and otherwise silently fails to index part of the file
  system.

  *********************************************************************************
   Some completely automatic solution is needed so that Beagle "just works".
  *********************************************************************************

  Here are some ideas (all completely speculative on my part!) :

  TEMP FIX #1:  Make the beagle package install scripts modify
  /etc/sysctl.conf to set fs.inotify.max_user_watches to a huge value
  "high enough for almost everybody" (256K?  512K?).  This is not a good
  solution, but may be good enough.

  I've read posts which say that kernel memory is only actually
  allocated in proportion to the number of directories watched; if that
  is correct then the max_user_watches limit has no useful purpose
  except to prevent using up so much memory that the system can not
  operate efficiently -- in which case Beagle can not scale to work on
  such a system.

  TEMP FIX #2: Instead of modifying /etc/sysctl.conf at install time,
  make beagled (via a suid-root helper program) dynamically set the
  limit in the running kernel.   This avoids hassles with the mods to
  /etc/sysctl.conf being lost during a system upgrade.  Also the limit
  could be set as a percentage of system ram, which would make choosing
  the limit somewhat easier.

  
  PERMANENT FIX #1: This is really a kernel design bug, because the kernel should not require locked memory proportional to the size of disk storage (which is apparently the case when Beagle is indexing "/").   This is a fundamental scalability problem.

  Instead, the kernel could keep a MRU cache of 'hot' directory nodes,
  with the complete watch list stored in user space, with a daemon
  process managing everything.  Cached watch info in the kernel would
  include entries for not-watched dirs (and perhaps whole sub-trees), so
  that accesses to non-watched directories would not always "miss the
  cache" and notify the daemon.    Inotify notifications would be routed
  through the daemon when a "cache miss" occurred; the daemon would
  issue the inotify event if appropriate, and adjust the kernel's cache
  as it deemed appropriate.

  PERM FIX #2: Change the inotify API so that the kernel is allowed to
  notify applications about changes outside of the directories they have
  requested; this would allow the kernel to "merge" watch-list entries
  when the table became full, retaining only the nearest common ancestor
  dir.  It would then be up to each application, such as Beagle, to
  filter out notifications it was not really interested in.   This would
  make the max_user_watches limit be just a performance-tuning
  parameter, not a functional limit.

  This might not be possible if directories can be opened by means other than a path-search, because in such cases there would be no way to know if a dir was under a sub-tree being watched.
  However if dirs are always first read into ram via path, e.g. as part of opendir, then two bits per in-memory inode would be enough storage (in addition to the watched-dir table). These 'watch state' bits ("this inode only", "this and all descendents", or "not watched") would be enough to propagate inclusive watches when relative paths were used to open subdirectories.

  ProblemType: Bug
  Architecture: amd64
  Date: Mon Oct 19 14:25:09 2009
  DistroRelease: Ubuntu 9.10
  ExecutablePath: /usr/lib/beagle/BeagleDaemon.exe
  InterpreterPath: /usr/bin/mono
  Package: beagle 0.3.9-3ubuntu1
  ProcEnviron:
   SHELL=/bin/bash
   PATH=(custom, user)
   LANG=en_US.UTF-8
  ProcVersionSignature: Ubuntu 2.6.31-12.41-generic
  SourcePackage: beagle
  Uname: Linux 2.6.31-12-generic x86_64
  XsessionErrors:
   (gnome-settings-daemon:1942): GLib-CRITICAL **: g_propagate_error: assertion `src != NULL' failed
   (gnome-settings-daemon:1942): GLib-CRITICAL **: g_propagate_error: assertion `src != NULL' failed
   (polkit-gnome-authentication-agent-1:2015): GLib-CRITICAL **: g_once_init_leave: assertion `initialization_value != 0' failed
   (nautilus:1977): Eel-CRITICAL **: eel_preferences_get_boolean: assertion `preferences_is_initialized ()' failed

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/beagle/+bug/455884/+subscriptions