JQuery Venetian Blinds Transition Effect
A Simple ISAPI Filter for Authentication on IIS
Getting Set up with Ogre 3D on Ubuntu
Book Review: How to Implement Design Patterns in PHP
Symfony 2 Crash Course
Development Resource Project

inotify resources exhausted

Friday, 11 October 13, 12:12 pm
inotify is a part of the Linux kernel which watches the filesystem for changes. It is used for many different applications that need to react when there are changes to files in a specific location. For each location an application wants to monitor, it has to add an inotify watch, so apps can easily need a *lot* of these watches.

The system defines a maximum number of inotify watches that can be active at any one time. To see what is set on your system, run this command:
cat /proc/sys/fs/inotify/max_user_watches
When all the available resources are used up, inotify can no longer be used to tell apps about filesystem changes. This can result in error messages, or sometimes in apps failing to run at all, or in other cases, an app may simply fail to update correctly.

For instance, if you try to use tail to follow a log file, it will report the following:
$ tail -f /var/log/httpd/error_log tail: inotify resources exhausted tail: inotify cannot be used, reverting to polling  
However it will work as expected. Trying to start the pulseaudio daemon will fail however, with this somewhat snarky error:
E: [pulseaudio] module-udev-detect.c: You apparently ran out of inotify watches, probably because Tracker/Beagle took them all away. I wished people would do their homework first and fix inotify before using it for watching whole directory trees which is something the current inotify is certainly not useful for. Please make sure to drop the Tracker/Beagle guys a line complaining about their broken use of inotify. E: [pulseaudio] module-udev-detect.c: You apparently ran out of inotify watches, probably because Tracker/Beagle took them all away. I wished people would do their homework first and fix inotify before using it for watching whole directory trees which is something the current inotify is certainly not useful for. Please make sure to drop the Tracker/Beagle guys a line complaining about their broken use of inotify. E: [pulseaudio] module-systemd-login.c: Failed to create session monitor: No space left on device E: [pulseaudio] module.c: Failed to load module "module-systemd-login" (argument: ""): initialization failed. E: [pulseaudio] main.c: Module load failed. E: [pulseaudio] main.c: Failed to initialize daemon.
The nautilus filemanager however looks like it's running OK, however changes to files and directories are not reflected automatically. Skype also can fail to report on messages received.

You can view which processes are using inotify by running the following command:
find /proc/*/fd/* -type l -lname 'anon_inode:inotify' -print
The output here shows the process ID of each process as the number immediately following '/proc'. For instance, the process IDs below are 2702, 2714 and 312:
/proc/2702/fd/9 /proc/2714/fd/7 /proc/312/fd/5
The following bash script, taken from www.tldp.org/LDP/abs/html/procref1.html, determines which application corresponds to a process ID:
#!/bin/bash # pid-identifier.sh: # Gives complete path name to process associated with pid.   ARGNO=1 # Number of arguments the script expects. E_WRONGARGS=65 E_BADPID=66 E_NOSUCHPROCESS=67 E_NOPERMISSION=68 PROCFILE=exe   # Check correct number of arguments used if [ $# -ne $ARGNO ] then echo "Usage: `basename $0` PID-number" >&2 # Error message >stderr. exit $E_WRONGARGS fi   # Check for valid process ID pidno=$( ps ax | grep $1 | awk '{ print $1 }' | grep $1 )   if [ -z "$pidno" ] # zero-length string means no running process corresponds to the pid given then echo "No such process running." exit $E_NOSUCHPROCESS fi   # Check for read permission if [ ! -r "/proc/$1/$PROCFILE" ] then echo "Process $1 running, but..." echo "Can't get read permission on /proc/$1/$PROCFILE." exit $E_NOPERMISSION # Ordinary user can't access some files in /proc. fi   # Find the executable exe_file=$( ls -l /proc/$1 | grep "exe" | awk '{ print $11 }' )   if [ -e "$exe_file" ] # If /proc/pid-number/exe exists, then #+ then the corresponding process exists. echo "Process #$1 invoked by $exe_file." else echo "No such process running." fi   exit 0
Save this with a suitable name, make it executable, and run it passing in the process ID that you want to identify. This should give you some suspects for what's using up all your inotify watches.

To verify your suspicions, you can enable tracing and restart your suspect apps to see how many watches they are in fact using. It's easiest to do this from a root console rather than just with sudo. First, enable tracing of new inotify watches:
echo 1 >> /sys/kernel/debug/tracing/events/syscalls/sys_exit_inotify_add_watch/enable
Next make sure tracing is enabled:
cat /sys/kernel/debug/tracing/tracing_on
This should return 1, if not then make it do so:
echo 1 > /sys/kernel/debug/tracing/tracing_on
Now debug tracing is all turned on, close and restart the suspect apps that you identified previously. Take a look at the trace output by having a peek at /sys/kernel/debug/tracing/trace. In my case, my suspect app - Java - was gobbling up around 100,000 watches. I can't really blame it though, as it's netbeans doing this, because I routinely have 20 or so large projects open at once, and understandably, netbeans needs to be informed whenever external apps make changes to files in an open project.

So, after having determined what was using up my watches, I decided to increase my system's maximum quota by running this command as root:
echo fs.inotify.max_user_watches=32768 >> /etc/sysctl.d/99-sysctl.conf
Note that this assumes your running an up-to-date version of systemd. Versions prior to September 2013 stored user-defined settings in /etc/sysctl.conf.

Also note this won't take effect until the next boot. To change the value without rebooting, run this (also as root):
echo 32768 > /proc/sys/fs/inotify/max_user_watches
You can pretty much set this as high as you like, as each watch uses a very small amount of resources (in the order of a few tens of bytes).
 
Leave Comment

/xkcd/ Interstellar Asteroid

About This Page