The system defines a maximum number of inotify watches that can be active at any one time. To see what is set on your system, run this command:
cat /proc/sys/fs/inotify/max_user_watches
When all the available resources are used up, inotify can no longer be used to tell apps about filesystem changes. This can result in error messages, or sometimes in apps failing to run at all, or in other cases, an app may simply fail to update correctly.For instance, if you try to use tail to follow a log file, it will report the following:
$ tail -f /var/log/httpd/error_log
tail: inotify resources exhausted
tail: inotify cannot be used, reverting to polling
However it will work as expected. Trying to start the pulseaudio daemon will fail however, with this somewhat snarky error:E: [pulseaudio] module-udev-detect.c: You apparently ran out of inotify watches, probably because Tracker/Beagle took them all away. I wished people would do their homework first and fix inotify before using it for watching whole directory trees which is something the current inotify is certainly not useful for. Please make sure to drop the Tracker/Beagle guys a line complaining about their broken use of inotify.
E: [pulseaudio] module-udev-detect.c: You apparently ran out of inotify watches, probably because Tracker/Beagle took them all away. I wished people would do their homework first and fix inotify before using it for watching whole directory trees which is something the current inotify is certainly not useful for. Please make sure to drop the Tracker/Beagle guys a line complaining about their broken use of inotify.
E: [pulseaudio] module-systemd-login.c: Failed to create session monitor: No space left on device
E: [pulseaudio] module.c: Failed to load module "module-systemd-login" (argument: ""): initialization failed.
E: [pulseaudio] main.c: Module load failed.
E: [pulseaudio] main.c: Failed to initialize daemon.
The nautilus filemanager however looks like it's running OK, however changes to files and directories are not reflected automatically. Skype also can fail to report on messages received. You can view which processes are using inotify by running the following command:
find /proc/*/fd/* -type l -lname 'anon_inode:inotify' -print
The output here shows the process ID of each process as the number immediately following '/proc'. For instance, the process IDs below are 2702, 2714 and 312:/proc/2702/fd/9
/proc/2714/fd/7
/proc/312/fd/5
The following bash script, taken from www.tldp.org/LDP/abs/html/procref1.html, determines which application corresponds to a process ID:#!/bin/bash
# pid-identifier.sh:
# Gives complete path name to process associated with pid.
ARGNO=1 # Number of arguments the script expects.
E_WRONGARGS=65
E_BADPID=66
E_NOSUCHPROCESS=67
E_NOPERMISSION=68
PROCFILE=exe
# Check correct number of arguments used
if [ $# -ne $ARGNO ]
then
echo "Usage: `basename $0` PID-number" >&2 # Error message >stderr.
exit $E_WRONGARGS
fi
# Check for valid process ID
pidno=$( ps ax | grep $1 | awk '{ print $1 }' | grep $1 )
if [ -z "$pidno" ] # zero-length string means no running process corresponds to the pid given
then
echo "No such process running."
exit $E_NOSUCHPROCESS
fi
# Check for read permission
if [ ! -r "/proc/$1/$PROCFILE" ]
then
echo "Process $1 running, but..."
echo "Can't get read permission on /proc/$1/$PROCFILE."
exit $E_NOPERMISSION # Ordinary user can't access some files in /proc.
fi
# Find the executable
exe_file=$( ls -l /proc/$1 | grep "exe" | awk '{ print $11 }' )
if [ -e "$exe_file" ] # If /proc/pid-number/exe exists,
then #+ then the corresponding process exists.
echo "Process #$1 invoked by $exe_file."
else
echo "No such process running."
fi
exit 0
Save this with a suitable name, make it executable, and run it passing in the process ID that you want to identify. This should give you some suspects for what's using up all your inotify watches.To verify your suspicions, you can enable tracing and restart your suspect apps to see how many watches they are in fact using. It's easiest to do this from a root console rather than just with sudo. First, enable tracing of new inotify watches:
echo 1 >> /sys/kernel/debug/tracing/events/syscalls/sys_exit_inotify_add_watch/enable
Next make sure tracing is enabled:cat /sys/kernel/debug/tracing/tracing_on
This should return 1, if not then make it do so:echo 1 > /sys/kernel/debug/tracing/tracing_on
Now debug tracing is all turned on, close and restart the suspect apps that you identified previously. Take a look at the trace output by having a peek at /sys/kernel/debug/tracing/trace. In my case, my suspect app - Java - was gobbling up around 100,000 watches. I can't really blame it though, as it's netbeans doing this, because I routinely have 20 or so large projects open at once, and understandably, netbeans needs to be informed whenever external apps make changes to files in an open project.So, after having determined what was using up my watches, I decided to increase my system's maximum quota by running this command as root:
echo fs.inotify.max_user_watches=32768 >> /etc/sysctl.d/99-sysctl.conf
Note that this assumes your running an up-to-date version of systemd. Versions prior to September 2013 stored user-defined settings in /etc/sysctl.conf.Also note this won't take effect until the next boot. To change the value without rebooting, run this (also as root):
echo 32768 > /proc/sys/fs/inotify/max_user_watches
You can pretty much set this as high as you like, as each watch uses a very small amount of resources (in the order of a few tens of bytes). 