合 Docker中使用systemctl命令时报Too many open files错误
Tags: 故障处理DockerToo many open files
自己的Docker环境
在docker的容器内使用systemctl启动服务时,报错:
1 2 3 | [root@lhrorchestrator /]# systemctl start mysql Error: Too many open files Authorization not available. Check if polkit service is running or see debug message for more information. |
这里不仅仅是mysql服务,任何服务使用systemctl启动都会报错,另外,这里绝对不是由于系统文件open-files达到最大数引起的。
查了好几天的原因,最后使用journalctl -xe命令查看日志,突然发现了一个不一样的错误:“inotify_init1() failed: Too many open files”
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | [root@lhrorchestrator /]# journalctl -xe Mar 03 12:15:53 lhrorchestrator systemd[1]: inotify_init1() failed: Too many open files Mar 03 12:15:53 lhrorchestrator systemd[1]: Unit systemd-udevd.service entered failed state. Mar 03 12:15:53 lhrorchestrator systemd[1]: systemd-udevd.service failed. Mar 03 12:15:53 lhrorchestrator systemd[1]: systemd-udevd.service has no holdoff time, scheduling restart. Mar 03 12:15:53 lhrorchestrator systemd[1]: Stopped udev Kernel Device Manager. -- Subject: Unit systemd-udevd.service has finished shutting down -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit systemd-udevd.service has finished shutting down. Mar 03 12:15:53 lhrorchestrator systemd[1]: inotify_init1() failed: Too many open files Mar 03 12:15:53 lhrorchestrator systemd[1]: start request repeated too quickly for systemd-udevd.service Mar 03 12:15:53 lhrorchestrator systemd[1]: Failed to start udev Kernel Device Manager. -- Subject: Unit systemd-udevd.service has failed -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit systemd-udevd.service has failed. -- -- The result is failed. Mar 03 12:15:53 lhrorchestrator systemd[1]: inotify_init1() failed: Too many open files Mar 03 12:15:53 lhrorchestrator systemd[1]: inotify_init1() failed: Too many open files Mar 03 12:15:53 lhrorchestrator systemd[1]: Unit systemd-udevd.service entered failed state. Mar 03 12:15:53 lhrorchestrator systemd[1]: systemd-udevd.service failed. |
百度搜索“inotify_init1() failed: Too many open files”:
notify是linux提供的一种监控机制,可以监控文件系统的变化。该机制受到2个内核参数的影响:“fs.inotify.max_user_instances”和“fs.inotify.max_user_watches”,其中“fs.inotify.max_user_instances”表示每个用户最多可以创建的inotify instances数量上限,“fs.inotify.max_user_watches”表示每个用户同时可以添加的watch数目,当出现too many open files问题而上面三种方法都无法解决时,可以尝试通过修改这2个内核参数来生效。修改方法是修改"/etc/sysctl.conf"文件,并执行"sysctl -p"。
所以,解决很简单:
1 2 3 4 5 6 7 8 9 | -- 修改内核参数,避免Error: Too many open files错误 cat >> /etc/sysctl.conf << "EOF" fs.file-max=9000000 fs.inotify.max_user_instances = 1000000 fs.inotify.max_user_watches = 1000000 EOF sysctl -p |
再 执行"sysctl -p"生效即可。
Too many open files的四种解决办法
1、单个进程打开文件句柄数过多
ulimit中的nofile表示单进程可以打开的最大文件句柄数,可以通过ulimit -a查看,子进程默认继承父进程的限制(注意,是继承,不是共享,子进程和父进程打开的文件句柄数是单独算的)。