解决 k3s Docker 后端 syslog 日志泛滥问题

解决 k3s Docker 后端 syslog 日志泛滥问题

1 问题信息

使用了 Docker 后端的 k3s 在运行过程中出现了大量不明日志:

Feb 22 15:37:02 entry-cn2 systemd[1]: run-docker-runtimex2drunc-moby-08443a6d4bb55d16658928101aacfb21e244a7d6d4968d32414a776f33fe055a-runc.oeUsHV.mount: Deactivated successfully.
Feb 22 15:37:03 entry-cn2 k3s[2610804]: time="2023-02-22T15:37:03+08:00" level=info msg="Using CNI configuration file /var/lib/rancher/k3s/agent/etc/cni/net.d/10-flannel.conflist"
Feb 22 15:37:04 entry-cn2 systemd[1]: run-docker-runtimex2drunc-moby-f080461821abae083e21120cfc237424b7e01cbf8e947323c98a1f4e227ae8ac-runc.fhtmBb.mount: Deactivated successfully.
Feb 22 15:37:08 entry-cn2 k3s[2610804]: time="2023-02-22T15:37:08+08:00" level=info msg="Using CNI configuration file /var/lib/rancher/k3s/agent/etc/cni/net.d/10-flannel.conflist"
Feb 22 15:37:09 entry-cn2 systemd[1]: run-docker-runtimex2drunc-moby-f080461821abae083e21120cfc237424b7e01cbf8e947323c98a1f4e227ae8ac-runc.XO6q4d.mount: Deactivated successfully.
Feb 22 15:37:13 entry-cn2 k3s[2610804]: time="2023-02-22T15:37:13+08:00" level=info msg="Using CNI configuration file /var/lib/rancher/k3s/agent/etc/cni/net.d/10-flannel.conflist"
Feb 22 15:37:14 entry-cn2 systemd[1]: run-docker-runtimex2drunc-moby-f080461821abae083e21120cfc237424b7e01cbf8e947323c98a1f4e227ae8ac-runc.WpvoB7.mount: Deactivated successfully.
Feb 22 15:37:14 entry-cn2 systemd[1]: run-docker-runtimex2drunc-moby-f080461821abae083e21120cfc237424b7e01cbf8e947323c98a1f4e227ae8ac-runc.y30O0g.mount: Deactivated successfully.
Feb 22 15:37:17 entry-cn2 systemd[1]: run-docker-runtimex2drunc-moby-08443a6d4bb55d16658928101aacfb21e244a7d6d4968d32414a776f33fe055a-runc.5DmUFy.mount: Deactivated successfully.
Feb 22 15:37:18 entry-cn2 k3s[2610804]: time="2023-02-22T15:37:18+08:00" level=info msg="Using CNI configuration file /var/lib/rancher/k3s/agent/etc/cni/net.d/10-flannel.conflist"
Feb 22 15:37:19 entry-cn2 systemd[1]: run-docker-runtimex2drunc-moby-f080461821abae083e21120cfc237424b7e01cbf8e947323c98a1f4e227ae8ac-runc.ceBNp0.mount: Deactivated successfully.
Feb 22 15:37:20 entry-cn2 systemd[1]: run-docker-runtimex2drunc-moby-f080461821abae083e21120cfc237424b7e01cbf8e947323c98a1f4e227ae8ac-runc.923MQk.mount: Deactivated successfully.
Feb 22 15:37:23 entry-cn2 k3s[2610804]: time="2023-02-22T15:37:23+08:00" level=info msg="Using CNI configuration file /var/lib/rancher/k3s/agent/etc/cni/net.d/10-flannel.conflist"
Feb 22 15:37:24 entry-cn2 systemd[1]: run-docker-runtimex2drunc-moby-f080461821abae083e21120cfc237424b7e01cbf8e947323c98a1f4e227ae8ac-runc.zifwM5.mount: Deactivated successfully.
Feb 22 15:37:28 entry-cn2 k3s[2610804]: time="2023-02-22T15:37:28+08:00" level=info msg="Using CNI configuration file /var/lib/rancher/k3s/agent/etc/cni/net.d/10-flannel.conflist"
Feb 22 15:37:29 entry-cn2 systemd[1]: run-docker-runtimex2drunc-moby-f080461821abae083e21120cfc237424b7e01cbf8e947323c98a1f4e227ae8ac-runc.9hcGFx.mount: Deactivated successfully.

或者:

Feb 22 02:45:50 node-jp systemd[1]: run-docker-runtimex2drunc-moby-a096fc426794fb53c491ddfe41c666b749d002d80796f411ca1e4cee9faf5596-runc.OjOem8.mount: Succeeded.
Feb 22 02:45:50 node-jp systemd[273787]: run-docker-runtimex2drunc-moby-a096fc426794fb53c491ddfe41c666b749d002d80796f411ca1e4cee9faf5596-runc.OjOem8.mount: Succeeded.
Feb 22 02:45:50 node-jp systemd[273787]: run-docker-runtimex2drunc-moby-a096fc426794fb53c491ddfe41c666b749d002d80796f411ca1e4cee9faf5596-runc.HJ0aIR.mount: Succeeded.
Feb 22 02:45:50 node-jp systemd[1]: run-docker-runtimex2drunc-moby-a096fc426794fb53c491ddfe41c666b749d002d80796f411ca1e4cee9faf5596-runc.HJ0aIR.mount: Succeeded.
Feb 22 02:45:51 node-jp k3s[4424]: time="2023-02-22T02:45:51-05:00" level=info msg="Using CNI configuration file /var/lib/rancher/k3s/agent/etc/cni/net.d/10-flannel.conflist"
Feb 22 02:45:55 node-jp systemd[1]: run-docker-runtimex2drunc-moby-a096fc426794fb53c491ddfe41c666b749d002d80796f411ca1e4cee9faf5596-runc.xmHreV.mount: Succeeded.
Feb 22 02:45:55 node-jp systemd[273787]: run-docker-runtimex2drunc-moby-a096fc426794fb53c491ddfe41c666b749d002d80796f411ca1e4cee9faf5596-runc.xmHreV.mount: Succeeded.
Feb 22 02:45:56 node-jp k3s[4424]: time="2023-02-22T02:45:56-05:00" level=info msg="Using CNI configuration file /var/lib/rancher/k3s/agent/etc/cni/net.d/10-flannel.conflist"
Feb 22 02:46:00 node-jp systemd[1]: run-docker-runtimex2drunc-moby-a096fc426794fb53c491ddfe41c666b749d002d80796f411ca1e4cee9faf5596-runc.vYBWRk.mount: Succeeded.
Feb 22 02:46:00 node-jp systemd[273787]: run-docker-runtimex2drunc-moby-a096fc426794fb53c491ddfe41c666b749d002d80796f411ca1e4cee9faf5596-runc.vYBWRk.mount: Succeeded.
Feb 22 02:46:01 node-jp k3s[4424]: time="2023-02-22T02:46:01-05:00" level=info msg="Using CNI configuration file /var/lib/rancher/k3s/agent/etc/cni/net.d/10-flannel.conflist"
Feb 22 02:46:05 node-jp systemd[273787]: run-docker-runtimex2drunc-moby-a096fc426794fb53c491ddfe41c666b749d002d80796f411ca1e4cee9faf5596-runc.JyRox6.mount: Succeeded.
Feb 22 02:46:05 node-jp systemd[1]: run-docker-runtimex2drunc-moby-a096fc426794fb53c491ddfe41c666b749d002d80796f411ca1e4cee9faf5596-runc.JyRox6.mount: Succeeded.

2 问题分析

共有两种类型日志刷屏:

run-docker-runtimex2drunc-moby-xxx-runc.xxx.mount: Deactivated successfully.
run-docker-runtimex2drunc-moby-xxx-runc.xxx.mount: Succeeded.

结合社区讨论,初步判断是 systemd 错误的记录了脏日志,可以使用 rsyslog 的过滤配置来进行过滤。

3 解决方法

  1. 创建或编辑过滤文件
    vim /etc/rsyslog.d/01-blocklist.conf
    		
  2. 填写如下内容:
    if $msg contains "run-docker-runtime" and $msg contains ".mount: Deactivated successfully." then {
    		   stop
    		} 
    		if $msg contains "run-docker-runtime" and $msg contains ".mount: Succeeded." then {
    		   stop
    		}
    		
  3. 重启 rsyslogd 服务
    systemctl restart rsyslog
    		

4 修复结果

可以观察到 syslog 中的脏日志已经被过滤:

5 参考文档

  1. https://stackoverflow.com/questions/63622619/docker-flooding-syslog-with-run-docker-runtime-logs
  2. https://www.xiewo.net/blog/show/481/
  3. https://www.claudiokuenzler.com/blog/1162/how-to-ignore-discard-certain-syslog-messages-rsyslogd-filters
© 版权声明
THE END
喜欢就支持一下吧
点赞7 分享
评论 抢沙发
头像
欢迎您留下宝贵的见解!
提交
头像

昵称

取消
昵称表情代码图片

    暂无评论内容