解决 k3s Docker 后端 syslog 日志泛滥问题-星域 Start

1 问题信息

使用了 Docker 后端的 k3s 在运行过程中出现了大量不明日志：

Feb 22 15:37:02 entry-cn2 systemd[1]: run-docker-runtimex2drunc-moby-08443a6d4bb55d16658928101aacfb21e244a7d6d4968d32414a776f33fe055a-runc.oeUsHV.mount: Deactivated successfully.
Feb 22 15:37:03 entry-cn2 k3s[2610804]: time="2023-02-22T15:37:03+08:00" level=info msg="Using CNI configuration file /var/lib/rancher/k3s/agent/etc/cni/net.d/10-flannel.conflist"
Feb 22 15:37:04 entry-cn2 systemd[1]: run-docker-runtimex2drunc-moby-f080461821abae083e21120cfc237424b7e01cbf8e947323c98a1f4e227ae8ac-runc.fhtmBb.mount: Deactivated successfully.
Feb 22 15:37:08 entry-cn2 k3s[2610804]: time="2023-02-22T15:37:08+08:00" level=info msg="Using CNI configuration file /var/lib/rancher/k3s/agent/etc/cni/net.d/10-flannel.conflist"
Feb 22 15:37:09 entry-cn2 systemd[1]: run-docker-runtimex2drunc-moby-f080461821abae083e21120cfc237424b7e01cbf8e947323c98a1f4e227ae8ac-runc.XO6q4d.mount: Deactivated successfully.
Feb 22 15:37:13 entry-cn2 k3s[2610804]: time="2023-02-22T15:37:13+08:00" level=info msg="Using CNI configuration file /var/lib/rancher/k3s/agent/etc/cni/net.d/10-flannel.conflist"
Feb 22 15:37:14 entry-cn2 systemd[1]: run-docker-runtimex2drunc-moby-f080461821abae083e21120cfc237424b7e01cbf8e947323c98a1f4e227ae8ac-runc.WpvoB7.mount: Deactivated successfully.
Feb 22 15:37:14 entry-cn2 systemd[1]: run-docker-runtimex2drunc-moby-f080461821abae083e21120cfc237424b7e01cbf8e947323c98a1f4e227ae8ac-runc.y30O0g.mount: Deactivated successfully.
Feb 22 15:37:17 entry-cn2 systemd[1]: run-docker-runtimex2drunc-moby-08443a6d4bb55d16658928101aacfb21e244a7d6d4968d32414a776f33fe055a-runc.5DmUFy.mount: Deactivated successfully.
Feb 22 15:37:18 entry-cn2 k3s[2610804]: time="2023-02-22T15:37:18+08:00" level=info msg="Using CNI configuration file /var/lib/rancher/k3s/agent/etc/cni/net.d/10-flannel.conflist"
Feb 22 15:37:19 entry-cn2 systemd[1]: run-docker-runtimex2drunc-moby-f080461821abae083e21120cfc237424b7e01cbf8e947323c98a1f4e227ae8ac-runc.ceBNp0.mount: Deactivated successfully.
Feb 22 15:37:20 entry-cn2 systemd[1]: run-docker-runtimex2drunc-moby-f080461821abae083e21120cfc237424b7e01cbf8e947323c98a1f4e227ae8ac-runc.923MQk.mount: Deactivated successfully.
Feb 22 15:37:23 entry-cn2 k3s[2610804]: time="2023-02-22T15:37:23+08:00" level=info msg="Using CNI configuration file /var/lib/rancher/k3s/agent/etc/cni/net.d/10-flannel.conflist"
Feb 22 15:37:24 entry-cn2 systemd[1]: run-docker-runtimex2drunc-moby-f080461821abae083e21120cfc237424b7e01cbf8e947323c98a1f4e227ae8ac-runc.zifwM5.mount: Deactivated successfully.
Feb 22 15:37:28 entry-cn2 k3s[2610804]: time="2023-02-22T15:37:28+08:00" level=info msg="Using CNI configuration file /var/lib/rancher/k3s/agent/etc/cni/net.d/10-flannel.conflist"
Feb 22 15:37:29 entry-cn2 systemd[1]: run-docker-runtimex2drunc-moby-f080461821abae083e21120cfc237424b7e01cbf8e947323c98a1f4e227ae8ac-runc.9hcGFx.mount: Deactivated successfully.

或者：

Feb 22 02:45:50 node-jp systemd[1]: run-docker-runtimex2drunc-moby-a096fc426794fb53c491ddfe41c666b749d002d80796f411ca1e4cee9faf5596-runc.OjOem8.mount: Succeeded.
Feb 22 02:45:50 node-jp systemd[273787]: run-docker-runtimex2drunc-moby-a096fc426794fb53c491ddfe41c666b749d002d80796f411ca1e4cee9faf5596-runc.OjOem8.mount: Succeeded.
Feb 22 02:45:50 node-jp systemd[273787]: run-docker-runtimex2drunc-moby-a096fc426794fb53c491ddfe41c666b749d002d80796f411ca1e4cee9faf5596-runc.HJ0aIR.mount: Succeeded.
Feb 22 02:45:50 node-jp systemd[1]: run-docker-runtimex2drunc-moby-a096fc426794fb53c491ddfe41c666b749d002d80796f411ca1e4cee9faf5596-runc.HJ0aIR.mount: Succeeded.
Feb 22 02:45:51 node-jp k3s[4424]: time="2023-02-22T02:45:51-05:00" level=info msg="Using CNI configuration file /var/lib/rancher/k3s/agent/etc/cni/net.d/10-flannel.conflist"
Feb 22 02:45:55 node-jp systemd[1]: run-docker-runtimex2drunc-moby-a096fc426794fb53c491ddfe41c666b749d002d80796f411ca1e4cee9faf5596-runc.xmHreV.mount: Succeeded.
Feb 22 02:45:55 node-jp systemd[273787]: run-docker-runtimex2drunc-moby-a096fc426794fb53c491ddfe41c666b749d002d80796f411ca1e4cee9faf5596-runc.xmHreV.mount: Succeeded.
Feb 22 02:45:56 node-jp k3s[4424]: time="2023-02-22T02:45:56-05:00" level=info msg="Using CNI configuration file /var/lib/rancher/k3s/agent/etc/cni/net.d/10-flannel.conflist"
Feb 22 02:46:00 node-jp systemd[1]: run-docker-runtimex2drunc-moby-a096fc426794fb53c491ddfe41c666b749d002d80796f411ca1e4cee9faf5596-runc.vYBWRk.mount: Succeeded.
Feb 22 02:46:00 node-jp systemd[273787]: run-docker-runtimex2drunc-moby-a096fc426794fb53c491ddfe41c666b749d002d80796f411ca1e4cee9faf5596-runc.vYBWRk.mount: Succeeded.
Feb 22 02:46:01 node-jp k3s[4424]: time="2023-02-22T02:46:01-05:00" level=info msg="Using CNI configuration file /var/lib/rancher/k3s/agent/etc/cni/net.d/10-flannel.conflist"
Feb 22 02:46:05 node-jp systemd[273787]: run-docker-runtimex2drunc-moby-a096fc426794fb53c491ddfe41c666b749d002d80796f411ca1e4cee9faf5596-runc.JyRox6.mount: Succeeded.
Feb 22 02:46:05 node-jp systemd[1]: run-docker-runtimex2drunc-moby-a096fc426794fb53c491ddfe41c666b749d002d80796f411ca1e4cee9faf5596-runc.JyRox6.mount: Succeeded.

2 问题分析

共有两种类型日志刷屏：

run-docker-runtimex2drunc-moby-xxx-runc.xxx.mount: Deactivated successfully.
run-docker-runtimex2drunc-moby-xxx-runc.xxx.mount: Succeeded.

结合社区讨论，初步判断是 systemd 错误的记录了脏日志，可以使用 rsyslog 的过滤配置来进行过滤。

3 解决方法

创建或编辑过滤文件
```
vim /etc/rsyslog.d/01-blocklist.conf
		
```

填写如下内容：

if $msg contains "run-docker-runtime" and $msg contains ".mount: Deactivated successfully." then {
		   stop
		} 
		if $msg contains "run-docker-runtime" and $msg contains ".mount: Succeeded." then {
		   stop
		}

重启 rsyslogd 服务
```
systemctl restart rsyslog
		
```

4 修复结果

可以观察到 syslog 中的脏日志已经被过滤：

5 参考文档

https://stackoverflow.com/questions/63622619/docker-flooding-syslog-with-run-docker-runtime-logs
https://www.xiewo.net/blog/show/481/
https://www.claudiokuenzler.com/blog/1162/how-to-ignore-discard-certain-syslog-messages-rsyslogd-filters

文章版权归作者所有，未经允许请勿转载。

THE END

踩坑记录
# 云原生 # K3S