k3s docker backend syslog flood with run-docker-runtime\x2drunc-moby-xxx-runc.xxx.mount: Deactivated successfully
问题信息
使用了 Docker 后端的 k3s 在运行过程中出现了大量不明日志：
Feb 22 15:37:02 entry-cn2 systemd[1]: run-docker-runtime\x2drunc-moby-08443a6d4bb55d16658928101aacfb21e244a7d6d4968d32414a776f33fe055a-runc.oeUsHV.mount: Deactivated successfully.
Feb 22 15:37:03 entry-cn2 k3s[2610804]: time="2023-02-22T15:37:03+08:00" level=info msg="Using CNI configuration file /var/lib/rancher/k3s/agent/etc/cni/net.d/10-flannel.conflist"
Feb 22 15:37:04 entry-cn2 systemd[1]: run-docker-runtime\x2drunc-moby-f080461821abae083e21120cfc237424b7e01cbf8e947323c98a1f4e227ae8ac-runc.fhtmBb.mount: Deactivated successfully.
Feb 22 15:37:08 entry-cn2 k3s[2610804]: time="2023-02-22T15:37:08+08:00" level=info msg="Using CNI configuration file /var/lib/rancher/k3s/agent/etc/cni/net.d/10-flannel.conflist"
Feb 22 15:37:09 entry-cn2 systemd[1]: run-docker-runtime\x2drunc-moby-f080461821abae083e21120cfc237424b7e01cbf8e947323c98a1f4e227ae8ac-runc.XO6q4d.mount: Deactivated successfully.
Feb 22 15:37:13 entry-cn2 k3s[2610804]: time="2023-02-22T15:37:13+08:00" level=info msg="Using CNI configuration file /var/lib/rancher/k3s/agent/etc/cni/net.d/10-flannel.conflist"
Feb 22 15:37:14 entry-cn2 systemd[1]: run-docker-runtime\x2drunc-moby-f080461821abae083e21120cfc237424b7e01cbf8e947323c98a1f4e227ae8ac-runc.WpvoB7.mount: Deactivated successfully.
Feb 22 15:37:14 entry-cn2 systemd[1]: run-docker-runtime\x2drunc-moby-f080461821abae083e21120cfc237424b7e01cbf8e947323c98a1f4e227ae8ac-runc.y30O0g.mount: Deactivated successfully.
Feb 22 15:37:17 entry-cn2 systemd[1]: run-docker-runtime\x2drunc-moby-08443a6d4bb55d16658928101aacfb21e244a7d6d4968d32414a776f33fe055a-runc.5DmUFy.mount: Deactivated successfully.
Feb 22 15:37:18 entry-cn2 k3s[2610804]: time="2023-02-22T15:37:18+08:00" level=info msg="Using CNI configuration file /var/lib/rancher/k3s/agent/etc/cni/net.d/10-flannel.conflist"
Feb 22 15:37:19 entry-cn2 systemd[1]: run-docker-runtime\x2drunc-moby-f080461821abae083e21120cfc237424b7e01cbf8e947323c98a1f4e227ae8ac-runc.ceBNp0.mount: Deactivated successfully.
Feb 22 15:37:20 entry-cn2 systemd[1]: run-docker-runtime\x2drunc-moby-f080461821abae083e21120cfc237424b7e01cbf8e947323c98a1f4e227ae8ac-runc.923MQk.mount: Deactivated successfully.
Feb 22 15:37:23 entry-cn2 k3s[2610804]: time="2023-02-22T15:37:23+08:00" level=info msg="Using CNI configuration file /var/lib/rancher/k3s/agent/etc/cni/net.d/10-flannel.conflist"
Feb 22 15:37:24 entry-cn2 systemd[1]: run-docker-runtime\x2drunc-moby-f080461821abae083e21120cfc237424b7e01cbf8e947323c98a1f4e227ae8ac-runc.zifwM5.mount: Deactivated successfully.
Feb 22 15:37:28 entry-cn2 k3s[2610804]: time="2023-02-22T15:37:28+08:00" level=info msg="Using CNI configuration file /var/lib/rancher/k3s/agent/etc/cni/net.d/10-flannel.conflist"
Feb 22 15:37:29 entry-cn2 systemd[1]: run-docker-runtime\x2drunc-moby-f080461821abae083e21120cfc237424b7e01cbf8e947323c98a1f4e227ae8ac-runc.9hcGFx.mount: Deactivated successfully.
或者：
Feb 22 02:45:50 node-jp systemd[1]: run-docker-runtime\x2drunc-moby-a096fc426794fb53c491ddfe41c666b749d002d80796f411ca1e4cee9faf5596-runc.OjOem8.mount: Succeeded.
Feb 22 02:45:50 node-jp systemd[273787]: run-docker-runtime\x2drunc-moby-a096fc426794fb53c491ddfe41c666b749d002d80796f411ca1e4cee9faf5596-runc.OjOem8.mount: Succeeded.
Feb 22 02:45:50 node-jp systemd[273787]: run-docker-runtime\x2drunc-moby-a096fc426794fb53c491ddfe41c666b749d002d80796f411ca1e4cee9faf5596-runc.HJ0aIR.mount: Succeeded.
Feb 22 02:45:50 node-jp systemd[1]: run-docker-runtime\x2drunc-moby-a096fc426794fb53c491ddfe41c666b749d002d80796f411ca1e4cee9faf5596-runc.HJ0aIR.mount: Succeeded.
Feb 22 02:45:51 node-jp k3s[4424]: time="2023-02-22T02:45:51-05:00" level=info msg="Using CNI configuration file /var/lib/rancher/k3s/agent/etc/cni/net.d/10-flannel.conflist"
Feb 22 02:45:55 node-jp systemd[1]: run-docker-runtime\x2drunc-moby-a096fc426794fb53c491ddfe41c666b749d002d80796f411ca1e4cee9faf5596-runc.xmHreV.mount: Succeeded.
Feb 22 02:45:55 node-jp systemd[273787]: run-docker-runtime\x2drunc-moby-a096fc426794fb53c491ddfe41c666b749d002d80796f411ca1e4cee9faf5596-runc.xmHreV.mount: Succeeded.
Feb 22 02:45:56 node-jp k3s[4424]: time="2023-02-22T02:45:56-05:00" level=info msg="Using CNI configuration file /var/lib/rancher/k3s/agent/etc/cni/net.d/10-flannel.conflist"
Feb 22 02:46:00 node-jp systemd[1]: run-docker-runtime\x2drunc-moby-a096fc426794fb53c491ddfe41c666b749d002d80796f411ca1e4cee9faf5596-runc.vYBWRk.mount: Succeeded.
Feb 22 02:46:00 node-jp systemd[273787]: run-docker-runtime\x2drunc-moby-a096fc426794fb53c491ddfe41c666b749d002d80796f411ca1e4cee9faf5596-runc.vYBWRk.mount: Succeeded.
Feb 22 02:46:01 node-jp k3s[4424]: time="2023-02-22T02:46:01-05:00" level=info msg="Using CNI configuration file /var/lib/rancher/k3s/agent/etc/cni/net.d/10-flannel.conflist"
Feb 22 02:46:05 node-jp systemd[273787]: run-docker-runtime\x2drunc-moby-a096fc426794fb53c491ddfe41c666b749d002d80796f411ca1e4cee9faf5596-runc.JyRox6.mount: Succeeded.
Feb 22 02:46:05 node-jp systemd[1]: run-docker-runtime\x2drunc-moby-a096fc426794fb53c491ddfe41c666b749d002d80796f411ca1e4cee9faf5596-runc.JyRox6.mount: Succeeded.
问题分析
共有两种类型日志刷屏：
run-docker-runtime\x2drunc-moby-xxx-runc.xxx.mount: Deactivated successfully.
run-docker-runtime\x2drunc-moby-xxx-runc.xxx.mount: Succeeded.
结合社区讨论，初步判断是 systemd 错误的记录了脏日志，可以使用 rsyslog 的过滤配置来进行过滤。
解决方法
- 创建或编辑过滤文件
vim /etc/rsyslog.d/01-blocklist.conf
- 填写如下内容：
if $msg contains "run-docker-runtime" and $msg contains ".mount: Deactivated successfully." then {
stop
}
if $msg contains "run-docker-runtime" and $msg contains ".mount: Succeeded." then {
stop
}
- 重启 rsyslogd 服务
systemctl restart rsyslog
修复结果
可以观察到 syslog 中的脏日志已经被过滤：
参考文档：