Skip to content

fix(gateway): kubelet fails with permission denied on /dev/kmsg under rootless Podman #457

@blues-man

Description

@blues-man

Bug Report

Description

Gateway deployment fails on systems using rootless Podman (e.g. Fedora with docker aliased to podman). The embedded K3s kubelet crashes immediately with a permission error accessing /dev/kmsg, which is not available inside rootless (user-namespaced) containers.

openshell doctor check passes — it validates the Docker/Podman version and socket but does not detect that the container runtime is rootless Podman, which lacks the privileges K3s requires.

Steps to Reproduce

  1. Install OpenShell on Fedora 43 (or any distro where docker is Podman via podman-docker)
  2. Run openshell sandbox create -- claude
  3. Gateway auto-starts and immediately fails

Observed Behavior

E0318 21:59:40.848321  117 kubelet.go:568] "Failed to create an oomWatcher
  (running in UserNS, Hint: enable KubeletInUserNamespace feature flag to
  ignore the error)" err="open /dev/kmsg: operation not permitted"
Error: failed to run Kubelet: failed to create kubelet: open /dev/kmsg: operation not permitted

The kubelet also fails to write /proc/self/oom_score_adj (permission denied), which is another symptom of running inside a user namespace without sufficient privileges.

Expected Behavior

Either:

  • The gateway starts successfully under rootless Podman (e.g. by enabling the KubeletInUserNamespace feature gate in the K3s configuration and mounting /dev/kmsg into the container), or
  • openshell doctor check detects rootless Podman and reports a clear error with guidance (e.g., suggesting podman machine with root, or using Docker Engine)

Diagnostic Output

openshell doctor check:

Docker ............. ok (version 5.7.1)
DOCKER_HOST ........ unix:///run/user/1000/podman/podman.sock
All checks passed.

openshell doctor logs --lines 50: (key lines)

I0318 21:59:40.848112  117 server.go:893] "Failed to ApplyOOMScoreAdj" err="write /proc/self/oom_score_adj: permission denied"
E0318 21:59:40.848321  117 kubelet.go:568] "Failed to create an oomWatcher (running in UserNS, Hint: enable KubeletInUserNamespace feature flag to ignore the error)" err="open /dev/kmsg: operation not permitted"
Error: failed to run Kubelet: failed to create kubelet: open /dev/kmsg: operation not permitted
time="2026-03-18T21:59:40Z" level=error msg="Shutdown request received: kubelet exited: failed to run Kubelet: failed to create kubelet: open /dev/kmsg: operation not permitted"

Environment

Field Value
OS Fedora 43 (KDE), kernel 6.18.12-200.fc43.x86_64
Container runtime Podman 5.7.1 (rootless, via podman-docker)
DOCKER_HOST unix:///run/user/1000/podman/podman.sock
cgroup v2, systemd
OpenShell 0.0.10

Suggested Fix

  1. K3s config: Pass --kubelet-arg=feature-gates=KubeletInUserNamespace=true to the K3s server in the gateway container, and ensure /dev/kmsg is bind-mounted or a workaround is applied.
  2. Doctor check: Detect rootless Podman (e.g. check DOCKER_HOST for /run/user/ or run podman info --format '{{.Host.Security.Rootless}}') and warn the user before gateway deployment.

Signed-off-by: bluesman bluesman@users.noreply.github.com

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:clusterRelated to running OpenShell on k3s/docker

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions