Does selinux work with kernel namespaces?

Discussion:

Matt Callaway

2018-02-07 21:10:04 UTC

Hello,

I am attempting to run Docker on CentOS 7.4 with selinux and kernel
namespaces enabled. When I do so I observe an error that leads me to
an issue filed in github and a kernel patch that suggests that the
cause should be fixed in kernel 4.11+. Yet I cannot run docker
containers in this fashion on a 4.15 kernel.

Should docker with selinux-enabled work on a 4.15.1 kernel on CentOS
7.4 with namespaces enabled?

This might be a docker question, but the details I'll present below
suggest it might be more appropriate for this forum.

Details about the host and environment:

[***@localhost ~]# uname -r
4.15.1-1.el7.elrepo.x86_64

[***@localhost ~]# cat /etc/redhat-release
CentOS Linux release 7.4.1708 (Core)

[***@localhost ~]# docker --version
Docker version 17.12.0-ce, build c97c6d6

This is the latest docker-ce package from Docker's repository:

[***@localhost ~]# repoquery -i docker-ce

Name : docker-ce
Version : 17.12.0.ce
Release : 1.el7.centos
Architecture: x86_64
Size : 128453687
Packager : Docker <***@docker.com>
Group : Tools/Docker
URL : https://www.docker.com
Repository : docker-ce-stable
Summary : The open-source application container engine
Source : docker-ce-17.12.0.ce-1.el7.centos.src.rpm

The kernel is 4.15.1 from ElRepo, because that seems to be the
accepted way to get a 4.x kernel on CentOS, which I did because data
suggested I needed at least 4.11+

[***@localhost ~]# repoquery -i kernel-ml

Name : kernel-ml
Version : 4.15.1
Release : 1.el7.elrepo
Architecture: x86_64
Size : 204626242
Packager : Alan Bartlett <***@elrepo.org>
Group : System Environment/Kernel
URL : https://www.kernel.org/
Repository : elrepo-kernel
Summary : The Linux kernel. (The core of any Linux-based operating system.)
Source : kernel-ml-4.15.1-1.el7.elrepo.src.rpm

Here we see selinux-enabled is true and userns-remap is set to default:

[***@localhost ~]# cat /etc/docker/daemon.json
{
"debug": true,
"selinux-enabled": true,
"userns-remap": "default"
}

[***@localhost ~]# docker info 2>&1 | grep -A3 Security
Security Options:
seccomp
Profile: default
selinux

So when I try it I get:

[***@localhost ~]# docker run hello-world
docker: Error response from daemon: OCI runtime create failed:
container_linux.go:296: starting container process caused
"process_linux.go:301: running exec setns process for init caused
\"exit status 40\"": unknown.

When running in Permissive mode I see a different error:

[***@localhost ~]# setenforce 0

[***@localhost ~]# docker run hello-world
docker: Error response from daemon: OCI runtime create failed:
container_linux.go:296: starting container process caused
"process_linux.go:398: container init caused \"rootfs_linux.go:58:
mounting \\\"devpts\\\" to rootfs
\\\"/var/lib/docker/1001.1001/overlay2/6798981d1cf925a748187e0f2e9151f47bca9352457aa5b933a2bcb55eff9570/merged\\\"
at \\\"/var/lib/docker/1001.1001/overlay2/6798981d1cf925a748187e0f2e9151f47bca9352457aa5b933a2bcb55eff9570/merged/dev/pts\\\"
caused \\\"invalid argument\\\"\"": unknown.

Looking around for these symptoms I find these references...

First message goes here:

https://github.com/moby/moby/issues/35336

Suggests the namespace.unpriv_enable=1 flag. I have already enabled that:

[***@localhost ~]# grep unpriv /boot/grub2/grub.cfg
linux16 /vmlinuz-4.15.1-1.el7.elrepo.x86_64
root=/dev/mapper/VolGroup00-LogVol00 ro no_timer_check console=tty0
console=ttyS0,115200n8 net.ifnames=0 biosdevname=0 crashkernel=auto
rd.lvm.lv=VolGroup00/LogVol00 rd.lvm.lv=VolGroup00/LogVol01 rhgb quiet
LANG=en_US.UTF-8 namespace.unpriv_enable=1

Then I do setenforce 0 and we get the second devpts error which leads to:

https://github.com/opencontainers/runc/issues/1215

which leads to:

https://bugzilla.redhat.com/show_bug.cgi?id=1401537

which leads to a kernel patch:

https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/?id=01593d3299a1cfdb5e08acf95f63ec59dd674906

I've since discovered via the author that that patch was included in
kernel 4.11.

So... what am I missing? All signs suggest that this *should* work,
and yet does not.

Thank you for your time.

Stephen Smalley

2018-02-08 14:09:02 UTC

Permalink

Post by Matt Callaway
Hello,
I am attempting to run Docker on CentOS 7.4 with selinux and kernel
namespaces enabled. When I do so I observe an error that leads me to
an issue filed in github and a kernel patch that suggests that the
cause should be fixed in kernel 4.11+. Yet I cannot run docker
containers in this fashion on a 4.15 kernel.
Should docker with selinux-enabled work on a 4.15.1 kernel on CentOS
7.4 with namespaces enabled?

(cc Dan Walsh for insight on docker)

I would say yes, although it seems you are using third party packages
for both your kernel and your docker rather than the CentOS packages,
and those are coming from two different sources, so it is hard to know
how those might interact. Did you actually try just using the stock
CentOS kernel and docker first?

Post by Matt Callaway
This might be a docker question, but the details I'll present below
suggest it might be more appropriate for this forum.
4.15.1-1.el7.elrepo.x86_64
CentOS Linux release 7.4.1708 (Core)
Docker version 17.12.0-ce, build c97c6d6

Why are you using this package rather than the docker provided by
CentOS? Do you need specific functionality it provides that is lacking
in CentOS?

Post by Matt Callaway
Name : docker-ce
Version : 17.12.0.ce
Release : 1.el7.centos
Architecture: x86_64
Size : 128453687
Group : Tools/Docker
URL : https://www.docker.com
Repository : docker-ce-stable
Summary : The open-source application container engine
Source : docker-ce-17.12.0.ce-1.el7.centos.src.rpm
The kernel is 4.15.1 from ElRepo, because that seems to be the
accepted way to get a 4.x kernel on CentOS, which I did because data
suggested I needed at least 4.11+

By "data", do you mean the fact that the SELinux patch you cite below
wasn't included until 4.11? But that patch in turn was a fix for a
patch that was first included in 4.8 (so it wasn't necessary/relevant
for kernels < 4.8, e.g. the stock CentOS kernel). And all of that is
only relevant if performing context mounts within user namespaces.

Post by Matt Callaway
Name : kernel-ml
Version : 4.15.1
Release : 1.el7.elrepo
Architecture: x86_64
Size : 204626242
Group : System Environment/Kernel
URL : https://www.kernel.org/
Repository : elrepo-kernel
Summary : The Linux kernel. (The core of any Linux-based
operating system.)
Source : kernel-ml-4.15.1-1.el7.elrepo.src.rpm
Here we see selinux-enabled is true and userns-remap is set to
{
"debug": true,
"selinux-enabled": true,
"userns-remap": "default"
}
seccomp
Profile: default
selinux
container_linux.go:296: starting container process caused
"process_linux.go:301: running exec setns process for init caused
\"exit status 40\"": unknown.

Since you get the same error while enforcing but a different error in
permissive, that implies that there is a SELinux permission denial when
enforcing. You should be able to see the specific denial by checking
journalctl for messages with "avc: denied" (or if running auditd,
ausearch -m AVC,SELINUX_ERR). However, since you still get an error
while permissive, that implies that there is still a separate error
that is not a SELinux permission denial.

Post by Matt Callaway
container_linux.go:296: starting container process caused
mounting \\\"devpts\\\" to rootfs
\\\"/var/lib/docker/1001.1001/overlay2/6798981d1cf925a748187e0f2e9151
f47bca9352457aa5b933a2bcb55eff9570/merged\\\"
at
\\\"/var/lib/docker/1001.1001/overlay2/6798981d1cf925a748187e0f2e9151
f47bca9352457aa5b933a2bcb55eff9570/merged/dev/pts\\\"
caused \\\"invalid argument\\\"\"": unknown.

You got an "invalid argument" from the mount, or EINVAL.
The patch you cited was instead fixing a case where EACCES (Permission
denied) was being returned and instead allowing that to happen for
specific whitelisted filesystem types. The EINVAL cases in the SELinux
mount option processing code should have caused a kernel warning to be
logged in your dmesg or journalctl output with a SELinux: prefix.
Absent that, I'm not sure SELinux is the culprit for this error. A
quick way to check would be to disable SELinux altogether (edit
/etc/selinux/config to specify SELINUX=disabled by hand or via system-
config-selinux and reboot) and retry.

Post by Matt Callaway
Looking around for these symptoms I find these references...
https://github.com/moby/moby/issues/35336
linux16 /vmlinuz-4.15.1-1.el7.elrepo.x86_64
root=/dev/mapper/VolGroup00-LogVol00 ro no_timer_check console=tty0
console=ttyS0,115200n8 net.ifnames=0 biosdevname=0 crashkernel=auto
rd.lvm.lv=VolGroup00/LogVol00 rd.lvm.lv=VolGroup00/LogVol01 rhgb quiet
LANG=en_US.UTF-8 namespace.unpriv_enable=1
https://github.com/opencontainers/runc/issues/1215
https://bugzilla.redhat.com/show_bug.cgi?id=1401537
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/c
ommit/?id=01593d3299a1cfdb5e08acf95f63ec59dd674906
I've since discovered via the author that that patch was included in
kernel 4.11.
So... what am I missing? All signs suggest that this *should* work,
and yet does not.

Can we back up? Do you truly need to use docker-ce instead of the
CentOS docker? If not, don't. If so, what happened when you tried
using docker-ce on the stock CentOS kernel? And can you confirm that
it is in fact SELinux that is preventing the operation above from
succeeding, as described above by disabling SELinux and retrying?

Daniel Walsh

2018-02-08 14:12:13 UTC

Permalink

Not sure what you mean by Kernel Namespace, are you talking about User
Namespace?

Post by Matt Callaway
Should docker with selinux-enabled work on a 4.15.1 kernel on CentOS
7.4 with namespaces enabled?

Yes.

Post by Matt Callaway
This might be a docker question, but the details I'll present below
suggest it might be more appropriate for this forum.

What AVC messages are you seeing?

Post by Matt Callaway
4.15.1-1.el7.elrepo.x86_64
CentOS Linux release 7.4.1708 (Core)
Docker version 17.12.0-ce, build c97c6d6
Name : docker-ce
Version : 17.12.0.ce
Release : 1.el7.centos
Architecture: x86_64
Size : 128453687
Group : Tools/Docker
URL : https://www.docker.com
Repository : docker-ce-stable
Summary : The open-source application container engine
Source : docker-ce-17.12.0.ce-1.el7.centos.src.rpm
The kernel is 4.15.1 from ElRepo, because that seems to be the
accepted way to get a 4.x kernel on CentOS, which I did because data
suggested I needed at least 4.11+
Name : kernel-ml
Version : 4.15.1
Release : 1.el7.elrepo
Architecture: x86_64
Size : 204626242
Group : System Environment/Kernel
URL : https://www.kernel.org/
Repository : elrepo-kernel
Summary : The Linux kernel. (The core of any Linux-based operating system.)
Source : kernel-ml-4.15.1-1.el7.elrepo.src.rpm
{
"debug": true,
"selinux-enabled": true,
"userns-remap": "default"
}
seccomp
Profile: default
selinux
container_linux.go:296: starting container process caused
"process_linux.go:301: running exec setns process for init caused
\"exit status 40\"": unknown.
container_linux.go:296: starting container process caused
mounting \\\"devpts\\\" to rootfs
\\\"/var/lib/docker/1001.1001/overlay2/6798981d1cf925a748187e0f2e9151f47bca9352457aa5b933a2bcb55eff9570/merged\\\"
at \\\"/var/lib/docker/1001.1001/overlay2/6798981d1cf925a748187e0f2e9151f47bca9352457aa5b933a2bcb55eff9570/merged/dev/pts\\\"
caused \\\"invalid argument\\\"\"": unknown.
Looking around for these symptoms I find these references...
https://github.com/moby/moby/issues/35336
linux16 /vmlinuz-4.15.1-1.el7.elrepo.x86_64
root=/dev/mapper/VolGroup00-LogVol00 ro no_timer_check console=tty0
console=ttyS0,115200n8 net.ifnames=0 biosdevname=0 crashkernel=auto
rd.lvm.lv=VolGroup00/LogVol00 rd.lvm.lv=VolGroup00/LogVol01 rhgb quiet
LANG=en_US.UTF-8 namespace.unpriv_enable=1
https://github.com/opencontainers/runc/issues/1215
https://bugzilla.redhat.com/show_bug.cgi?id=1401537
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/?id=01593d3299a1cfdb5e08acf95f63ec59dd674906
I've since discovered via the author that that patch was included in
kernel 4.11.
So... what am I missing? All signs suggest that this *should* work,
and yet does not.
Thank you for your time.

Matt Callaway

2018-02-10 01:19:54 UTC

Permalink

I joined the selinux list and sent the above, but I have not seen it posted.

I rebuilt my test system with stock centos 7.4 with the docker that it
comes with and the kernel that it comes with. It runs fine before
enabling userns-remap:

[***@localhost ~]# uname -r
3.10.0-693.17.1.el7.x86_64
[***@localhost ~]# cat /etc/redhat-release
CentOS Linux release 7.4.1708 (Core)
[***@localhost ~]# docker --version
Docker version 1.12.6, build 3e8e77d/1.12.6
[***@localhost ~]# grep 3.10.0-693.17.1.el7.x86_64 /boot/grub2/grub.cfg
menuentry 'CentOS Linux (3.10.0-693.17.1.el7.x86_64) 7 (Core)' --class
centos --class gnu-linux --class gnu --class os --unrestricted
$menuentry_id_option
'gnulinux-3.10.0-693.11.6.el7.x86_64-advanced-3ac2b526-6c37-46f7-8539-67bc4e55dd49'
{
linux16 /vmlinuz-3.10.0-693.17.1.el7.x86_64
root=/dev/mapper/VolGroup00-LogVol00 ro no_timer_check console=tty0
console=ttyS0,115200n8 net.ifnames=0 biosdevname=0 crashkernel=auto
rd.lvm.lv=VolGroup00/LogVol00 rd.lvm.lv=VolGroup00/LogVol01 rhgb quiet
LANG=en_US.UTF-8 user_namespace.enable=1 namespace.unpriv_enable=1
initrd16 /initramfs-3.10.0-693.17.1.el7.x86_64.img
[***@localhost ~]# docker run hello-world | head -n2

Hello from Docker!

Then when I add userns-remap it fails:

[***@localhost ~]# cat /etc/docker/daemon.json
{
"userns-remap": "default"
}
[***@localhost ~]# systemctl restart docker
[***@localhost ~]# docker run hello-world
nsenter: unable to unshare namespaces: Invalid argument
container_linux.go:247: starting container process caused
"process_linux.go:245: running exec setns process for init caused
\"exit status 1\""
/usr/bin/docker-current: Error response from daemon: invalid header
field value "oci runtime error: container_linux.go:247: starting
container process caused \"process_linux.go:245: running exec setns
process for init caused \\\"exit status 1\\\"\"\n".

This symptom is reported here:

https://bugzilla.redhat.com/show_bug.cgi?id=1441993

CLOSED INSUFFICIENT_DATA

Mr. Walsh commented in that:

Daniel Walsh 2017-04-13 09:15:06 EDT

"Why would usernamespace be required for this? You might want to try
user namespace with docker-latest, but as of now we don't support user
namespace on RHEL."

But that was obviously several months ago.

Also similar here:

https://github.com/moby/moby/issues/25929

But that suggests adding:

user_namespace.enable=1

which I've done already:

[***@localhost ~]# grep user_namespace.enable /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.10.0-693.17.1.el7.x86_64
root=/dev/mapper/VolGroup00-LogVol00 ro no_timer_check console=tty0
console=ttyS0,115200n8 net.ifnames=0 biosdevname=0 crashkernel=auto
rd.lvm.lv=VolGroup00/LogVol00 rd.lvm.lv=VolGroup00/LogVol01 rhgb quiet
LANG=en_US.UTF-8 user_namespace.enable=1 namespace.unpriv_enable=1

I feel like this problem must be solved, but it doesn't appear solved
with the "stock" system.

Thoughts?

Matt Callaway

2018-02-10 01:50:25 UTC

Permalink

Then updating to docker-ce again via the docker instructions:

https://docs.docker.com/install/linux/docker-ce/centos/#set-up-the-repository

Then I get:

[***@localhost ~]# docker run hello-world
docker: Error response from daemon: OCI runtime create failed:
container_linux.go:296: starting container process caused
"process_linux.go:301: running exec setns process for init caused
\"exit status 40\"": unknown.

Which leads me to:

https://github.com/moby/moby/issues/35336

Which suggests that centos 7.4 doesn't support namespaces with docker,
but one might be able to test by adding:

namespace.unpriv_enable=1

I add that but still get:

[***@localhost ~]# setenforce 0
[***@localhost ~]# docker run hello-world
docker: Error response from daemon: OCI runtime create failed:
container_linux.go:296: starting container process caused
"process_linux.go:301: running exec setns process for init caused
\"exit status 40\"": unknown.

So maybe this just doesn't work yet.

Post by Matt Callaway
I joined the selinux list and sent the above, but I have not seen it posted.
I rebuilt my test system with stock centos 7.4 with the docker that it
comes with and the kernel that it comes with. It runs fine before
3.10.0-693.17.1.el7.x86_64
CentOS Linux release 7.4.1708 (Core)
Docker version 1.12.6, build 3e8e77d/1.12.6
menuentry 'CentOS Linux (3.10.0-693.17.1.el7.x86_64) 7 (Core)' --class
centos --class gnu-linux --class gnu --class os --unrestricted
$menuentry_id_option
'gnulinux-3.10.0-693.11.6.el7.x86_64-advanced-3ac2b526-6c37-46f7-8539-67bc4e55dd49'
{
linux16 /vmlinuz-3.10.0-693.17.1.el7.x86_64
root=/dev/mapper/VolGroup00-LogVol00 ro no_timer_check console=tty0
console=ttyS0,115200n8 net.ifnames=0 biosdevname=0 crashkernel=auto
rd.lvm.lv=VolGroup00/LogVol00 rd.lvm.lv=VolGroup00/LogVol01 rhgb quiet
LANG=en_US.UTF-8 user_namespace.enable=1 namespace.unpriv_enable=1
initrd16 /initramfs-3.10.0-693.17.1.el7.x86_64.img
Hello from Docker!
{
"userns-remap": "default"
}
nsenter: unable to unshare namespaces: Invalid argument
container_linux.go:247: starting container process caused
"process_linux.go:245: running exec setns process for init caused
\"exit status 1\""
/usr/bin/docker-current: Error response from daemon: invalid header
field value "oci runtime error: container_linux.go:247: starting
container process caused \"process_linux.go:245: running exec setns
process for init caused \\\"exit status 1\\\"\"\n".
https://bugzilla.redhat.com/show_bug.cgi?id=1441993
CLOSED INSUFFICIENT_DATA
Daniel Walsh 2017-04-13 09:15:06 EDT
"Why would usernamespace be required for this? You might want to try
user namespace with docker-latest, but as of now we don't support user
namespace on RHEL."
But that was obviously several months ago.
https://github.com/moby/moby/issues/25929
user_namespace.enable=1
BOOT_IMAGE=/vmlinuz-3.10.0-693.17.1.el7.x86_64
root=/dev/mapper/VolGroup00-LogVol00 ro no_timer_check console=tty0
console=ttyS0,115200n8 net.ifnames=0 biosdevname=0 crashkernel=auto
rd.lvm.lv=VolGroup00/LogVol00 rd.lvm.lv=VolGroup00/LogVol01 rhgb quiet
LANG=en_US.UTF-8 user_namespace.enable=1 namespace.unpriv_enable=1
I feel like this problem must be solved, but it doesn't appear solved
with the "stock" system.
Thoughts?

Daniel Walsh

2018-02-11 10:46:06 UTC

Permalink

Docker and container runtimes on RHEL7.4 is not fully supported. We are
hoping to change that in RHEL7.5

There are lots of updates going into the kernel to allow usernamespace
to work properly. It should work with SELinux at that time.

Post by Matt Callaway
https://docs.docker.com/install/linux/docker-ce/centos/#set-up-the-repository
container_linux.go:296: starting container process caused
"process_linux.go:301: running exec setns process for init caused
\"exit status 40\"": unknown.
https://github.com/moby/moby/issues/35336
Which suggests that centos 7.4 doesn't support namespaces with docker,
namespace.unpriv_enable=1
container_linux.go:296: starting container process caused
"process_linux.go:301: running exec setns process for init caused
\"exit status 40\"": unknown.
So maybe this just doesn't work yet.

Matt Callaway

2018-02-11 14:04:38 UTC

Permalink

Thank you.

Docker and container runtimes on RHEL7.4 is not fully supported. We are hoping to change that in RHEL7.5
There are lots of updates going into the kernel to allow usernamespace to work properly. It should work with SELinux at that time.