If you are reporting a new issue, make sure that we do not have any duplicates already open. You can ensure this by searching the issue list for this repository. If there is a duplicate, please close your issue and add a comment to the existing issue instead.

If you suspect your issue is a bug, please edit your issue description to include the BUG REPORT INFORMATION shown below. If you fail to provide this information within 7 days, we cannot debug your issue and will close it. We will, however, reopen it if you later provide the information.

For more information about reporting issues, see CONTRIBUTING.md.

You don't have to include this information if this is a feature request

(This is an automated, informational response)

BUG REPORT INFORMATION

Use the commands below to provide key information from your environment:

docker version:
docker info:

Provide additional environment details (AWS, VirtualBox, physical, etc.):

List the steps to reproduce the issue:
1.
2.
3.

Describe the results you received:

Describe the results you expected:

Provide additional info you think is important:

----------END REPORT ---------

#ENEEDMOREINFO

Note that the best possible outcome here would be for mknod and chmod during image builds to be executed from outside of any chroot, in order to maintain compatibility with these useful kernel security features (CONFIG_GRKERNSEC_CHROOT_MKNOD more so than CONFIG_GRKERNSEC_CHROOT_CHMOD). See deny chmod and deny mknod documentation.

Related bug against Gentoo docker package @ https://bugs.gentoo.org/show_bug.cgi?id=574664

For everyone who is using Alpine Linux and getting the following error :
failed to register layer: ApplyLayer exit status 1 stdout: stderr: chmod /bin/mount permission denied
Disabling chroot_deny_chmod and chroot_deny_mknod also fixes the problem.

This seems to be out of scope for this project. grsec is somewhat of a moving target because the stable kernel patches aren't available for the general public to use. We can't implement changes in Docker which would weaken security on all other Linux distributions in order to improve grsec compatibility.

The best thing to do is to figure out how all these issues can be addressed and maybe send a PR to add documentation on how to run in such a setup.

I'm going to close this issue now since there's nothing actionable for the Docker community to work on this. Please feel free to continue the discussion.

@unclejack The relevant grsec functionality has been stable for many years and is a core feature. It does not vary with Linux distribution or manual patching, it is in fact constant.

Let me try to rephrase the issue.

The purpose of grsec's 'deny mknod in chroot' feature is to stop people breaking out of chroot (eg. by creating and manipulating a device representing an important filesystem on the system).

The purpose of grsec's 'deny chmod in chroot' feature is to stop people breaking out of chroot. (eg. by changing privileges on an exposed device to permit an attack similar to the above)

The problem is that Docker currently executes the mknod and chmod commands when setting up a layer from within a chroot. By changing Docker's code to execute these commands from outside of a chroot, the problem would be solved.

In short, Docker makes it impossible to use useful security features of grsec, and as a result, forces even careful users to accept a higher level of minimum exposure. This is a security anti-feature.

Ideally, Docker would fix the problem by making the changes suggested above.

Alternatively, Docker should at least check that, if the kernel sysctls above are present, that their value is zero, and if not then give a meaningful error message.

Closing this issue after months languishing here with no useful feedback demonstrates an unimpressive level of commitment to user security on Docker's behalf.

Not surprised to see no response here after one week... the docker team's attitude to security seems to speak for itself.

The best thing to do is to figure out how all these issues can be addressed and maybe send a PR to add documentation on how to run in such a setup.

@globalcitizen if you have time to create a PoC PR, feel free to do so. Please note that Docker is open source, and "the docker team" includes you. @unclejack raised his concerns that we may not be able to support this on all distros, without weakening security for some. If you have a PoC PR to discuss this, and shows that that change does not negatively affect other distros, feel free to do so.

@thaJeztah I have made quite a few security-related contributions to docker, but I'm not paid, so there is a limit to my contribution. For example, I am not going to learn an entirely new language go just to make a PoC for an issue that I consider, given your supposed commercial trajectory toward production deployments, should be resolved by your paid staff rapidly on security grounds.

From comments it seems @unclejack does not comprehend the nature of grsec, which is a set of kernel patches that can be applied to any distro and usually result in exactly the same sysctls being added in a distro-generic manner.

There is no threat of weakening security, this is an entirely baseless accusation, which I assume is based upon the straw man that running mknod and chmod outside of a chroot is a threat.

In fact, mknod and chmod are perfectly safe to run outside of a chroot constructing an image layer if their arguments are validated or limited (eg. path-prefixed) in a standard unix-like fashion (as is no doubt done in many other parts of the docker codebase). This will probably enhance potential portability to different/older/newer Unices since jail-style systems are far from unified and by removing that implicit behavioral dependency within the image layer construction process and replacing it with more portable code you could even open the door to systems without any chroot-like features (eg. perhaps Windows kernels).

@globalcitizen nice to hear that you contribute back. I think it is reasonable that you "scratch your own itch" when you don't pay for the product. In fact this issue affects me too but I have not (yet) bothered enough to actually fix it.

After a quick look it does seems fixable without too much effort. Apparently, the chroot in question is there to prevent extracting tarballs with symlinks pointing to places outside the given root so I don't think we can simply remove the chroot without first verify that the tarball extraction is safe.

I can have a look at it later, so I vote for re-opening it.

@globalcitizen I'm very sorry if my comment made it seem I don't understand what the grsec patches do and what they are. Leaving that aside, what I was saying is that we shouldn't make changes to Docker if that means having less secure systems in the absence of the grsec patched kernels.

The points I was making in the previous comment are still valid.
Grsec isn't completely available in the open with all stable patches and all changes. We know why it's not open and that's well understood (trademark violations, abuse and so on). The stories behind this can be read online. We won't discuss this.

There's no simple way for the Docker community to provide support for would be grsec+Docker users. The issue of support is a very serious one because we can't test the exact same setup you're using for grsec. Do we all need Gentoo to run the exact same kernel with the exact same configuration, down to the exact same gcc version with the exact same flags? Do we need to have n versions of the Linux kernel with m versions of the grsec patches to make sure it's all OK?
This goes beyond getting it to work once. It's more about supporting it on the long term and being able to help users when things break due to interactions between kernel code changes and grsec or because of Docker code changes. Everyone expects things to work once a setup is described as OK.

Please keep in mind I'm not against grsec or supporting it with Docker. It's a bad idea if we're giving up security elsewhere to enable it to run on some systems.

@cpuguy83 Thanks for re-opening

@unclejack I feel like you have repeated yourself with the hand-waving suggestion there is some kind of problem 'supporting' grsec. Could you please explicitly explain what you mean about "giving up security elsewhere"? This does not make sense to me.

The situation right now is that kernels with grsec work fine already, regardless of version/distro, except for some specific grsec features which have been identified.

Looking forward, I have outlined two approaches which would be trivial and help to resolve this problem which multiple users have already reported.

The first and preferred approach is changing docker to not chmod/mknod inside a chroot when creating an image layer. This would make docker fully compatible even with the most secure and stringent configurations of grsec, at least as far as current user community testing indicates (my own included).

The second, which is more of a failure mode and statement of no intent to resolve the problem, is simply checking for the sysctls in question and exiting with a reasonable error message advising the user that those grsec features are incompatible with docker. Frankly though, the first approach is so easy the second approach should probably not be implemented.

I feel that your testing comments are a straw man, as test coverage for an arbitrary-kernel system does not need to be perfect and we can generally assume the Linux kernel does not break things between releases.

seems like the untar in chroot was introduced with 1cb17f0 but it does not say anything about why or what problem it is supposed to solve.

So I can only guess why its done that way. One possible reason might be to prevent overwriting files from locations outside the target dir. For example the tar archive could have an entry with ../../../etc/shadow and if you extract that without necessary check you may end up overwrite things you don't want.

It looks like there is a check for preventing that https://github.com/docker/docker/blob/master/pkg/archive/archive.go#L714 to happen, but it does not seem to handle the case where you have a symlink pointing to a directory. For example if you had a symlink in archive ./tmp/.ssh -> /root/.ssh and then had a file later in the archive ./tmp/.ssh/authorized_keys then you would end up overwriting /root/.ssh/authorized_keys.

chroot will prevent that. grsecurity alone will not.

The above example may be fixed with something like (untested):

diff --git a/pkg/archive/archive.go b/pkg/archive/archive.go
index 5f8cd09..0ab28e3 100644
--- a/pkg/archive/archive.go
+++ b/pkg/archive/archive.go
@@ -707,7 +707,7 @@ loop:
                }

                path := filepath.Join(dest, hdr.Name)
-               rel, err := filepath.Rel(dest, path)
+               rel, err := filepath.Rel(filepath.EvalSymlinks(dest), filepath.EvalSymlinks(path))
                if err != nil {
                        return err
                }

But I don't know if there are more similar cases. So I don't think we can simply remove the chroot without fully understanding the consequences.

@unclejack do you have any url to issue or discussion that led to the introduction of the chrootarchive?

We might actually be able to avoid the chmod by setting the correct mode during open(2) or possibly by setting umask. That way we'd be able to keep the chroot but avoid the grsecurity error.

But I don't think we can work around the mknod without doing it outside chroot.

Another option might be to check content of /proc/sys/kernel/grsecurity/chroot_deny_mknod and if its 1 then extract without chroot (like it is done on Windows). Then would only grsecurity users be exposed to symlink attacks.

@ncopa I think if you look at all the the changes (and tests) for the 1.3.2 release combined, it should give some context: #9323 (and https://github.com/docker/docker/blob/master/CHANGELOG.md#132-2014-11-20)

@ncopa It was for security reasons. That shouldn't be bypassed in any way.

@unclejack i figured it was for security reasons.

However, since chroot never was intended as a security feature, it is generally recommended to avoid using it as any kind of security tool. So I was interested in what exact (security) reasons led to the introduction of chroot. (it looks like it was to prevent symlink attacks. Thanks @thaJeztah)

The problem at hand is that to be able to use docker with grsecurity patched kernel, you have to system wide disable 2 other security features which will actually make chroot usable for security. So the dilemma is that both things are there for security reasons, but you have to disable one of them.

My feeling so so far have been that docker's use of chroot for untar is ok, and that it does not matter if grsecurity users disables the grsecurity chroot protections because you shouldn't use it as a security feature in the first place - which is what docker does. So its self contradictory 😃

@ncopa You're probably familiar with the reason why the chroot is necessary now. We could also replace that chroot with a container.

I've opened #22506 which replaces chroot with pivot_root.

pivot_root is now used by default on Linux.
Were there some other issues that need tobe addressed?

This covers all the issues raised, so I think we can close this now. Feel free to open other issues around grsec, I plan to do some more testing.

Folks, I'll close this now and will re-open if needed. Please feel free to comment.

run command : sysctl -w kernel.grsecurity.chroot_deny_chmod=0

Thank you so much @globalcitizen - I had this exact issue and didn't feel comfortable to weaken my kernel. Temporarily changing the sysctl flags is also a solution I read in grsec's manuals, so I am doing that before building a docker image to then reenable the protection. That still feels a bit scary, but it works. :)

@orangecms Docker 1.12 only does a chroot as a fallback if there was some issue with pivot_root.