@jpetazzo can you take a look at this one?

This is a kind of bug in AUFS. When a directory has a given permission mask in a lower layer, the upper layers cannot have a broader mask. Well, they can, but the more restrictive permission mask will be enforced anyways.

The rationale is the following:

• suppose that you have directory /secret with permissions 0700, containing file /secret/key.pem
• in an upper layer, you give /secret permissions 0755
• now /secret/key.pem could become reachable

Multiple behaviors could be considered "acceptable" in this scenario:

• give access to the file anyway (but this option was vetoed because it was deemed insecure)
• prevent access
• place a kind of "tombstone" or "opaque whiteout" for the whole directory so that the directory below becomes "opaqued" or "whited out", and the new one takes precedence

My understanding is that the last solution should be used, but for some reason, AUFS doesn't behave correctly. It might be because the directory exists in a lower layer, then doesn't exist anymore (because of the rm), then exists again (because of the ADD).

I'm willing to take a guess: the logic that decides whether or not to do a whiteout is not exactly the same as the one looking up permissions; so the first one stops when /etc/puppet is marked as non-existent in the middle layer, while the latter goes bottom up.

Anyway!

As a workaround, you can rm /etc/puppet/* instead of rm /etc/puppet, and that will do the trick.

Labelling as aufs-related.

Since this is documented aufs behavior, we can 1) close this as wontfix, 2) close + document the behavior in the docker docs, or 3) other?

@jpetazzo what do you think?

Hmmm... What about a "KNOWN BUGS AND ISSUES" section in the documentation? /cc @metalivedev

Ok, some discussion here: https://botbot.me/freenode/docker-dev/msg/6889335/
I'll look at how to provide the known issues in context, which implies that the docs need some place to describe the file system in use.

This bit me.

@xaviershay Could you give me the steps to reproduce the problem? I haven't been able to see an error with @astraw 's steps.

vagrant@precise64:~/src/783$ docker version
Client version: 0.6.4
Go version (client): go1.1.2
Git commit (client): 2f74b1c
Server version: 0.6.4
Git commit (server): 2f74b1c
Go version (server): go1.1.2
Last stable version: 0.6.4

vagrant@precise64:~/src/783$ docker build -t dockbug .
Uploading context 10240 bytes
Step 1 : FROM ubuntu:12.10
Pulling repository ubuntu
...
Step 6 : RUN chmod 755 /etc/puppet
 ---> Running in cf062fd724fb
 ---> 17620a2e9ea7
Successfully built 17620a2e9ea7

vagrant@precise64:~/src/783$ echo "note the directory is owned by puppet with full read/write/execute privs"
note the directory is owned by puppet with full read/write/execute privs

vagrant@precise64:~/src/783$ docker run dockbug ls -al /etc/puppet
total 12
drwxr-xr-x  2 puppet puppet 4096 Oct 19 00:18 .
drwxr-xr-x 78 root   root   4096 Oct 19 00:19 ..
-rw-rw-r--  1 puppet puppet    3 Oct 19 00:18 hello.txt

vagrant@precise64:~/src/783$ echo "but we get a permission error here"
but we get a permission error here

vagrant@precise64:~/src/783$ docker run dockbug sudo -u puppet ls -al /etc/puppet
total 12
drwxr-xr-x  2 puppet puppet 4096 Oct 19 00:18 .
drwxr-xr-x 78 root   root   4096 Oct 19 00:19 ..
-rw-rw-r--  1 puppet puppet    3 Oct 19 00:18 hello.txt

# HMM, no permission error

Having trouble replicating again.

The shape of my setup was building an image on top of itself (which is potentially a terrible idea):

# Dockerfile.bootstrap
FROM ubuntu:12.10

# Dockerfile
FROM thisimage

RUN rm -Rf /app
RUN mkdir /app

docker build -t thisimage - < Dockerfile.boostrap
docker build -t thisimage - < Dockerfile
docker build -t thisimage - < Dockerfile # Do this a couple of times.

That looks to me like you'd run into the AUFS 42 layer limit pretty quickly.

Yeah that was the next bug I hit :)

Have since redone my process to always build off a single base image.

Since Docker is moving away from AUFS to use DM in the next major release, this bug won't appear anymore.
Therefore, I'm closing it.
Feel free to reopen if you think it should still be open for a specific reason, though!

We're still going to have AUFS in 0.7, so I'm reopening this. :)

Just got bit by this too. We've got sequential builds, and at several points we need to obliterate a directory that was ADDed in a previous build step. I was surprised that the following command silently fails:

RUN rm -Rf /var/shared/sites/coursecal

I'm not 100% sure why, but the workaround suggested by @jpetazzo above seems to work:

RUN rm -Rf /var/shared/sites/coursecal/* /var/shared/sites/coursecal/.*git

The only reason we have to do the "rm" in the first place is because the following line is really a tar x, and it leaves existing files around, as documented here

ADD . /var/shared/sites/coursecal

The above discussion from several months ago at version 0.7 suggests that DeviceMapper became the recommended backend. But now we're on 0.9.1 and AUFS still seems to be the default backend. For end-users like me, default=recommended. This means docker is inconsistent.

What should I be using? Also, where is the documentation on AUFS vs DeviceMapper vs whatever else? I don't see any.

Also @jpetazzo

Docker is moving away from AUFS to use DM in the next major release

What is the next major release? Did you mean 1.0?

James, aufs is the default if your system supports it. This is simply because it is the most battle-tested storage driver and has less moving parts in general.

Devmapper is the default for non-aufs systems, which is the majority of systems in the wild since aufs is not part of the upstream kernel.

In doubt, we recommend using the default - keeping in mind the default might be different from system to system.

On Wed, Mar 26, 2014 at 12:26 PM, James Harrison Fisher
notifications@github.com wrote:

Also @jpetazzo

Docker is moving away from AUFS to use DM in the next major release

What is the next major release? Did you mean 1.0?

Reply to this email directly or view it on GitHub:
#783 (comment)

As for Jerome's statement, it is no longer accurate. Our initial plan was to phase out aufs completely because devmapper appeared to be the best option in 100% of cases. That turned out not to be trye, there are tradeoffs depending on your situation and so we are continuing to maintain both.

On Wed, Mar 26, 2014 at 12:26 PM, James Harrison Fisher
notifications@github.com wrote:

Also @jpetazzo

Docker is moving away from AUFS to use DM in the next major release

What is the next major release? Did you mean 1.0?

Reply to this email directly or view it on GitHub:
#783 (comment)

@shykes okay, what are the tradeoffs? Are the issues with devmapper, say, performance tradeoffs, or outright bugs, like this one with aufs?

If there are tradeoffs and the user is expected to make that decision, then those tradeoffs should be documented.

Frankly, I believe that I should be allowed to shoot myself in the foot with the aufs thing. This isn't a funky implementation bug, it was a conscious decision. Does anyone have a patch for aufs that can toggle this behavior?

@jameshfisher you are right but let's discuss this somewhere else, we are veering off-topic for this issue.

Any progress on this? We are consistently seeing this behaviour with our mysql /var/lib/mysql directories under docker 0.11.1 and periodically in 0.9.1

@jberkus thanks for the info. I was able to reproduce it on Ubuntu 14.04 on kernel 3.19.something. Would be nice to know which version of aufs its solved in, possibly the Docker Mac team can bump the version to include the patch that's needed

I can confirm encountering this issue on the the docker os x beta-15 (current version). I have switched back to docker-machine in the meantime.

rm /etc/puppet/* didn't work for me, instead I found that deleting individual files on the same layer works perfectly well:

find /etc/pappet -type f | xargs -L1 rm -f

Here is link to my Gist

We are seeing a very similar issue with Overlay.

Our current guess is that it is caused by the bug fixed in kernel 4.4.6 by this commit: https://lkml.org/lkml/2016/1/31/82

We have not yet managed to test with 4.4.6.

rm /etc/puppet/* didn't work for me, instead I found that deleting individual files on the same layer works perfectly well:

find /etc/pappet -type f | xargs -L1 rm -f

Here is link to my Gist

Thanks, it works for me!