@jperville have you tried apt-get installing the ceph-common package?

Hi @maclof ,

I have installed the ceph-common package from ceph.org, in version 0.94 hammer because the documentation recommends version >= 0.87 and Ubuntu 14.04 only packages version 0.80 out of the box.

I successfully ran sudo ceph -s to make sure that tge ceph demo cluster is working.

I have created the rbd volume from the host with no problem so I think that the ceph-common package on host is compatible with the ceph inside the ceph/demo image.

CC @rootfs @kubernetes/rh-storage

modprobe rbd failure is the problem. What kubelet container image you are using? can you install modprobe in your image?

Hi @rootfs, as explained above I am using gcr.io/google_containers/hyperkube-amd64:v1.2.1 as kubelet image and there was no modprobe installed inside so I manually ran apt-get update && apt-get install kmod but still same error hence this issue.

I'm not sure if /sbin is in the path so I will try symlinking in a few hours when I wake up.

After installing the kmod package in the container, /sbin/modprobe is available, however there is no /lib/modules in the container so invoking modprobe rbd from the kubelet container failed; I worked around the problem by ln -s /rootfs/lib/modules /lib/modules.

The container creation is now stuck at another step.

Unable to mount volumes for pod "rbd2_default(199392ab-fc91-11e5-8533-28d2444cbe8c)": rbd: map failed executable file not found in $PATH 

This time it is the 'rbd' executable missing. After redeploying the kubelet container with --volume=/sbin/modprobe:/sbin/modprobe:ro --volume=/lib/modules:/lib/modules:ro --volume=/etc/ceph:/etc/ceph:rw added to the docker run command-line, and reinstalling ceph-common inside the kubelet container, the progress is now stuck at:

E0407 07:54:12.034478   22122 kubelet.go:1780] Unable to mount volumes for pod "rbd2_default(e3f1e923-fc95-11e5-9dcb-28d2444cbe8c)": Could not map image: Timeout after 10s; skipping pod
E0407 07:54:12.034485   22122 pod_workers.go:138] Error syncing pod e3f1e923-fc95-11e5-9dcb-28d2444cbe8c, skipping: Could not map image: Timeout after 10s
I0407 07:54:26.592172   22122 rbd.go:89] ceph secret info: key/AQCyJQVXJV4gERAA1q7y4Wi6MiuO8UahSQoIrg==
I0407 07:54:26.594563   22122 nsenter_mount.go:179] Failed findmnt command: exit status 1
I0407 07:54:27.597245   22122 rbd_util.go:229] rbd: map mon 172.17.42.1:6789
I0407 07:54:30.593309   22122 nsenter_mount.go:179] Failed findmnt command: exit status 1

Every 25 seconds or so the 'foo' rbd device is mapped again (eg. /dev/rbd0 is mapped, then /dev/rbd1, then /dev/rbd2 etc) and every time the kubelet container logs the time out.

I finally made it work by applying several hacks:

1. running the kubelet container with --volume=/sbin/modprobe:/sbin/modprobe:ro --volume=/lib/modules:/lib/modules:ro --volume=/etc/ceph:/etc/ceph:rw
2. installing ceph-common manually in the kubelet container : curl https://raw.githubusercontent.com/ceph/ceph/master/keys/release.asc | apt-key add - && echo deb http://download.ceph.com/debian-hammer/ jessie main | tee /etc/apt/sources.list.d/ceph.list && apt-get update && apt-get install -y ceph-common
3. pre-creating some symlinks so that /dev/rbdX redirects to /rootfs/dev/rbdX : for i in 0 1 2 3 4 5 6 ; do ln -s /rootfs/dev/rbd${i} /dev/rbd${i} ; done

Workaround 3 is necessary because the getDevFromImageAndPool helper returns a device path such as /dev/rdb0, where the actual device path (as seen from the kubelet container) is /rootfs/dev/rdb0.

I traced the algorithm of the getDevFromImageAndPool helper using the shell in the kubelet container:

# cd /sys/bus/rbd/devices/
/sys/bus/rbd/devices# ls
0
/sys/bus/rbd/devices# name=0
/sys/bus/rbd/devices# cd ${name}
/sys/bus/rbd/devices/0# cat pool name 
rbd
foo
/sys/bus/rbd/devices/0# devicePath="/dev/rdb${name}" # pool 'rbd' and image 'foo' match
/sys/bus/rbd/devices/0# echo ${devicePath}
/dev/rbd0

For the moment I will prepare a custom hyperkube image with my workarounds but this is hackish.

Nice hack to get it running. For step 3, what if you bind mount host /dev (i.e. -v /dev:/dev) or there is an issue with that?

If i bind-mount /dev on the host to /dev on the kubelet container, the kubelet container will mess up with the pts on my host, resulting in being unable to start new terminals (gnome-terminal will fail with the "getpt failed" message) and making impossible to properly shutdown my workstations.

EDIT: after checking the issue tracker, the breakage resulting from bind-mounting /dev from host into the kubelet container is documented in #18230.

What I finally did was to wrap the hyperkube image like this:

# apply hacks from https://github.com/kubernetes/kubernetes/issues/23924#issuecomment-206803980
# so that pods that use rbd persistent resources work in the single-node docker setup.
# Build with the following command: `docker build -t custom/hyperkube-amd64:v1.2.1 .`

FROM gcr.io/google_containers/hyperkube-amd64:v1.2.1

RUN curl https://raw.githubusercontent.com/ceph/ceph/master/keys/release.asc | apt-key add - && \
    echo deb http://download.ceph.com/debian-hammer/ jessie main | tee /etc/apt/sources.list.d/ceph.list && \
    apt-get update && \
    DEBIAN_FRONTEND=noninteractive apt-get install -q -y ceph-common && \
    apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

And then run the kubelet container like this:

    docker run \
      --volume=/:/rootfs:ro \
      --volume=/sys:/sys:rw \                                               # necessary to do mount from container
      --volume=/var/lib/docker/:/var/lib/docker:rw \
      --volume=/var/lib/kubelet/:/var/lib/kubelet:rw \
      --volume=/var/run:/var/run:rw \
      --volume=/sbin/modprobe:/sbin/modprobe:ro \          # to skip having to install in container
      --volume=/lib/modules:/lib/modules:ro \                     # to make `modprobe rbd` work
      --volume=/etc/ceph:/etc/ceph:ro \                             # to copy ceph config from host
      --volume=/dev/rbd0:/rootfs/dev/rbd0:ro \                  # workaround for point 3 above
      --net=host \
      --pid=host \
      --privileged=true \
      --name=kubelet \
      -d \
      custom/hyperkube-amd64:v${K8S_VERSION} \        # image with ceph-common vendored-in
      /hyperkube kubelet \
      --containerized \
      --hostname-override="127.0.0.1" \
      --address="0.0.0.0" \
      --api-servers=http://localhost:8080 \
      --config=/etc/kubernetes/manifests \
      --cluster-dns=10.0.0.10 \
      --cluster-domain=cluster.local \
      --allow-privileged=true --v=2

Then I can use rbd persistent volumes from my dockerized kubernetes setup.

I hope that ptmx bug is fixed.

Is binding /dev/rbd0 working if no rbd device is there or rbd device is created after container is up?

@jperville what docker version are you using?

@pmorie : Using docker version 1.9.1 (cannot use 1.10 because of layer format changes that are incompatible with a tool we are using).

@rootfs : ptmx bug still here (but there is a workaround); regarding /dev/rbd0 you are right, it works if the device already exists on the host but not if there is no device yet (docker creates an empty directory on the node, and that makes the mount fail later). Have to find another hack (maybe wrap the CMD to create some symlinks before booting containerized kubelet).

@jperville Ok, the issue w/ pseudoterminals is supposed to be resolved in 1.10

@jperville moby/moby#16639

I ended up adding a wrapper into my custom hyperkube image for now.

Would adding some code to the getDevFromImageAndPool helper to understand the --containerized option (passed to kubelet) help to calculate a working path (have no go knowledge sadly).

@jperville In containerized openshift, the node (kubelet) bind mount /sys. rbd map works in this environment.

Hi guys, I am also having this issue, which i can work around by adding args to kubelet to mount /sbin/modprobe and /lib/modules. Is there any plan to fix this so that we dont have to manually feed this workaround to kubelet?

@kokhang which kubelet distribution you are using?

@rootfs Im using the kubelet that comes with coreos 4.7.3

I am not quite familiar with that setup. Do do use docker or rkt? Can you post docker inspect your_kubelet_container?

Taking this conversation with rootfs offline. But im still curious if there are any plans of making these modules enabled by default for RBD block storage

@jperville I still got stuck in Could not map image: Timeout after 10s; skipping after using your final solution，could you tell me how to solve this problem?

Also dealing with "Could not map image: Timeout after 10s". Is there a solution?

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close