Skip to content

Unable to push/pull from private registry #1486

New issue
New issue
Closed
Closed
Unable to push/pull from private registry#1486
@mmoulton

Description

@mmoulton
mmoulton
opened on Aug 10, 2013

Pushing / pulling from a private registry does not seem to be working as I would expect. Here is the setup:

I have an image I built, tagged with <hostname>/meltmedia/base where <hostname> points to a instance of the registry running, backed by s3, proxied with nginx.

When I do a docker push <hostname>/meltmedia/base I get output such as:

vagrant@vagrant-ubuntu-raring-64:/vagrant/meltmedia-base$ docker push <hostname>/meltmedia/base
The push refers to a repository [<hostname>/meltmedia/base] (len: 1)
Processing checksums
Sending image list
Pushing repository <hostname>/meltmedia/base (1 tags)
Pushing 7552a9e2b58881d086ff09179118bf07911cc85714cafd3e4082f73f7f23368a

This is where it starts to get odd, this operation is very quick. Looking at S3, only a _index_images file is written under a path of meltmedia/base. I would expect the actual layers to have been written as well.

To ensure that I wasn't simply missing something, I removed the images from my local docker instance using docker rmi. A docker images show that the build is indeed been removed.

I then proceeded to do a docker pull <hostname>/meltmedia/base where I get the following output:

vagrant@vagrant-ubuntu-raring-64:/vagrant/meltmedia-base$ docker pull <hostname>/meltmedia/base
Pulling repository <hostname>/meltmedia/base

After this a docker images does not show the meltmedia/base image as I would expect.

Inspecting the logs of the registry I notice 404's being returned when requesting the ancestry:

"[10/Aug/2013:00:45:58] "GET /v1/images/meltmedia/base/ancestry HTTP/1.0" 404 233 "-" "docker/0.5.1 go/go1.1 kernel/3.8.0-27-generic"

Is this a bug? Configuration issue?

I am using docker 0.5.1, docker-registry 0.5.5. Note, I have tried 0.5.2 with no luck either.

Regards,
Mike

Activity

creack

creack commented on Aug 10, 2013

@creack
creack
on Aug 10, 2013
Contributor

/cc @samalba

samalba

samalba commented on Aug 10, 2013

@samalba
samalba
on Aug 10, 2013
Contributor

Hi Mike,

it's hard to say at this point. Could you give me some logs written when you trigger the push? Especially the logs from nginx and the registry app.

A known problem is that nginx < 1.3.9 does not support chunked encoding, which is required by Docker.

mmoulton

mmoulton commented on Aug 10, 2013

@mmoulton
mmoulton
on Aug 10, 2013
Author

Thanks for the quick response. I have confirmed I am using the nginx 1.4 series. Log are as follows...

docker.log from executing docker push <hostname>/meltmedia/base

debug] api.go:925 Calling POST /images/{name:.*}/push from 127.0.0.1:34308
2013/08/10 02:03:30 POST /v1.4/images/<hostname>/meltmedia/base/push
[debug] registry.go:90 Registry https://<hostname>/v1/ does not work (Get https://<hostname>/v1/_ping: x509: certificate signed by unknown authority), falling back to http
[debug] registry.go:437 PUT http://<hostname>/v1/repositories/meltmedia/base/
[debug] registry.go:438 Image list pushed to index:
[{"id":"7552a9e2b58881d086ff09179118bf07911cc85714cafd3e4082f73f7f23368a","checksum":"sha256:694d0a1804ad8c113bfd5988590d037816e91841e5b3984f452a6680e48b67f0","Tag":"latest"},{"id":"9ce37638e0602cc01e329775d6b68e7d224da8eea0080a1824067eb20c732dcf","checksum":"sha256:9cc038204453973dbec92d74db76ff6957c323e35f9b53b621eebc2c6edd27d4","Tag":"latest"},{"id":"52dbc0e3cd5ab819d590a5112c17cfbe8d79359ef0b494fd63387dc94a8901f2","checksum":"sha256:cd8361e07e9cd0185b6738cf192e874587cd639ebf0290f8fcb6ce102aff349e","Tag":"latest"},{"id":"019aefe50749e69cb7ee3deaba01661dd1c2382658fa353434367269a7d9e6e8","checksum":"sha256:ad21c27316d72504651c695123ff31a9cbfc6a7596de5e7f3cae7f2ef889cf85","Tag":"latest"},{"id":"ad1a40142f3ebab04fbf7f929e1103bca8f9c8190a633fe25df19ed7aa7d721a","checksum":"sha256:9c10ae5a28c1ee62b7ecb537a8d1f11d5bb9f3b4c7745d3ebbe4e7ef298a8488","Tag":"latest"},{"id":"142457458ea2f2149366c32226fd617cf70ca1177c48c50432b448829c479934","checksum":"sha256:48f3f21470958b0ea9f90f24077dc93eb0e3dc5591a4d982b5ec39785db5db5d","Tag":"latest"},{"id":"853fceb3e04ce9aeef8541685df5c816b1e75bd80b4e4c91501fc6268227b476","checksum":"sha256:d15565067594215545ec81bb90caf86322380aa4b1097c1147492594f68677d2","Tag":"latest"},{"id":"42ee3131c802420cc1e43ada1bbd7c6fe8d7665d010225099429f2ae44e59945","checksum":"sha256:30d7fb31ab9b5276293f812589a45e162f4873634200d90ba84306207aa2a4f3","Tag":"latest"},{"id":"5523c50a8dfdb2e8952b36407eb8679fd5acc49a92989e3c72e7642e84bca474","checksum":"sha256:702774220212d610da57e32ff21ad44e962931461f4d5c5ce6fcb5ee734fc65c","Tag":"latest"},{"id":"68d823c84edada1e6cb68c750f4002cd78dc6e3d3846da406b29c4b5ab6c7de5","checksum":"sha256:fa3e65b1650bd836c0d3f4efca7134d10ecc4de42e647fd088d7689d295a4b8f","Tag":"latest"},{"id":"94942ba595a6a8d09ac116b6605c3726eac8188721809520c8b5cc00ed6ecb88","checksum":"sha256:966bd9f4baeba9fbfa942e12febfe9af072f66259e45b497e3c29cdb588b23b0","Tag":"latest"}]

[debug] registry.go:490 Auth token: [Token signature=QDSXSQZSRI8M4NUN,repository="meltmedia/base",access=write]
[debug] registry.go:345 Setting checksum for 7552a9e2b58881d086ff09179118bf07911cc85714cafd3e4082f73f7f23368a: sha256:694d0a1804ad8c113bfd5988590d037816e91841e5b3984f452a6680e48b67f0

nginx access.log:

<IP> - - [10/Aug/2013:01:59:43 +0000] "GET /v1/_ping HTTP/1.1" 200 4 "-" "Go 1.1 package http" "<IP>"
<IP> - mmoulton [10/Aug/2013:01:59:43 +0000] "PUT http://<hostname>/v1/repositories/meltmedia/base/ HTTP/1.1" 200 2 "-" "docker/0.5.1 go/go1.1 kernel/3.8.0-27-generic" "<IP>"

registry log:

2013-08-10 01:59:43 [667] [DEBUG] GET /v1/_ping
2013-08-10 01:59:43,254 DEBUG: GET /v1/_ping
2013-08-10 01:59:43,255 INFO: "<IP> - - [10/Aug/2013:01:59:43] "GET /v1/_ping HTTP/1.0" 200 4 "-" "Go 1.1 package http"
2013-08-10 01:59:43 [674] [DEBUG] PUT /v1/repositories/meltmedia/base/
2013-08-10 01:59:43,410 DEBUG: PUT /v1/repositories/meltmedia/base/
2013-08-10 01:59:43,489 INFO: "<IP> - - [10/Aug/2013:01:59:43] "PUT /v1/repositories/meltmedia/base/ HTTP/1.0" 200 2 "-" "docker/0.5.1 go/go1.1 kernel/3.8.0-27-generic"

Let me know if there is anything else I can provide that might help debug this.

-- Mike

mmoulton

mmoulton commented on Aug 10, 2013

@mmoulton
mmoulton
on Aug 10, 2013
Author

I was able to get pushing to a private registry to work by removing nginx from the chain. Regardless of the version of nginx, I could not get past the initial PUT request. The nginx config was as described by docker-registry:

server {
  location / {
    proxy_pass        http://localhost:5000;
    proxy_set_header  X-Real-IP  $remote_addr;
  }
}

I have ended up with hipache in front of gunicorn as there seems to be a ssl(?) check occurring in the beginning of the push that causes a plain gunicorn to not fail fast and requires the worker to exhaust it's timeout before starting the push. This check fails fast with hipache.

samalba

samalba commented on Aug 13, 2013

@samalba
samalba
on Aug 13, 2013
Contributor

We're using hipache + gunicorn on prod. @sdouche got nginx working (cf. docker mailing list) with the following options:

        client_max_body_size 800M;
        location / {
                proxy_pass        http://localhost:5000;
                proxy_set_header Host $http_host;


        }

I am still looking for a good nginx config file to contribute to the docker-registry repos. If it works for you too, let me know.

mmoulton

mmoulton commented on Aug 17, 2013

@mmoulton
mmoulton
on Aug 17, 2013
Author

Since this is working with hipache + gunicorn I'm going to close this down. Thanks!

mmoulton
closed this as completedon Aug 17, 2013
denmat

denmat commented on Sep 17, 2013

@denmat
denmat
on Sep 17, 2013

FYI -- I followed this thread but needed the following:
ubuntu - nginx-extras installed.

  client_max_body_size 0;

  chunkin on;

  error_page 411 = @my_411_error;
  location @my_411_error {
    chunkin_resume;
  }
sdouche

sdouche commented on Sep 17, 2013

@sdouche
sdouche
on Sep 17, 2013

uh?! @denmat can you explain why you need these lines?

denmat

denmat commented on Sep 18, 2013

@denmat
denmat
on Sep 18, 2013

I'm not really sure why I needed them, but they worked :)

Here is my vanilla nginx reverse proxy set up (nothing else running on nginx), everything else is default out of the box.

server { 
  listen 80;
  server_name docker.local;

  access_log /var/log/nginx/docker.access.log;

  location /docker_images {
    alias /usr/share/nginx/html/images;
  }

  location / {
    proxy_pass        http://localhost:49154;
    proxy_set_header  X-Real-IP  $remote_addr;
    proxy_set_header  Host       $http_host;
  }
}

Results:

Sending image list
Pushing repository docker.local:80/oracle_63_base (1 tags)
Pushing 4e6bd7e9bab93bd0b1686274ce82f870db546fa3191dc0087e6b5986d4083bce

2013/09/05 17:28:08 Received HTTP code 411 while uploading layer: <html>
<head><title>411 Length Required</title></head>
<body bgcolor="white">
<center><h1>411 Length Required</h1></center>
<hr><center>nginx/1.2.6 (Ubuntu)</center>
</body>
</html>

==> /var/log/nginx/docker.access.log <==
192.168.34.112 - - [05/Sep/2013:17:27:18 +1000] "GET /v1/_ping HTTP/1.1" 200 4 "-" "Go 1.1 package http"
192.168.34.112 - - [05/Sep/2013:17:27:19 +1000] "PUT /v1/repositories/oracle_63_base/ HTTP/1.1" 200 2 "-" "docker/0.6.1 go/go1.1.2 git-commit/5105263 kernel/3.8.0-29-generic"
192.168.34.112 - - [05/Sep/2013:17:27:23 +1000] "\x16\x03\x01\x00o\x01\x00\x00k\x03\x01R(2V(r\xF7y\xA2\xCE\xD3\xAEl]" 400 181 "-" "-"
192.168.34.112 - - [05/Sep/2013:17:27:28 +1000] "GET /v1/images/4e6bd7e9bab93bd0b1686274ce82f870db546fa3191dc0087e6b5986d4083bce/json HTTP/1.1" 404 34 "-" "docker/0.6.1 go/go1.1.2 git-commit/5105263 kernel/3.8.0-29-generic"
192.168.34.112 - - [05/Sep/2013:17:27:28 +1000] "PUT /v1/images/4e6bd7e9bab93bd0b1686274ce82f870db546fa3191dc0087e6b5986d4083bce/json HTTP/1.1" 200 4 "-" "docker/0.6.1 go/go1.1.2 git-commit/5105263 kernel/3.8.0-29-generic"
192.168.34.112 - - [05/Sep/2013:17:28:08 +1000] "PUT /v1/images/4e6bd7e9bab93bd0b1686274ce82f870db546fa3191dc0087e6b5986d4083bce/layer HTTP/1.1" 411 189 "-" "docker/0.6.1 go/go1.1.2 git-commit/5105263 kernel/3.8.0-29-generic"

With the following set ...

client_max_body_size 0;

Pushing 4e6bd7e9bab93bd0b1686274ce82f870db546fa3191dc0087e6b5986d4083bce

2013/09/05 17:30:42 Received HTTP code 411 while uploading layer: <html>
<head><title>411 Length Required</title></head>
<body bgcolor="white">
<center><h1>411 Length Required</h1></center>
<hr><center>nginx/1.2.6 (Ubuntu)</center>
</body>
</html>

==> /var/log/nginx/docker.access.log <==
192.168.34.112 - - [05/Sep/2013:17:30:18 +1000] "GET /v1/_ping HTTP/1.1" 200 4 "-" "Go 1.1 package http"
192.168.34.112 - - [05/Sep/2013:17:30:18 +1000] "PUT /v1/repositories/oracle_63_base/ HTTP/1.1" 200 2 "-" "docker/0.6.1 go/go1.1.2 git-commit/5105263 kernel/3.8.0-29-generic"
192.168.34.112 - - [05/Sep/2013:17:30:19 +1000] "GET /v1/images/4e6bd7e9bab93bd0b1686274ce82f870db546fa3191dc0087e6b5986d4083bce/json HTTP/1.1" 400 55 "-" "docker/0.6.1 go/go1.1.2 git-commit/5105263 kernel/3.8.0-29-generic"
192.168.34.112 - - [05/Sep/2013:17:30:23 +1000] "\x16\x03\x01\x00o\x01\x00\x00k\x03\x01R(3" 400 181 "-" "-"
192.168.34.112 - - [05/Sep/2013:17:30:28 +1000] "PUT /v1/images/4e6bd7e9bab93bd0b1686274ce82f870db546fa3191dc0087e6b5986d4083bce/json HTTP/1.1" 200 4 "-" "docker/0.6.1 go/go1.1.2 git-commit/5105263 kernel/3.8.0-29-generic"
192.168.34.112 - - [05/Sep/2013:17:30:42 +1000] "PUT /v1/images/4e6bd7e9bab93bd0b1686274ce82f870db546fa3191dc0087e6b5986d4083bce/layer HTTP/1.1" 411 189 "-" "docker/0.6.1 go/go1.1.2 git-commit/5105263 kernel/3.8.0-29-generic"

And with the following set ...

  client_max_body_size 0;

  chunkin on;

  error_page 411 = @my_411_error;
  location @my_411_error {
    chunkin_resume;
  }

192.168.34.112 - - [05/Sep/2013:17:32:38 +1000] "GET /v1/_ping HTTP/1.1" 200 4 "-" "Go 1.1 package http"
192.168.34.112 - - [05/Sep/2013:17:32:39 +1000] "PUT /v1/repositories/oracle_63_base/ HTTP/1.1" 200 2 "-" "docker/0.6.1 go/go1.1.2 git-commit/5105263 kernel/3.8.0-29-generic"
192.168.34.112 - - [05/Sep/2013:17:32:43 +1000] "\x16\x03\x01\x00o\x01\x00\x00k\x03\x01R(3\x96o\xC8\xC2S\x9C\xBC\xD3f\xF7Vv\xD8%\xC1C\xF3\x86^ntxD\xA7\x10\xC6_&\xBA\x00\x00\x10\x00\x05\x00" 400 181 "-" "-"
192.168.34.112 - - [05/Sep/2013:17:32:48 +1000] "GET /v1/images/4e6bd7e9bab93bd0b1686274ce82f870db546fa3191dc0087e6b5986d4083bce/json HTTP/1.1" 400 55 "-" "docker/0.6.1 go/go1.1.2 git-commit/5105263 kernel/3.8.0-29-generic"
192.168.34.112 - - [05/Sep/2013:17:32:48 +1000] "PUT /v1/images/4e6bd7e9bab93bd0b1686274ce82f870db546fa3191dc0087e6b5986d4083bce/json HTTP/1.1" 200 4 "-" "docker/0.6.1 go/go1.1.2 git-commit/5105263 kernel/3.8.0-29-generic"
192.168.34.112 - - [05/Sep/2013:17:35:34 +1000] "PUT /v1/images/4e6bd7e9bab93bd0b1686274ce82f870db546fa3191dc0087e6b5986d4083bce/layer HTTP/1.1" 200 4 "-" "docker/0.6.1 go/go1.1.2 git-commit/5105263 kernel/3.8.0-29-generic"
192.168.34.112 - - [05/Sep/2013:17:35:38 +1000] "PUT /v1/images/4e6bd7e9bab93bd0b1686274ce82f870db546fa3191dc0087e6b5986d4083bce/checksum HTTP/1.1" 200 4 "-" "docker/0.6.1 go/go1.1.2 git-commit/5105263 kernel/3.8.0-29-generic"
192.168.34.112 - - [05/Sep/2013:17:35:39 +1000] "PUT /v1/repositories/oracle_63_base/tags/latest HTTP/1.1" 200 4 "-" "docker/0.6.1 go/go1.1.2 git-commit/5105263 kernel/3.8.0-29-generic"
192.168.34.112 - - [05/Sep/2013:17:35:48 +1000] "PUT /v1/repositories/oracle_63_base/images HTTP/1.1" 204 0 "-" "docker/0.6.1 go/go1.1.2 git-commit/5105263 kernel/3.8.0-29-generic"

This is running the latest samalba/docker-registry without modification (runs in the Dev environment mode).
sudo docker version
Client version: 0.6.1
Server version: 0.6.1
Git commit: 5105263
Go version: go1.1.2

Size of the image is around 800M.

anapsix
mentioned this on Dec 5, 2013
angstwad
mentioned this on Apr 4, 2014
hordemark
mentioned this on Dec 5, 2014
metakermit
added a commit that references this issue on May 1, 2015

use extras because of moby/moby#1486

f33b4af
metakermit
mentioned this on May 1, 2015
chenjie901
mentioned this on May 20, 2015
wemeya
mentioned this on Sep 24, 2015
GreatSUN
mentioned this on Aug 18, 2016
bacongobbler
mentioned this on Oct 28, 2016
k8s-github-robot
added a commit that references this issue on Dec 8, 2016

Merge pull request #35797 from bacongobbler/registry-proxy

aa8a03e
kkrisz1
mentioned this on May 7, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @sdouche@samalba@creack@mmoulton@denmat

        Issue actions
          Unable to push/pull from private registry · Issue #1486 · moby/moby