pip doesn't use proxy for search command #1104

albgus · 2013-08-02T08:19:41Z

I'm attempting to use pip to install packages to a python virtualenv. I have however discovered that when trying to search for commands, the proxy is not used. running for example pip search pika fails while pip install pika works fine.

I have set the environment variables http_proxy, https_proxy and no_proxy accordingly and I have tried to set the --proxy option with no success. It does however ask for a password if I specify a username but no password, so at least it acknowledges the options (the proxy is anonymous login)

I run into this issue on RedHat 6.4, with the default Python 2.6.6, and Python 2.7.5 from the tarball distribution.

Here are the logs:

Traceback (most recent call last):
  File "/home/gusson/.Python/vcdhelper26/lib/python2.6/site-packages/pip/basecommand.py", line 134, in main
    status = self.run(options, args)
  File "/home/gusson/.Python/vcdhelper26/lib/python2.6/site-packages/pip/commands/search.py", line 38, in run
    pypi_hits = self.search(query, index_url)
  File "/home/gusson/.Python/vcdhelper26/lib/python2.6/site-packages/pip/commands/search.py", line 52, in search
    hits = pypi.search({'name': query, 'summary': query}, 'or')
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1489, in __request
    verbose=self.__verbose
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1235, in request
    self.send_content(h, request_body)
  File "/usr/lib64/python2.6/xmlrpclib.py", line 1349, in send_content
    connection.endheaders()
  File "/usr/lib64/python2.6/httplib.py", line 908, in endheaders
    self._send_output()
  File "/usr/lib64/python2.6/httplib.py", line 780, in _send_output
    self.send(msg)
  File "/usr/lib64/python2.6/httplib.py", line 739, in send
    self.connect()
  File "/usr/lib64/python2.6/httplib.py", line 1112, in connect
    sock = socket.create_connection((self.host, self.port), self.timeout)
  File "/usr/lib64/python2.6/socket.py", line 567, in create_connection
    raise error, msg
timeout: timed out

Storing complete log in /home/gusson/.pip/pip.log

If I run a strace on the process it stops on this message, indicating that it tries to contact the source directly without going via the proxy, thus gets blocked by the firewall.

connect(3, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("199.27.73.185")}, 16) = -1 EINPROGRESS (Operation now in progress)
poll([{fd=3, events=POLLOUT}], 1, 15000

monkeypants · 2013-10-10T04:51:38Z

pip search uses xmlrpclib, which isn't natively proxy aware. pip install uses urllib2, which is (uses the HTTP_PROXY environment variable etc).

There are various examples of people creating a urllib2 transport and tunneling xmlrpclib over it, see for example https://gist.github.com/nathforge/980961 .There is even something like this happening in vendor/distlib/util.py.

Simply patching commands/search.py to use code from nathforge's gist didn't work though, possibly because pip search it's using xmlrpclib from pip.backwardcompat rather than the native one.

kkdd · 2013-10-19T06:08:41Z

The replacement of xmlrpclib.ServerProxy() in search.py by the xmlrpclib_ServerProxy() defined below works in my environment of Python 2.7.5 and pip 1.4.1.

import os

def get_proxy():
    proxy = os.getenv('http_proxy')
    if proxy and proxy.startswith("http://"):
        proxy = proxy[len('http://'):]
    return proxy

def xmlrpclib_ServerProxy(index_url):
    if get_proxy():
        p = ProxiedTransport()
        return xmlrpclib.ServerProxy(index_url, transport=p)
    else:
        return xmlrpclib.ServerProxy(index_url)

import httplib

class ProxiedTransport(xmlrpclib.Transport):
    def make_connection(self, host):
        self.realhost = host
        return httplib.HTTPConnection(get_proxy())
    def send_request(self, connection, handler, request_body):
        connection.putrequest("POST", 'http://%s%s' % (self.realhost, handler))
    def send_host(self, connection, host):
        connection.putheader('Host', self.realhost)

kkdd · 2013-11-13T13:47:52Z

The above version of ProxiedTransport() used httplib. In contrast, the following uses urllib2. Also works in my environment of Python 2.7.5 and pip 1.4.1.

import urllib2

class ProxiedTransport(xmlrpclib.Transport):
    SCHEME = 'https'
    def __init__(self, *args, **argd):
        xmlrpclib.Transport.__init__(self, *args, **argd)
        handlers = []
        proxy = urllib2.ProxyHandler({'http':get_proxy()})
        handlers.append(proxy)
        self.opener = urllib2.build_opener(*handlers)
    def request(self, host, handler, request_body, verbose=0):
        self.verbose = 0
        scheme = self.SCHEME
        url = '%(scheme)s://%(host)s%(handler)s' % locals()
        req = urllib2.Request(url, data=request_body,
                              headers={'Content-Type':'text/xml'})
        fp = self.opener.open(req)
        return self.parse_response(fp)

drakeguan · 2013-11-18T02:55:47Z

To help make this issue proceed, there is a pull request, #1180, which provides a HttpProxyTransport for xmlrpclib.ServerProxy to work perfectly with proxy. I have tested it and it worked well here. That pull request might be one of options for this issues?

monkeypants · 2013-11-18T05:03:54Z

I think that fixes the bug (--proxy ignored) and provides a work arround (pass in --proxy).

It would be nice if pip fell back to use HTTP_PROXY environment variable too, but not essential.

ash211 · 2014-04-16T17:10:38Z

Observed in pip 1.3.1 on CentOS 6.4 with EPEL's packages.

Works:

pip install $package

Doesn't work:

pip search $package

spuk- · 2014-04-24T14:18:37Z

You don't need to set the proxy handler manually for urllib2 as it automatically detects *_proxy env vars. Also, the code above didn't work for me, didn't care to check throughly why, but it simply wasn't using the proxy although get_proxy() appeared to be finding it. Simplified patch here: https://gist.github.com/spuk-/11256147#file-pip-search-urllib2

jrosiek · 2014-05-13T09:03:27Z

spuk's solutions is great! - simple and working. Please incorporate it into next release, thanks!

Ivoz · 2014-06-12T12:30:17Z

Relevant PRs #932, #1180

Alternate fix would be through #395

Proposed fix to #1180 (and maybe #932 & #1104)

Ivoz · 2014-07-04T05:00:24Z

Should be fixed in 1.6 through #1902 as above

kkdd mentioned this issue Jan 4, 2014

search does not use proxy on pip 1.3.1 #1017

Closed

aldanor mentioned this issue May 19, 2014

conda skeleton uses xmlrpc which ignores proxies conda/conda-build#109

Closed

Ivoz mentioned this issue Jun 12, 2014

pip search does not use proxy settings #1862

Closed

tnotstar mentioned this issue Jun 27, 2014

Proposed fix to #1180 (and maybe #932 & #1104) #1902

Merged

dstufft added a commit that referenced this issue Jul 3, 2014

Merge pull request #1902 from tnotstar/develop

d359d1a

Proposed fix to #1180 (and maybe #932 & #1104)

Ivoz closed this as completed Jul 4, 2014

doegox mentioned this issue Dec 28, 2016

Frida setup.py doesn't honor https_proxy frida/frida-python#99

Closed

lock bot added the auto-locked Outdated issues that have been locked by automation label Jun 5, 2019

lock bot locked as resolved and limited conversation to collaborators Jun 5, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pip doesn't use proxy for search command #1104

pip doesn't use proxy for search command #1104

albgus commented Aug 2, 2013

monkeypants commented Oct 10, 2013

kkdd commented Oct 19, 2013

kkdd commented Nov 13, 2013

drakeguan commented Nov 18, 2013

monkeypants commented Nov 18, 2013

ash211 commented Apr 16, 2014

spuk- commented Apr 24, 2014

jrosiek commented May 13, 2014

Ivoz commented Jun 12, 2014

Ivoz commented Jul 4, 2014

pip doesn't use proxy for search command #1104

pip doesn't use proxy for search command #1104

Comments

albgus commented Aug 2, 2013

monkeypants commented Oct 10, 2013

kkdd commented Oct 19, 2013

kkdd commented Nov 13, 2013

drakeguan commented Nov 18, 2013

monkeypants commented Nov 18, 2013

ash211 commented Apr 16, 2014

spuk- commented Apr 24, 2014

jrosiek commented May 13, 2014

Ivoz commented Jun 12, 2014

Ivoz commented Jul 4, 2014