Skip to content

Broken jupyter notebook kernel integration with conda environments #245

Closed
@dhimmel

Description

@dhimmel

Greetings. I'm currently in a Jupyter kernel hell. I can't figure how to fix it. And I'm not sure whether it's a Jupyter or conda issue, so thanks for the patience!

Here's my setup. I use conda to manage my python environments. I work on many different projects that use Jupyter notebooks, each project having its own environment (specified in environment.yml files).

I recently reinstalled conda to update my root environment to Python 3.6 and conda 4.3.13. As part of this reinstallation, I tried to remove files that I thought would get freshly regenerated including jupyter configuration files (IIRC).

This reinstallation of conda triggered the issue:

  • when I run jupyter notebook with the conda environment activated, the specific conda environment is not available in the spawned notebook browser.
  • If I run python -m ipykernel.kernelspec (I think) then I the activated conda environment becomes the default Python 3 kernel for all my notebook servers.
  • Creating new conda environments with jupyter installed does not add them to the list of kernels in jupyter.

Before I reinstalled conda (and deleted jupyter config directories since I wanted to freshly reinstall everything related to python):

  • if I launched a notebook from an activated conda environment, the default Python 3 kernel would correspond to that notebook
  • if I created a new conda environment, the kernel would show up as a choice in all jupyter sessions regardless of which environment jupyter notebook was running from
  • I never had to mess around with configuration files.

So I think I've broken something and I want the old (or any environment aware) behavior back. Tagging @minrk @jakevdp @kalefranz @asmeurer in case you or someone you mention can help. Will really appreciate any guidance!

Activity

jkingphd

jkingphd commented on Mar 14, 2017

@jkingphd

For what it's worth, I am experiencing the similar issues. I am on Windows 10, and I completely removed Anaconda using the recommended steps (anaconda-clean, then uninstall from add/remove programs). I installed the Python 3.6 version of Anaconda, and now my Jupyter Notebook is completely borked. I can launch the server, but I can't access any of my conda environments (old or new). Only one kernel is listed: "Python 3." If I install Jupyter from within the environment and launch the notebook server while the environment is activated, sys.path still refers to the base installation path. Is something broken with respect to 3.6?

takluyver

takluyver commented on Mar 14, 2017

@takluyver
Member

I think Continuum stopped shipping their environments-as-kernels plugin by default - you can probably still install it, or you can set up kernels for environments manually. We're thinking about better ways to expose environments to Jupyter.

jkingphd

jkingphd commented on Mar 14, 2017

@jkingphd

Thanks for the quick reply, this restored my workflow entirely. I think you are right about Continuum. I guess it serves me right for relying on pre-built distros, heh.

dhimmel

dhimmel commented on Mar 16, 2017

@dhimmel
Author

I think Continuum stopped shipping their environments-as-kernels plugin by default - you can probably still install it

@takluyver, thanks for that info! It was enough for me to track down nb_conda_kernels (anaconda cloud, repo). Once I'm back at the affected machine, I'll let you know if this fixes my issues.

nb_conda also looks interesting (allows you to manage conda environments in a Jupyter browser tab).

In worst case, I should be able to use the single kernel installation commands you link to:

source activate myenv
python -m ipykernel install --user --name myenv --display-name "Python (myenv)"
source activate other-env
python -m ipykernel install --user --name other-env --display-name "Python (other-env)"

But I'm not a huge fan of this solution as it's a manual step. Furthermore, when we have several people collaborating on a repo, we want them to all have the same kernel names for conda environments.

dhimmel

dhimmel commented on Mar 16, 2017

@dhimmel
Author

conda install nb_conda_kernels failed with:

UnsatisfiableError: The following specifications were found to be in conflict:
  - nb_conda_kernels -> _nb_ext_conf -> nb_anacondacloud -> python 2.7* -> openssl 1.0.1*
  - python 3.6*

So I installed nb_conda_kernels from conda-forge using:

conda install --channel=conda-forge nb_conda_kernels

Now I can choose from my conda environments in the Jupyter kernel menu. Phew!

Note that the nb_conda_kernels README states:

To use a conda environment as a kernel, don't forget to install ipykernel in this environment or it won't show up in the kernel list.

dhimmel

dhimmel commented on Mar 16, 2017

@dhimmel
Author

@takluyver or other repo members: close this issue at will. I may still report some additional findings here...

takluyver

takluyver commented on Mar 16, 2017

@takluyver
Member

Thanks

xiaotangfeidao

xiaotangfeidao commented on Apr 29, 2017

@xiaotangfeidao

I had similar issue, this was really helpful. Thanks

tsu-nera

tsu-nera commented on May 1, 2017

@tsu-nera

Thank you. I had same problem.

bede

bede commented on Jul 12, 2017

@bede

Such a useful thread. IMO it is more than a little counterintuitive that notebooks do not use the active conda env by default, nor that they are even visible!

JyiHUO

JyiHUO commented on Aug 1, 2017

@JyiHUO

Thank you, this is very helpful for me. I am sucking in this problem about two day

NumesSanguis

NumesSanguis commented on Aug 29, 2017

@NumesSanguis

I did the following:

python -m ipykernel install --user --name StoAU2 --display-name "Python (StoAU2)"
/home/user/anaconda3/envs/StoAU2/bin/python: No module named ipykernel
[Errno 13] Permission denied: '/usr/local/share/jupyter'`

So then I tried:

source deactivate
python -m ipykernel install --user --name StoAU2 --display-name "Python (StoAU2)"
Installed kernelspec StoAU2 in /home/user/.local/share/jupyter/kernels/stoau2
python -m ipykernel install --user --name StoAU --display-name "Python 3 (StoAU)"
Changed Kernel -> Python (StoAU2).

Then in a Jupyter notebook:

import sys
print(sys.executable)
from jupyter_core.paths import jupyter_data_dir
print(jupyter_data_dir())`

/home/user/anaconda3/bin/python
/home/user/.local/share/jupyter

Still not referring to the right kernel... This is quite an annoying change that terminal kernel and Notebook terminal are not the same.

This is what I tried (based on: https://github.com/Anaconda-Platform/nb_conda_kernels/blob/fe861722902df9c97f47bbf6e16c8c634d17144f/README.md):

conda install -c conda-forge nb_conda_kernels
conda create -n test
package plan Proceed ([y]/n)? y
conda install -n nb_conda_kernels --file requirements.txt -c r

But I got an error that I don't have the requirement.txt (sorry lost actual error). So I reinstalled everything and did the following (which finally worked):

conda create --name StoAU
source activate StoAU
conda install ipykernel
python -m ipykernel install --user --name StoAU --display-name "Python 3 (StoAU)"

I might be a beginner, but I don't understand what the purpose is of creating an environment and installing all packages there, if in the end the notebook doesn't use this kernel with what you've just installed.

5 remaining items

anshbansal

anshbansal commented on Oct 4, 2017

@anshbansal

dhimmel commented on Mar 16

That worked for me on Windows 10 machine

anshbansal

anshbansal commented on Oct 4, 2017

@anshbansal

I changed my .condarc from

channels:
  - defaults
ssl_verify: true

to

channels:
  - defaults
  - conda-forge
create_default_packages:
  - pip
  - ipykernel
  - nb_conda_kernels
ssl_verify: true

and now whenever I am creating a new virtualenv it gets automatically added to the list of environments. I don't even have to run the commands anymore.

And they got removed when I remove the env using conda remove env -n NAME_OF_ENV_HERE

thvasilo

thvasilo commented on Dec 1, 2017

@thvasilo

For future Googlers the simplest solution seems to be to install the nb_conda package: https://stackoverflow.com/a/38880722/209882

jbrant

jbrant commented on Jun 26, 2018

@jbrant

nb_conda doesn't seem to work with jupyter lab. The only option that worked for me was manually registering the applicable kernels as demonstrated in the fifth post from the top of this thread.

KODeKarnage

KODeKarnage commented on Jul 13, 2018

@KODeKarnage

Installing nb_conda_kernels in the activated environment works for me with jupyter lab on Windows10.

It comes with a tonne of dependencies though, so may not be worth it.

rakeshch

rakeshch commented on Sep 29, 2018

@rakeshch

Very helpful post, Thank you

ghost
mtshikomba

mtshikomba commented on May 30, 2020

@mtshikomba

Refer to this package: nb_conda_kernels

It worked for me.

DJD-Code-Studio

DJD-Code-Studio commented on Nov 16, 2020

@DJD-Code-Studio

I couldn't run jupyter kernelspec list
It failed with the error --

Traceback (most recent call last):
File "C:\DEB_pythons\anaconda3\Scripts\jupyter-kernelspec-script.py", line 5, in
fromTraceback (most recent call last):
File "C:\DEB_pythons\anaconda3\lib\runpy.py", line 185, in _run_module_as_main
mod_name, mod_spec, code = get_module_details(mod_name, Error)
File "C:\DEB_pythons\anaconda3\lib\runpy.py", line 144, in get_module_details
return get_module_details(pkg_main_name, error)
File "C:\DEB_pythons\anaconda3\lib\runpy.py", line 111, in get_module_details
import(pkg_name)
File "C:\DEB_pythons\anaconda3\lib\site-packages\ipykernel_init
.py", line 2, in
from .connect import *
File "C:\DEB_pythons\anaconda3\lib\site-packages\ipykernel\connect.py", line 18, in
import jupyter_client
File "C:\DEB_pythons\anaconda3\lib\site-packages\jupyter_client_init
.py", line 4, in
from .connect import *
File "C:\DEB_pythons\anaconda3\lib\site-packages\jupyter_client\connect.py", line 21, in
import zmq
File "C:\DEB_pythons\anaconda3\lib\site-packages\zmq_init.py", line 47, in
from zmq import backend
File "C:\DEB_pythons\anaconda3\lib\site-packages\zmq\backend_init.py", line 40, in
reraise(*exc_info)
File "C:\DEB_pythons\anaconda3\lib\site-packages\zmq\utils\sixcerpt.py", line 34, in reraise
raise value
File "C:\DEB_pythons\anaconda3\lib\site-packages\zmq\backend_init.py", line 27, in
ns = select_backend(first)
File "C:\DEB_pythons\anaconda3\lib\site-packages\zmq\backend\select.py", line 28, in select_backend
mod = import(name, fromlist=public_api)
File "C:\DEB_pythons\anaconda3\lib\site-packages\zmq\backend\cython_init
.py", line 6, in
from . import (constants, error, message, context,
ImportError: DLL load failed while importing error: The specified module could not be found.

Which led me this ---

(at a python prompt)
import KernelSpecApp
Traceback (most recent call last):
File "", line 1, in
ModuleNotFoundError: No module named 'KernelSpecApp'

Then similar result followed in case of --

(at a python prompt)
import ipykernel
Traceback (most recent call last):
File "", line 1, in
ModuleNotFoundError: No module named 'ipykernel'

I checked my Anaconda3 installation which was done via silent mode - the module ipykernel was definitely there.
But in spite of that python was failing to locate them which indicates in all probability a 'path not found' kind of issue.

That reminded me that I had opted for not setting the path during the installation by

using the parameter AddToPath=0 during the silent installation.

Hence the solution was to add the 3 vital Anaconda3 paths to the system level "path" variable.
First I made a new environment variable called -

ANACONDA_PATHS and added the 3 following paths there-
C:\DEB_pythons\anaconda3
C:\DEB_pythons\anaconda3\Scripts
C:\DEB_pythons\anaconda3\Library\bin\

Btw, the first is the Anaconda3 root location in my set up.
And voila...
I was able to run --

python -m ipykernel

with no errors.
This also solved my initial problem ...

jupyter kernelspec list
Available kernels:
python3 C:\DEB_pythons\anaconda3\share\jupyter\kernels\python3
anaconda3 C:\ProgramData\jupyter\kernels\anaconda3
py390_ds_env C:\ProgramData\jupyter\kernels\py390_ds_env
C:

SouravPal1995

SouravPal1995 commented on Feb 20, 2021

@SouravPal1995

I was facing an issue with this method. If i freshly create a conda environment and then create a kernel with it, I am being able to access those kernels from the base environment, however, if I update the conda environment, then jupyter from the base environment is not being able to connect with this kernel.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @takluyver@tsu-nera@bede@dhimmel@barnabytprowe

        Issue actions

          Broken jupyter notebook kernel integration with conda environments · Issue #245 · jupyter/jupyter