Troubleshooting Common Problems#

When troubleshooting, you may see unexpected behaviors or receive an error message. This section provides advice on how to identify and fix some of the most commonly encountered issues.

Most of the docker run flags used in this document are explained in detail in the Common Features, Docker Options section of the documentation.

Permission denied when mounting volumes#

If you are running a Docker container while mounting a local volume or host directory using the -v flag like so:

docker run -it --rm \
    -p 8888:8888 \
    -v <my-vol>:<container-dir> \
    quay.io/jupyter/minimal-notebook:latest

you might face permissions issues when trying to access the mounted volume:

# assuming we mounted the volume in /home/jovyan/stagingarea
# root is the owner of the mounted volume
ls -ld ~/stagingarea/
# drwxr-xr-x 2 root root 4096 Feb  1 12:55 stagingarea/

touch stagingarea/kale.txt
# touch: cannot touch 'stagingarea/kale.txt': Permission denied

In this case, the user of the container (jovyan) and the owner of the mounted volume (root) have different permission levels and ownership over the container’s directories and mounts. The following sections cover a few of these scenarios and how to fix them.

Some things to try:

  1. Change ownership of the volume mount

    You can change the ownership of the volume mount using the chown command. In the case of the docker-stacks images, you can set the CHOWN_EXTRA and CHOWN_EXTRA_OPTS environment variables.

    For example, to change the ownership of the volume mount to the jovyan user (non-privileged default user in the Docker images):

    # running in detached mode - can also be run in interactive mode
    docker run --detach \
        -v <my-vol>:<container-dir> \
        -p 8888:8888 \
        --user root \
        -e CHOWN_EXTRA="<container-dir>" \
        -e CHOWN_EXTRA_OPTS="-R" \
        quay.io/jupyter/minimal-notebook
    

    where:

    • CHOWN_EXTRA=<some-dir>,<some-other-dir>: will change the ownership and group of the specified container directory (non-recursive by default). You need to provide full paths starting with /.

    • CHOWN_EXTRA_OPTS="-R": will recursively change the ownership and group of the directory specified in CHOWN_EXTRA.

    • --user root: you must run the container with the root user to change ownership at runtime.

    Now accessing the mount should work as expected:

    # assuming we mounted the volume in /home/jovyan/stagingarea
    ls -ld ~/stagingarea
    # drwxr-xr-x 2 jovyan users 4096 Feb  1 12:55 stagingarea/
    
    touch stagingarea/kale.txt
    # jovyan is now the owner of /home/jovyan/stagingarea
    # ls -la ~/stagingarea/
    # -rw-r--r-- 1 jovyan users    0 Feb  1 14:41 kale.txt
    

    Additional notes

    • If you are mounting your volume inside the /home/ directory, you can use the -e CHOWN_HOME=yes and CHOWN_HOME_OPTS="-R" flags instead of the -e CHOWN_EXTRA and -e CHOWN_EXTRA_OPTS in the example above.

    • This solution should work in most cases where you have created a docker volume (i.e. using the docker volume create --name <my-volume>command) and mounted it using the -v flag in docker run.

  2. Matching the container’s UID/GID with the host’s

    Docker handles mounting host directories differently from mounting volumes, even though the syntax is essentially the same (i.e. -v).

    When you initialize a Docker container using the -vflag, the host directories are bind-mounted directly into the container. Therefore, the permissions and ownership are copied over and will be the same as the ones in your local host (including user ids) which may result in permissions errors when trying to access directories or create/modify files inside.

    Suppose your local user has a UID and GID of 1234 and 5678, respectively. To fix the UID discrepancies between your local directories and the container’s directories, you can run the container with an explicit NB_UID and NB_GID to match that of the local user:

    docker run -it --rm \
        --user root \
        -p 8888:8888 \
        -e NB_UID=1234 \
        -e NB_GID=5678 \
        -v "${PWD}"/test:/home/jovyan/work \
        quay.io/jupyter/minimal-notebook:latest
    
    # you should see an output similar to this
    # Update jovyan's UID:GID to 1234:5678
    # Running as jovyan: bash
    

    where:

    • NB_IUD and NB_GID should match the local user’s UID and GID.

    • You must use --user root to ensure that the UID and GID are updated at runtime.

Additional notes

  • The caveat with this approach is that since these changes are applied at runtime, you will need to re-run the same command with the appropriate flags and environment variables if you need to recreate the container (i.e. after removing/destroying it).

  • If you pass a numeric UID, it must be in the range of 0-2147483647

  • This approach only updates the UID and GID of the existing jovyan user instead of creating a new user. From the above example:

    id
    # uid=1234(jovyan) gid=5678(jovyan) groups=5678(jovyan),100(users)
    

Permission issues after changing the UID/GID and USER in the container#

If you have also created a new user, you might be experiencing any of the following issues:

  • the root user is the owner of /home or a mounted volume

  • when starting the container, you get an error such as Failed to change ownership of the home directory.

  • getting permission denied when trying to conda install packages

Some things to try:

  1. Ensure the new user has ownership of /home and volume mounts

    For example, say you want to create a user callisto with a GID and UID of 1234. You will have to add the following flags to the docker run command:

     docker run -it --rm \
         -p 8888:8888 \
         --user root \
         -e NB_USER=callisto \
         -e NB_UID=1234 \
         -e NB_GID=1234 \
         -e CHOWN_HOME=yes \
         -e CHOWN_HOME_OPTS="-R" \
         -w "/home/callisto" \
         -v "${PWD}"/test:/home/callisto/work \
         quay.io/jupyter/minimal-notebook
    
     # Updated the jovyan user:
     # - username: jovyan       -> callisto
     # - home dir: /home/jovyan -> /home/callisto
     # Update callisto UID:GID to 1234:1234
     # Attempting to copy /home/jovyan to /home/callisto...
     # Success!
     # Ensuring /home/callisto is owned by 1234:1234
     # Running as callisto: bash
    

    where:

    • -e NB_USER=callisto: will create a new user callisto and automatically add it to the users group (does not delete jovyan)

    • -e NB_UID=1234 and -e NB_GID=1234: will set the UID and GID of the new user (callisto) to 1234

    • -e CHOWN_HOME_OPTS="-R" and -e CHOWN_HOME=yes: ensure that the new user is the owner of the /home directory and subdirectories (setting CHOWN_HOME_OPTS="-R will ensure this change is applied recursively)

    • -w "/home/callisto" sets the working directory to be the new user’s home

    Additional notes

    In the example above, the -v flag is used to mount the local volume onto the new user’s /home directory.

    However, if you are mounting a volume elsewhere, you also need to use the -e CHOWN_EXTRA=<some-dir> flag to avoid any permission issues (see the section Permission denied when mounting volumes on this page).

  2. Dynamically assign the user ID and GID

    The above case ensures that the /home directory is owned by a newly created user with a specific UID and GID, but if you want to assign the UID and GID of the new user dynamically, you can make the following adjustments:

    docker run -it --rm \
        -p 8888:8888 \
        --user root \
        -e NB_USER=callisto \
        -e NB_UID="$(id -u)" \
        -e NB_GID="$(id -g)"  \
        -e CHOWN_HOME=yes \
        -e CHOWN_HOME_OPTS="-R" \
        -w "/home/callisto" \
        -v "${PWD}"/test:/home/callisto/work \
        quay.io/jupyter/minimal-notebook
    

    where:

    • "$(id -u)" and "$(id -g)" will dynamically assign the UID and GID of the user executing the docker run command to the new user (callisto)

Problems installing conda packages from specific channels#

By default, the docker-stacks images have the conda channels priority set to strict. This may cause problems when trying to install packages from a channel with lower priority.

conda config --show | grep priority
# channel_priority: strict

# to see its meaning
conda config --describe channel_priority

# checking the current channels
conda config --show default_channels
# default_channels:
# - https://repo.anaconda.com/pkgs/main
# - https://repo.anaconda.com/pkgs/r

Installing packages from alternative channels:

You can install packages from other conda channels (e.g. bioconda) by disabling the channel_priority setting:

# install by disabling channel priority at еру command level
conda install --no-channel-priority -c bioconda bioconductor-geoquery

Additional details are provided in the Using Alternative Channels section of the Common Features page.

Tokens are being rejected#

If you are a regular user of VSCode and the Jupyter extension, you might experience either of these issues when using any of the docker-stacks images:

  • when clicking on the URL displayed on your command line logs, you face a “This site cannot be reached” page on your web browser

  • using the produced token and/or URL results in an “Invalid credentials” error on the Jupyter “Token authentication is enabled” page

    # example log output from the docker run command
    
    # [...]
    # Or copy and paste one of these URLs:
    #   http://3d4cf3809e3f:8888/?token=996426e890f8dc22fa6835a44442b6026cba02ee61fee6a2
    #   or http://127.0.0.1:8888/?token=996426e890f8dc22fa6835a44442b6026cba02ee61fee6a2
    

Some things to try:

  1. Find lingering Jupyter processes in the background

    The first thing you want to try is to check that no other Jupyter processes are running in the background:

    ps aux | grep jupyter
    

    If there are existing processes running, you can kill them with:

    # example output from the above command
    # my-user 3412 ... --daemon-module=vscode_datascience_helpers.jupyter_daemon
    
    # using the pid from the above log
    kill 3412
    
  2. Turn off Jupyter auto-start in VSCode

    Alternatively - you might want to ensure that the Jupyter: Disable Jupyter Auto Start setting is turned on to avoid this issue in the future.

    You can achieve this from the Settings > Jupyter menu in VScode:

    VSCode Settings UI - Jupyter: Disable Jupyter Auto Start checkbox checked

  3. Route container to unused local port

    Instead of mapping Docker port 8888 to local port 8888, map to another unused local port. You can see an example of mapping to local port 8001:

    docker run -it --rm -p 8001:8888 quay.io/jupyter/datascience-notebook
    

    When the terminal provides the link to access Jupyter: http://127.0.0.1:8888/lab?token=80d45d241a1ba4c2…, change the default port value of 8888 in the URL to the port value mapped with the docker run command.

    In this example, we use 8001, so the edited link would be: http://127.0.0.1:8001/lab?token=80d45d241a1ba4c2….

    Note: Port mapping for Jupyter has other applications outside of Docker. For example, it can be used to allow multiple Jupyter instances when using SSH to control cloud devices.