Using Jupyter notebooks

C3SE provides the possibility to run jupyter notebooks on its systems. This page will guide you on how to use this feature. If you find any relevant information is missing on this page, please contact us at support@c3se.chalmers.se and let us know.

1- Preparation

To be able to start up a jupyter notebook session (hereinafter abbreviated as JN), you need to have the following python script in your working directory:

import socket
hostname = socket.gethostname()
c.NotebookApp.base_url = '/{0}/'.format(hostname)
c.NotebookApp.allow_origin = '*'
c.NotebookApp.ip = '*'
c.NotebookApp.port_retries = 10
c.NotebookApp.open_browser = False

Save the script as jupyter_notebook_config.py and proceed to the next step.

2- Environment setup

Setup the environment by loading the modules that are required in your notebook. For the JN itself, the IPython module is an absolute necessity:

module load IPython

3- Running the notebook

Simple notebooks that do not contain computationally intensive work may be tested on the login nodes by running jupyter notebook on the command prompt. This returns with the following guidline if step 1 was done correctly:

To access the notebook, open this file in a browser:
        file:///cephyr/users/yourCID/machine/.local/share/jupyter/runtime/nbserver-xxx-open.html
    Or copy and paste one of these URLs:
        http://xxx:8888/xxx/?token=xxx
     or http://127.0.0.1:8888/xxx/?token=xxx

Currently, access to the notebook server file mentioned in the first line above is a work-in-progress. The two links that follow it should work though. Copy and paste either of them in a web browser, and replace the initial part with

https://proxy.c3se.chalmers.se:8888/.

For computationally intensive workflows, you should use the back-end nodes by submitting an interactive job:

srun -A yourAccount -p targetPartition -t 00:30:00 --pty jupyter notebook

If running on the new system Alvis, you must also specify the number of GPUs by e.g. including --gpus-per-node=1 in the above submission command. In the command above, you should specify the -A and the -p values as you normally do in your regular job submission scripts. Note that the waiting time for launching the interactive job is dependent on the workload of the machine, which means that at peak times you may have to wait long to get an interactive session.

Jupyter notebooks and TensorFlow

Note that if you want to use an MPI-based program like e.g. TensorFlow in your notebook, the above method of launching the notebook directly through srun will fail at runtime. That is because the process management interface (PMI) must be linked to properly (almost like running an MPI application from within a container image https://github.com/c3se/containers/blob/master/README.md#running-singularity-with-mpi-across-multiple-nodes). In those cases, the correct way of starting up a JN would be to have mpirun launch it:

mpirun -n 1 jupyter notebook, where the -n value can be as many cores as your notebook needs to run the job. For anything above -n 1, the use of the login nodes is not permitted, and you should launch the notebook through an interactive session:

• first ask for an interactive shell in your submission command: srun -A yourAccount -p targetPartition -t 00:30:00 --pty bash

• then launch JN using mpirun -n XX jupyter notebook