Running notebooks in the cluster¶

Description¶

Running a jupyter notebook in the cluster allows you to work with a notebook which will be running even if you disconnect from the cluster.

This is especially useful for time-consuming/memory-consuming processes or notebooks with a high number of variables/packages needed, so that you have more computational power than your local computer, you can leave them running in the background without the fear of accidentally disconnecting and losing all the progress and you can come back to a notebook without the need of loading all the variables/packages again.

To run a notebook in the cluster, a screen and an interactive will be used.

Create a notebook¶

You will need to follow the next steps:

Connect to the cluster:
```
ssh -p 22022 <username>@bbgcluster
```

Open a screen:

<username>@login01:~$ screen -S <screen_name>

Run an interactive job and remember the node you are assigned to (e.g. bbgn005)
```
[screen_name] <username>@login01:~$ interactive
```
Info

If your notebook needs more than 8G and 2 cores, you can specify it here -- see interactive section.

Activate conda base or the conda environment that you need in your notebook:

[screen_name] <username>@bbgn005:~$ conda activate <your environment>

Go to the folder that you wish to run the notebook:

(base)[screen_name] <username>@bbgn005:~$ cd /workspace/folder

Run the jupyter notebook:

NotebookLab

Copy the following command:

unset XDG_RUNTIME_DIR && jupyter notebook --ip=0.0.0.0

Copy the following command:

unset XDG_RUNTIME_DIR && jupyter lab --ip=0.0.0.0

Keep the URL with the token and the port (e.g.8888) in which the interactive is running:

[I 10:37:20.371 NotebookApp] The Jupyter Notebook is running at: http://127.0.0.1:8888/?token=730ea7a95c02207c9fb7cbd434c2de81e03168845d42c23c

Now, your notebook is running and you can dettach from the screen by pressing Ctrl + A + D. You can now close the terminal and the notebook will continue running in the cluster.

Open a notebook¶

In order to open an already existing notebook, you'll need to know the port (e.g 8888) and the node of the cluster (e.g bbgn005) where you created it in the previous step.

ssh -L <port>:<node>:<port> -p 22022 <username>@bbgcluster
# For example: ssh -L 8888:bbgn005:8888 -p 22022 clopeze@bbgcluster

Note

If you don't want to remember these commands, you can create an alias. You can do this by adding the following lines to your ~/.bashrc file or ~/.bash_aliases file:

notebook(){
    # Ask for cluster user
    echo "Enter cluster user: "
    read user
    # Ask for node
    echo "Enter node name: "
    read node
    # Ask for port
    echo "Enter port: "
    read port
    # Connect to ssh cluster
    ssh -L $port:$node:$port -p 22022 ${user}@bbgcluster
}

Open the URL you obtain when creating the notebook in the cluster (step 7).

# For example: http://127.0.0.1:8888/?token=730ea7a95c02207c9fb7cbd434c2de81e03168845d42c23c

Close a notebook¶

When you don't need the notebook to continue running in the cluster, reconnect to the screen:

screen -r <screen_name>

And kill jupyter (Ctrl + C) and exit the screen (write exit in the terminal and press enter)

Errors and solutions¶

I forgot the URL of the notebook¶

Enter to the cluster and check your screens:
```
screen -ls
```
Enter the screen where you have your notebook:
```
screen -r <screen name>
```

Scroll up until you find the URL, which should look like:

http://127.0.0.1:8888/?token=730ea7a95c02207c9fb7cbd434c2de81e03168845d42c23c

My notebook doesn't open¶

One possibility is that the running notebook in the cluster has crashed. You can check this by going to the cluster, entering the screen where you have the notebook and check if it is still running.

If not, you should create a notebook following the steps at the beginning of this page (Create a new notebook).

Reference¶

Jordi Deu-Pons
Miguel Grau
Carlos López-Elorduy
Paula Gomis
Federica Brando