Jupyter notebooks are intended to be an interactive IDE environment. While we have streamlined and optimized much of the notebook workflow, please note that the best practice is to:
- Use the notebook to develop and test your application
- Once you have validated your algorithm save/export the notebook as a script
- Run your application interactively on a compute node or as a batch job through the scheduler
In the long run this will be far more efficient than always using the notebook.
In order to enable a greater number of Jupyter notebook jobs to run simultaneously on the SuperCloud system, we have partitions dedicated to Jupyter notebook with short time limits based on the requested resources. The table below summarizes the behavior (in terms of the number of users on a particular node and the time limits on notebook jobs), based on the resources that you request for your Jupyter notebook.
| Jupyter Resource Selection | Partition | # Cores | # GPUs | # Notebooks Per Node | Time Limit |
|---|---|---|---|---|---|
Default* Up to 4 cores | Jupyter CPU | 1,2*,3,4 | NA | Up to 48 | 12 hours |
| Exclusive CPU Node | xeon-p8 | 48 | NA | 1 | 4 hours |
| 1 GPU | Jupyter GPU | 20 | 1 | Up to 2 | 4 hours |
| 2 GPUs | Jupyter GPU | 40 | 2 | 1 | 2 hours |
Here are a few examples.
If you request the default parameters for a notebook, your notebook will:
- Run on the Jupyter CPU partition
- Be allocated 2 xeon-p8 cores and 0 GPUs
- Run on a node with as many as 48 notebook jobs
- Be allowed to run for up to 12 hours
If you request up to 4 cores and 0 GPUs for your notebook, your notebook will:
- Run on the Jupyter CPU partition
- Be allocated up to 4 xeon-p8 cores and 0 GPU
- Run on a node with less than 48 notebook jobs
- Be allowed to run for up to 12 hours
If you request an exclusive CPU node and 0 GPU for your notebook, your notebook will:
- Run on the xeon-p8 partition
- Be allocated 48 xeon-p8 cores (the whole node) and 0 GPUs
- Run on an exclusive node (only your notebook is running on the node)
- Be allowed to run for up to 4 hours
If you request 1 GPU for your notebook, your notebook will:
- Run on the Jupyter GPU partition
- Be allocated 20 xeon-g6 cores and 1 volta GPU
- Run on a node with up to 2 notebook jobs
- Be allowed to run for up to 4 hours
If you request 2 GPUs for your notebook, your notebook will:
- Run on the Jupyter GPU partition
- Be allocated 40 xeon-g6 cores and 2 volta GPUs
- run on an exclusive node (only your notebook is running on the node)
- be allowed to run for up to 2 hours
Reminder
Jupyter notebooks are meant to be used for development and testing, and for running interactive analysis. It is for this reason that we’ve implemented short time limits for notebook jobs that utilize multi-CPUs or GPUs.