Software and Package Management

The standard environment on the MIT Supercloud System is sufficient for most. If it is not, first check to see if the tool you need is included in a module. Modules contain environment variables that set you up to use other software, packages, or compilers not included in the standard stack. If there is no module with what you need, you can often install your package or software in your home directory. If you have explored both these options or are having trouble, contact us.

Modules

Modules are a handy way to set up environment variables for particular work, especially in a shared environment. They provide an easy way to load a particular version of a language or compiler.

To see what modules are available, type the command:

module avail

See a list currently the modules available here.

To load a module, use the command:

module load moduleName

Where moduleName can be any of the modules listed by the module avail command.

If you want to list the modules you currently have loaded, you can use the module list command:

module list

If you want to change to a different version of the module you have loaded, you can switch the module you have loaded. This is important to do when loading a different version of a module you already have loaded, as environment variables from one version could interfere with those of another. To switch modules:

module switch oldModuleName newModuleName

Where oldModuleName is the name of the module you currently have loaded, and newModuleName is the new module that you would like to load.

If you would like to unload the module, or remove the changes the module has made to your environment, use the following command:

module unload moduleName

Finally, in order to use the module command inside a script, you will need to initialize it first.

The following shows a Bourne shell script example:

#!/bin/sh
# Initialize the module command first source
source /etc/profile
# Then use the module command to load the module needed for your work
module load anaconda3-2019a

Installing Software/Packages in your Home Directory

Many packages and software can be installed in user space, meaning they are installed just for the user installing the package or software. Often the flag to do this is "--user". The way to do this for Julia, Python, and R packages is described below, for other tools refer to its documentation to find out how to do this.

Julia Packages

Since Julia is a new evolving language, there are different ways to install packages depending on the version. For Julia version 0.6 you load the beta-julia-0.6 module, Julia 1.0 is the julia-1.0 module, and Julia 1.1 is the julia-latest module. We recommend you use the latest Julia version on our system, if possible.

Julia-1.1

Adding new packages in Julia 1.1 doesn't require doing anything special. On the login node, load the julia-latest module and start Julia. You can enter package mode by pressing the "]" key and entering add packagename, where packagename is the name of your package. Or you can load Pkg and run Pkg.add("packagename").

The easiest way to check if a package already exists is to try to load it by running using packagename. The Pkg.status() command will only show packages that you have added to your home environment. If you would like a list of the packages we have installed, the following lines should do the trick:

using Pkg; Pkg.activate(DEPOT_PATH[2]*"/environments/v1.1"); installed_pkgs = Pkg.installed(); Pkg.activate(DEPOT_PATH[1]*"/environments/v1.1"); installed_pkgs

Julia-1.0

Julia's package manager changed drastically with the new Julia 1.0 release. Future releases should support having both a system package directory and a user package directory, but for now if there is a package you need that we don't already have installed, you will have to create your own package directory.

To see what packages you have installed in the default environment, start Julia, load Pkg, and run Pkg.status() (in lieu of loading Pkg, you can press the "]" key to enter package mode and type "st").

If you decide you need to create your own package directory, you set the JULIA_DEPOT_PATH environment variable. At the command line:

export JULIA_DEPOT_PATH=/home/gridsan/USERNAME/.julia
echo $JULIA_DEPOT_PATH

When you start Julia you will be able to add packages to this new path (you can use Pkg.add("packageName"), or type "]" and enter the command "add packageName").

Note: If you are using Jupyter there is an additional step you need to complete: You'll need to add the export JULIA_DEPOT_PATH=/home/gridsan/USERNAME/.julia line to the .jupyter/llsc_notebook_bashrc file. See the section on the bottom of the Jupyter page for more information. You will also need to add the IJulia package to your custom package directory.

Julia-0.6

The Julia modules have package locations in the llgrid_beta directory, which cannot be written to by the general user. Before you add a package, check to see whether it is included in the Julia module.

First load the Julia module. The package directory can be changed to the user’s directory by changing the environment variable "JULIA_PKGDIR”:

For example:

export JULIA_PKGDIR=/home/gridsan/USERNAME/.julia/packages

Where USERNAME is your username.

Then start up Julia. Remember, you can use a module to select the version of Julia you want to use. You can double check that your package directory is set properly by entering the command “Pkg.dir()”. There is one more step before adding packages, and that is initializing. Run the command:

Pkg.init()

It may take a bit of time because it is downloading a number of default packages from Github. After you do this, you can add your packages. These will also take a bit of time when you add them.

If you are using Jupyter, you'll have to first add the IJulia package to your package directory. In Julia, run:

Pkg.add("IJulia")

You will also need to add the above export JULIA_PKGDIR ... line to the .jupyter/llsc_notebook_bashrc file. See the page on Jupyter notebooks for more information.

Note: If you are using Jupyter there is an additional step you need to complete: You'll need to add the export JULIA_DEPOT_PATH=/home/gridsan/USERNAME/.julia line to the .jupyter/llsc_notebook_bashrc file. See the section on the bottom of the Jupyter page for more information. You will also need to add the IJulia package to your custom package directory.

Python Packages

Many python packages are included in the Anaconda distribution. The quickest way to check this is to load the anaconda module, start python, and try to import the package you are interested in.

If the package you are looking to install is not included in Anaconda, then you can install it in user space- this allows you to install the package for you to use without affecting other users.

First, load the Anaconda module that you want to use if you haven’t already:

module load anaconda3-5.0.1

Here we are loading version 5.0.1 with python 3.6, feel free to load any version of Anaconda you like. Then, install the package as you normally would, but with the --user flag:

pip install --user packageName

Where packageName is the name of the package that you are installing.

Note: If you are using a conda environment and would like to install the package with pip in that environment rather than in the standard home directory location, you should not include the --user flag. Further, if you are using a conda environment and want Python to use packages in your environment first, you can run the following two commands:

unset PYTHONPATH
export PYTHONNOUSERSITE=True

The first of these will make sure your conda environment packages are used before the packages installed with the Anaconda module, and the second will make sure your conda environment packages will be chosen before those that may be installed in your home directory.

R Libraries

There are many R packages that are installed as part of Anaconda. First load the Ananconda module and see if your R package is available (if you are using Jupyter the Anaconda module is already loaded). To see all installed packages, run the installed.packages() command in R.

If your package is not already part of the system, you can install it in your home directory. Because this can only be done on the login node, installing the notebook cannot be done inside a Jupyter notebook. When you try to install a library R will install it in a default location in your home directory. Alternatively you can set your own install location (see instructions at the bottom of this section).

If you want to install in the default location, start R and run the command install.packages("myPackage", repos="http://cran.r-project.org"), where myPackage is the name of the package you want to install (see below). The first time you run it, will get a warning that the library location is not writable, and asked two questions, answer y to both:

[ledwards@login-1 ~]$ R

R version 3.4.0 (2017-04-21) -- "You Stupid Darkness"
Copyright (C) 2017 The R Foundation for Statistical Computing
Platform: x86_64-redhat-linux-gnu (64-bit)

...

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

 

>install.packages("myPackage", repos="http://cran.r-project.org")
Installing package into ‘/usr/lib64/R/library’
(as ‘lib’ is unspecified)
Warning in install.packages("corpus", repos = "http://cran.r-project.org") :
  'lib = "/usr/lib64/R/library"' is not writable
Would you like to use a personal library instead?  (y/n) y
Would you like to create a personal library
~/R/x86_64-redhat-linux-gnu-library/3.2
to install packages into?  (y/n) y

If you are using Jupyter notebooks you shouldn't have to do anything else to make your installed libraries available in Jupyter.

Custom Directory Install

To install in a custom directory, first create that directory in the terminal using mkdir:

mkdir ~/R_libs

Then start up R (the command is just R). Install the package you want:

install.packages("myPackage", repos="http://cran.r-project.org", lib="~/R_libs")

You can add your new library to the list of libraries that R searches when loading packages, which allows you to leave out the last argument. Simply set the environment variable R_LIBS to the directory you made at the terminal command line:

export R_LIBS=~/R_libs/

If you are using Jupyter notebooks, then you may want to add this line to the .jupyter/llsc_notebook_bashrc file. See the page on Jupyter notebooks for more information.