Getting started in Multispeech

The aim of this document is to provide some practical and technical information for new members of the Multispeech team. In addition to the information provided here, you can consult the dedicated Q&A platform to ask your diverse questions, or to help others by answering their questions, if possible.

Also, check different channels of Multispeech on Mattermost. In this platform, you can directly communicate with others, ask/answer questions, share your interesting stuffs, etc.

For more detailed information on the team and Inria/Loria life, we highly recommend you to explore Welcome to Multispeech! documentation on GitLab, especially this section.

Content

IT infrastructure

Computational cluster

IT infrastructure

The IT Portal is here. For more details, you can also visit this website.

Professional Email

You can access your Inria email account following this website.

Remember to always use your Inria email instead of a personal one.

Access from outside Loria

WiFi

When you are outside your office but still at Loria, you can follow these instructions to connect to the WiFi.

Git

You should use Inria GitLab to host/share your codes. You can create a new group associated with your project. This repo will host all the codes (Python and LaTeX) that you will develop (but not data or PDFs, which are not meant to be stored in a Git repo).

Shared disk space

The three disk spaces are structured similarly to ⁣/local/multispeech_ge but there is much more space available.

Create your own folder in one of the multispeech/calcul/users/ directories in order to store the intermediate/final results of your experiments. See the following section for more details.
Remember to always monitor the stored data and remove your useless data.

Computational cluster

We do our experiments on Grid’5000, which is a large-scale and flexible testbed for experiment-driven research, with powerful GPU and CPU machines. Here, we provide a step-by-step guide to get started with Grid’5000.

Please also check this useful doc if you need more detailed information.

  1. Create an account here (enter Multispeech for “Group Granting Access”).

  2. Set-up SSH key pairs. See here on how to do that.

  3. Paste the public key’s content (run cat ~/.ssh/id_rsa.pub) in your account space on Grid5000 (manage account -> SSH keys).

  4. Create a config file for easy connecting (see the example SSH config file available here) and put it in ~/.ssh; you’ll also need to locate your generated public and private keys there.

  5. Simply connect to the Nancy workstation by: ssh nancy.g5k.

Setting up Conda

Once you are done with the previous set-up, you can create a folder with your username at /srv/storage/talc3@storage4.nancy.grid5000.fr/multispeech/calcul/users. Then, follow the below instructions, while being connected to the cluster, to set up the conda environment for running your codes:

  1. Install anaconda by first downloading an appropriate version, e.g.:
    wget https://repo.anaconda.com/archive/Anaconda3-2020.11-Linux-x86_64.sh.
    Then, install it using bash Anaconda3-2020.11-Linux-x86_64.sh.

  2. Activate the conda environment: source ~/.bashrc.

  3. Create a virtual environment using conda: conda create --name venv. See this website for more details.

  4. Activate the virtual environment: conda activate venv.

  5. Install PyTorch. You can follow the official PyTorch’s website for the installation details.

To know the CUDA version of the GPU machine that you are going to use for model training, you can simply run nvidia-smi in the terminal connected to that cluster. You can also see hardware properties of different clusters available on Grid`5000.

Important- It might happen that PyTorch is not installed with CUDA enabled. To resolve this issue, first make sure that Torch is uninstalled. You can uninstall it with pip uninstall torch, and then reinstall it using Wheel, e.g.:
pip install torch==1.12.1+cu102 torchvision==0.13.1+cu102 torchaudio==0.12.1 --extra-index-url https://download.pytorch.org/whl/cu102
Then, if you get the following outputs, it means that PyTorch has been successfully compiled with CUDA enabled:

>>> import torch
>>> torch.__version__
'1.12.1+cu102'
>>> torch.zeros(1).cuda()
tensor([0.], device='cuda:0')
>>> torch.cuda.is_available()
True

Reserving nodes on Grid5k

See here for more details on how to book a node. You can choose different clusters and wall time depending on your needs. See the list of available clusters here.

You can also run your Python code from inside a bash script in passive mode:
oarsub -S ./train.sh -q production -p "cluster='graffiti'" -l nodes=1,walltime=30:00:00 -O OUT/oar_job.%jobid%.output -E OUT/oar_job.%jobid%.error
where, train.sh looks like this:

#!/bin/bash

# This is to activate a python environment with conda
source ~/.bashrc
conda activate venv

# Run the training script
python train.py

File transfer

You can easily transfer files between your local machine and Grid’5000 using rsync:

rsync -avzP /path/to/fileOrFolder/ username@nancy.g5k:/srv/storage/talc3@storage4.nancy.grid5000.fr/multispeech/calcul/users/username

You can reverse the two paths for transferring from Grid5k to your computer.

Job monitoring

You can see the real-time status of Grid’5000 following this link.

You can also monitor the status of your launched jobs by typing oarstat | grep username in the command line.

To delete your job, simply run oardel job_id, where job_id is the ID assigned to your job.

To connect to your running job, run oarsub -C job_id.

You can run watch -n 1 'nvidia-smi' to see the status of the GPU machine over time.

Datasets

You can find several datasets (corpora) on Grid5000 in the following directories:

Setting up Jupyter notebook

When you launch a job in the interactive mode, e.g., for debugging your codes, it’s more comfortable to have a visual user interface. This can be achieved by Jupyter notebook, for which you will first need reserve a node on Grid5000 (cf. Reserving nodes on Grid5k) and then install Jupyter lab in your virtual environment by pip install jupyterlab.

Once done, follow these steps (you can also connect to your running job in the passive mode (cf. Job monitoring), and launch a Jupyter notebook):

  1. Activate your conda env, and then run this: jupyter lab --ip 0.0.0.0:
>> source ~/.bashrc
>> conda activate venv
>> jupyter lab --ip 0.0.0.0

This will give you a URL link.

  1. On your local machine run ssh g5k -L 8888:grele-10.nancy.grid5000.fr:8888 -v, where grele-10 is the node’s name that you have booked. Also, make sure that the port 8888 is the same as the one given in the output of the previous step.
  2. Paste the URL link given in Step 1 in your browser.

Important- When your computer gets disconnected from the Internet, which could happen, e.g., when your computer goes into sleep, you will lose the interactive session and have to redo all the above steps, which is annoying. To avoid this, you can use the GNU screen, to set up a protected screen session.

In order to do that, once you connect to Grid’5000, e.g., following ssh username@nancy.g5k, run a screen session by typing screen, and then launch your interactive job, e.g., oarsub -l gpu=1,walltime=00:30:00 -I -q production. You can then follow Step 1 above. Once you get the URL link, you need to press CTRL+a, d to detach from your screen session, and then follow Steps 2-3 above.

To attach to the screen session, type screen -d -r. If you have multiple screen sessions running, you might have to give the ID of the session that you’re going to attach to. More information on screen sessions.

Visual Studio Code (VSC) for remote code editing and notebooks running

Prepared by Antoine Bruez.

It can be really helpful to use your favorite IDE for editing remote source codes and run Jupyter notebooks. This is made possible by configuring VSC (Visual Studio Code) for SSH connections and running notebooks with g5h-hosted kernels. Additional help can be found on the official website.

Summary

  1. Prerequisites

  2. Setup SSH VSCode

  3. Run jupyter notebooks in VSC using remote kernels (hosted in g5k)

Prerequisites

Setup SSH VSC

Runing jupyter notebooks in VSC using remote kernels (hosted on G5K)

When dealing with a Jupyter notebook, one could be interested in editing it in VSC while running it on a G5K node. Let’s see how to do so.

Other Grid5000 resources