How to Efficiently use Remote Server’s Docker for Deep Learning

Photo credit to flickr

I’ve recently had a chance to set up our lab’s remote server and was hard to find a clear guideline for what I wanted. I was trying to manage each user’s server usage well-contained inside a Docker while being able to efficiently upload the local python project. Please note that this article is based on a local connecting to a Docker inside a Ubuntu remote server.

Why Docker?

I had a doubt about using Docker and because this is my first time using it, I kept wondering if this really is the best choice. Even though Docker provides a neat way to transfer the total package of environmental settings at once, it still required building and installing all the necessary libraries which also could be done in a virtual environment. However, with multiple users running several instances at the same time all requiring different versions of CUDA and cuDNN left me no choice but to use Docker.

Why PyCharm?

PyCharm is definitely one of my favorite IDE, especially with the python project. In my experience, Pycharm provides the best UI/UX for managing python interpreter configurations and with the professional or education editions, you can easily manage a remote server’s python interpreter inside a Docker. Also, I found out that PyCharm enables automatic synchronization, upload, and download with the project files from local to a remote server which makes it super efficient to make constant changes to local files and no need to update the ones at a server every single time.

Step 1: Install Docker on Local and Remote Server

If Docker is not installed on your local or remote server, please follow the installation guide to install Docker on both local computer and remote server before moving on to the next step!

If you need to enable GPU support for deep learning, then you need to additionally set up NVIDIA Container Toolkit on the remote server. Assuming that you've already installed Docker, you need to run the following commands on the server:

Step 2: Build Local Project Environment to Docker Image

Open your local python project on either PyCharm Professional or PyCharm Education, then create Dockerfile inside a root directory.

This is a sample Dockerfile using ubuntu:20.04 as a base image. You can change the following image to the preferred Ubuntu version or ones with CUDA and cuDNN installed. Various options can be found in Docker Hub.

As you can see, it builds an image based on Ubuntu 20.04 and installs Python 3.8 for the starter. Also, I needed various graphics libraries and CMake to successfully install all the required Python libraries. It’s actually your choice to install the Python libraries after creating a container, but I thought it would be better to set up all the environments beforehand only once since I’m going to run the same project multiple times with only a difference in training code.

Also, make sure you have the requirements.txt file inside a root directory.

Then run a Docker build command as below:

Save generated Docker image to remote server and load.

Step 3: Make Remote Server’s Docker Daemon Listens to Static Port

Although making the Docker daemon listen to a static port may raise security issues, it is a necessary step to connect the Docker from the local PyCharm.

First, ssh connect to a remote server by the command below:

On your remote server, set up a static port usually 2375 as default and you need to keep the connection alive while training to connect via PyCharm. But no need to worry about closing the local IDE nor static port connection, because the instance stays alive once the training is initiated and also can reconnect via PyCharm to see the logs.

Step 4: Connect Remote Server’s Docker via PyCharm

Go to File > Settings > Build, Execution, Deployment > Docker

Add a Docker with a TCP socket pointing to the server with the port number you used.

Then, go to the Deployment section and add a new server with SFTP type. You can add a new SSH configuration with your remote server on default port 22.

On the Mappings tab, you should add the deployment path usually in /tmp directory. This is where your local project is going to be uploaded inside a remote server.

Also, you can add excluded paths to disable the deployment of directories such as /venv or any unnecessary files.

Lastly, under Project > Python Interpreter, add a Python interpreter connected to the remote server’s Docker image and map project files to the directory inside a Docker container.

Step 5: Deploy Local Python Project Files and Add Run/Debug Configuration

Once you right-click on your root directory, you can see the Deployment > Upload to remote option. You can check the upload progress on the bottom File Transfer tab.

After the deployment is done, you should add Run/Debug Configurations for the python script you intend to run. If you want to run multiple instances at the same time, you should allow a parallel run on the top. Then, you need to add some container settings on the bottom.

The container path and the host path are the ones that we’ve used above. Each of the running options that I’ve used means the following:

Now you can enjoy deep learning with GPU support inside a Docker on a remote server!