How to Efficiently use Remote Server’s Docker for Deep Learning
I’ve recently had a chance to set up our lab’s remote server and was hard to find a clear guideline for what I wanted. I was trying to manage each user’s server usage well-contained inside a Docker while being able to efficiently upload the local python project. Please note that this article is based on a local connecting to a Docker inside a Ubuntu remote server.
Why Docker?
I had a doubt about using Docker and because this is my first time using it, I kept wondering if this really is the best choice. Even though Docker provides a neat way to transfer the total package of environmental settings at once, it still required building and installing all the necessary libraries which also could be done in a virtual environment. However, with multiple users running several instances at the same time all requiring different versions of CUDA and cuDNN left me no choice but to use Docker.
Why PyCharm?
PyCharm is definitely one of my favorite IDE, especially with the python project. In my experience, Pycharm provides the best UI/UX for managing python interpreter configurations and with the professional or education editions, you can easily manage a remote server’s python interpreter inside a Docker. Also, I found out that PyCharm enables automatic synchronization, upload, and download with the project files from local to a remote server which makes it super efficient to make constant changes to local files and no need to update the ones at a server every single time.
Step 1: Install Docker on Local and Remote Server
If Docker is not installed on your local or remote server, please follow the installation guide to install Docker on both local computer and remote server before moving on to the next step!
If you need to enable GPU support for deep learning, then you need to additionally set up NVIDIA Container Toolkit on the remote server. Assuming that you've already installed Docker, you need to run the following commands on the server:
distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \
&& curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add - \
&& curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.listsudo apt-get updatesudo apt-get install -y nvidia-docker2sudo systemctl restart docker
Step 2: Build Local Project Environment to Docker Image
Open your local python project on either PyCharm Professional or PyCharm Education, then create Dockerfile inside a root directory.
FROM ubuntu:20.04
MAINTAINER NAME EMAIL
ENV PATH /usr/local/bin:$PATH
ENV LANG C.UTF-8
RUN apt-get update
RUN apt-get install -y software-properties-common
RUN apt-get update
RUN apt-get install -y python3.8 python3.8-dev python3-pip python3-setuptools python3-wheel gcc
RUN apt-get install -y ffmpeg libsm6 libxext6 freeglut3-dev
RUN apt-get install -y git cmake
RUN python3.8 -m pip install pip --upgrade
RUN mkdir -p /PROJECT_NAME
WORKDIR /PROJECT_NAME
ADD requirements.txt /PROJECT_NAME
RUN pip install -r requirements.txtThis is a sample Dockerfile using ubuntu:20.04 as a base image. You can change the following image to the preferred Ubuntu version or ones with CUDA and cuDNN installed. Various options can be found in Docker Hub.
As you can see, it builds an image based on Ubuntu 20.04 and installs Python 3.8 for the starter. Also, I needed various graphics libraries and CMake to successfully install all the required Python libraries. It’s actually your choice to install the Python libraries after creating a container, but I thought it would be better to set up all the environments beforehand only once since I’m going to run the same project multiple times with only a difference in training code.
Also, make sure you have the requirements.txt file inside a root directory.
pip freeze > requirements.txtThen run a Docker build command as below:
sudo docker build -t IMAGE_NAME .Save generated Docker image to remote server and load.
sudo docker save IMAGE_NAME | ssh -C HOST_NAME@HOST_IP_ADDRESS docker loadStep 3: Make Remote Server’s Docker Daemon Listens to Static Port
Although making the Docker daemon listen to a static port may raise security issues, it is a necessary step to connect the Docker from the local PyCharm.
First, ssh connect to a remote server by the command below:
ssh HOST_NAME@HOST_IP_ADDRESSOn your remote server, set up a static port usually 2375 as default and you need to keep the connection alive while training to connect via PyCharm. But no need to worry about closing the local IDE nor static port connection, because the instance stays alive once the training is initiated and also can reconnect via PyCharm to see the logs.
sudo service docker stopsudo dockerd -H unix:///var/run/docker.sock -H tcp://HOST_IP_ADDRESS:2375
Step 4: Connect Remote Server’s Docker via PyCharm
Go to File > Settings > Build, Execution, Deployment > Docker
Add a Docker with a TCP socket pointing to the server with the port number you used.
Then, go to the Deployment section and add a new server with SFTP type. You can add a new SSH configuration with your remote server on default port 22.
On the Mappings tab, you should add the deployment path usually in /tmp directory. This is where your local project is going to be uploaded inside a remote server.
Also, you can add excluded paths to disable the deployment of directories such as /venv or any unnecessary files.
Lastly, under Project > Python Interpreter, add a Python interpreter connected to the remote server’s Docker image and map project files to the directory inside a Docker container.
Step 5: Deploy Local Python Project Files and Add Run/Debug Configuration
Once you right-click on your root directory, you can see the Deployment > Upload to remote option. You can check the upload progress on the bottom File Transfer tab.
After the deployment is done, you should add Run/Debug Configurations for the python script you intend to run. If you want to run multiple instances at the same time, you should allow a parallel run on the top. Then, you need to add some container settings on the bottom.
The container path and the host path are the ones that we’ve used above. Each of the running options that I’ve used means the following:
--rm: remove container after done
--gpus all: enable all GPU support
--shm-size: shared memory size
-d: detached mode, container running in the background
-t: allocate a pseudo-ttyNow you can enjoy deep learning with GPU support inside a Docker on a remote server!
