Run OpenMM Simulations in Docker with GPU support

Want to run MD simulation in Docker container, so that conflicting needs for CUDA versions can be avoided? Want to containerize everything so that different environments can be handled smoothly?

Docker containers are gaining popularity in recent years. Its shared host OS kernel, easy deployment in cloud computing, and wide usage in the industry have made it a choice other than virtual machine.

Welcome on-board for a Docker journey! Here we will use OpenMM as an example to demonstrate such a practice. And definitely we want GPU-supported OpenMM. (Nothing better than getting the jobs done sooner :)

Pre-requisites:

a. Have a simulation assembly ready.
b. Installed Docker.
c. Installed NVIDIA support for docker.

I provide an example here, which you can download from Github. You can get installation instructions for Docker on Mac, Window, Ubuntu and other Linux machines. And nvidia-docker installation for GPU support. (Check out the links for installation)

1. Pull Docker image with OpenMM-CUDA installed.

First, pull a Docker image from Docker hub that has OpenMM with GPU support: docker pull sha256feng/openmm-with-cuda . List the images we have on local machine by docker images .

List Docker images in local device

2. Create a Docker volume for linking to system files

We want to create a Docker volume in advance so that inside the container, programs can access the simulation files in our system. Otherwise, the daemon is totally traveling in an isolated ubuntu world and not working because it does not know where the heck those files are. The command below will create a volume that binds to ~/Documents/simulations , where we put all our simulation systems. The system we want to simulate is under folder ~/Documents/simulations/a-sample-lipid-simu .

docker volume create --driver local \
      --opt type=none \
      --opt device=~/Documents/simulations \
      --opt o=bind \
      simu_vol

We can verify the volume by docker volume ls , which would show

$ docker volume ls
DRIVER              VOLUME NAME
local               simu_vol

3. Initialize a Docker container

The image is like a factory function. To run MD simulations in Docker, we need to a working machine, which is built from the factory function. The command below means that we initialize a container from the image called redesignscience/openmm-cuda , the container name is ommcuda_app for future reference, -i means interactive, -t means opening a pseudo tty session, --mount source=simu_vol,target=/data means mount the ‘simu_vol’ we made to the ‘/data’ folder in container, --runtime nvidia enables NVIDIA runtime to run CUDA-supported GPU jobs.

docker run -it --rm --mount source=simu_vol,target=/data --runtime nvidia --name ommcuda_app sha256feng/openmm-with-cuda /bin/bash

This would create a container and we are in a different terminal now, which is inside the container. Because it is interactive, we can actually play around with it by ls /data the folder where files under system folder ~/Documents/simulations will all showup.

4. Check if OpenMM can run in the container

Type the following command in the container:

$python -m simtk.testInstallation
OpenMM Version: 7.4.1
Git Revision: 068f120206160d5151c9af0baf810384bba8d052
There are 3 Platforms available:
1 Reference - Successfully computed forces
2 CPU - Successfully computed forces
3 CUDA - Successfully computed forces
Median difference in forces between platforms:
Reference vs. CPU: 6.32164e-06
Reference vs. CUDA: 6.7308e-06
CPU vs. CUDA: 7.43677e-07
All differences are within tolerance.

This command shows you what computation systems are there and can help detect the GPUs.

Go to the CHARMM-GUI download assembly, and go to OpenMM folder, we will use the following script: run_job.csh to run simulations.

Then try to run sh run_job.sh and you can check the GPU utilization outside the container by typing in another terminal nvidia-smi . Yeah, the GPU 0 is used for OpenMM simulation. we have been progressing successfully on this road! Now ctrl+d to leave this interactive tty.

Output of “nvidia-smi” command

5. Run Docker detached

-d is detach, --rm is remove the container after exit, --mount is to mount the Docker volume from source to destination, --runtime as nvidia sets the Docker to run on GPU, --name specifies the name of the container, bash is bash terminal, -c is the command we ask docker container to perform.

docker run -d —-rm --mount source=simu_vol,target=/data --runtime nvidia --name ommcuda_app sha256feng/openmm-with-cuda bash -c "cd /data/a-sample-lipid-simu && sh run_job.sh"

Check the job execution by docker ps -a .

6. Alternative to docker run: use docker exec

If we want to keep and reuse the container for OpenMM simulations, we can remove --rm in previous docker run command. Then we can execute a task using docker exec command. This time we need to provide the workdir for execution, -w /data/2apb-2-lip instructs the commands to run in the folder /data/2apb-2-lip . -d is detach, bash is bash terminal, -c is the command we ask docker container to perform.

docker exec -d -w /data/a-sample-lipid-simu ommcuda_app bash -c ". /opt/conda/etc/profile.d/conda.sh && conda activate redesign && sh run_job.sh"

Check the job execution by docker ps -a .

Thank you for going through this Docker journey with me! Please reply below if you have any comments or questions. The most blogs I read about Docker, authors would humbly say they are not Docker wizard and welcome comments for smarter solutions. This applies to me as well.

Acknowledgment

Redesign Science, an NYC-based drug discovery biotech company, generously shared with me their Dockerfile and entrypoint.sh file for running OpenMM. My friend Shiyu Liu, a PhD student in the Computational Biology program at Duke, guided me onto the road of Docker. Thanks for all the help.