back
32 comments
If you're using DC/OS you can also run docker containers that leverage GPUs. We worked with Nvidia to mimic the same functionality provided by nvidia-docker. Here's a talk I gave at this years Nvidia GTC conference explaining how it works:

https://drive.google.com/file/d/0B7vZqCY-AJrpMEYyVmZkVGlyOEE...

https://docs.google.com/presentation/d/1q6brajoQkFZVtSocMPGg...

We use this in production every day. We've hit a couple bugs, but it's a surprisingly stable piece of software.

We hit an issue with user namespaces, and it not correctly chowning things when "exporting" them into Docker containers. I honestly forget how we solved this, but I believe it was just an upgrade.

We hit an issue with the first run of a Docker container on a system being extremely slow, to the point where downstream systems would give up, and retry. The issue was fixed by starting a container that leverages the driver at boot time, "warming" it up.

Given the description of your workaround, you may be running into an issue of long GPU driver load times for the nvidia driver. If that is the root cause the recommended solution for this is nvidia-persistenced, though it seems you may have rolled your own equivalent.

https://github.com/NVIDIA/nvidia-persistenced

Docs here http://docs.nvidia.com/deploy/driver-persistence/index.html

Is it possible that the slowness is caused by disk speed of loading a large file from disk?

I've observed the same thing but it seemed to be IO related, not CPU or GPU related.

It's mainly the speed of creating a huge number of small files on disk. When unpacking a docker image into an AUFS file system, it has to create every file in the shadow system. A robust Ubuntu install has around 100000 files. On AWS with default EBS storage, this can take a few minutes. At OpenAI we pre-built AMI instances with file systems pre-unpacked.
As and example, we used (nvidia-)docker images to make our object detection results reproducible:

https://github.com/umautobots/driving-in-the-matrix

docker has been great for this in the lab: only one person now goes through the pain to get latest framework + hack + model tweak work and then the rest can reuse.

Isn't AMD leaving a lot of potential revenue on the table without a comparable amdgpu-docker?

If I'm building an ML cluster, I'm going to go with the vendor that's easiest to containerize and deploy, and right now it looks like nvidia has a commanding advantage in software.

Why doesn't AMD throw a few hundred thousand dollars at some developers to get containerization parity?

Just for information, it's also possible to use NVIDIA GPUs inside Docker containers without using `nvidia-docker`:

https://stackoverflow.com/questions/25185405/using-gpu-from-...

In a nutshell, you can give the container access to the NVIDIA device file via the `--device` flag, so all you need is a container with the NVIDIA drivers. The added benefit of this is that you can use different versions of the drivers side-by-side (in my understanding).

I thought this might be relevant as some people might not want to use the `docker-nvidia` CLI to run containers (I'm not sure how you would use this via the Docker API for example).

We document how this on our wiki: https://github.com/NVIDIA/nvidia-docker/wiki/Internals

> The added benefit of this is that you can use different versions of the drivers side-by-side (in my understanding).

No, you can only have one driver version, the one that correspond to the loaded kernel modules. Installing the driver inside a Docker image makes it non-portable.

Ah thanks for the clarification, I was not aware of this!
It's also possible to set up CUDA in LXC https://sqream.com/cuda-in-lxc-containers/
I use NVidia-Docker extensively in my Open Source project Deep Video Analytics [1] when combined with TensorFlow (which allows explicit GPU memory allocation) its unbeatable in running multiple inference models on a single GPU in a reliable manner. Combining this setup with docker volumes on AWS EFS allows simple multi machine deployments.

[1] https://github.com/AKSHAYUBHAT/DeepVideoAnalytics

this is pretty cool! why do you use multiple packages like torch and tensorflow ?
Certain algorithms/models are implemented in PyTorch or Caffe and typically it's huge amount of work to convert them to TensorFlow while ensuring correctness / Parity. Also I personally like design of PyTorch.
coming from the facebook/reactjs weaponized patent grant problem.. caffe also has the same revocable patent grant.

Tensorflow is apache licensed. I think in general, the perception is that it is far safer to stay away from caffe.

Caffe 1 is developed by Berkeley and I think Apache or BSD licensed. The Patents.txt issue occurs with Caffe 2 which is developed by Facebook.
I wish they would add support for NVIDIA-docker to run on Windows but I think that is a tall order.
One of our engineers recently shared an intro-level talk at the Docker meetup about a ~year of (happy) experiences with this: https://www.meetup.com/Docker-Santa-Clara/events/240641246/ . It should be recorded somewhere, and may be helpful if you're considering production use, not just personal DL stuff.
A collection of deep learning nvidia-docker images to get you started: https://hub.docker.com/r/deepaiorg/

Floydhub also curates most frameworks as nvidia-docker images: https://hub.docker.com/r/floydhub/

Is there a processing-specific user case for this - are we going to see mining containers or game client containers?
Would this be useful for people who wanted to leverage GPUs for deep learning, but didn't have the wherewithal or the willingness to set up all the dependencies on the host machine?
Yes. I have made a Docker container containing Keras and the better CNTK backend (+ Jupyter) and it works great: https://github.com/minimaxir/keras-cntk-docker

There's another unexpected bonus to nvidia-docker; CNTK leaks memory if you abort training prematurely, causing the GPU to go OOM. But since the memory is owned by the nvidia-docker process, killing-and-restarting the container fixes the issue without having to restart the entire system.

It is used by this kind of popular pre-built Docker image for Deep Learning: https://github.com/floydhub/dl-docker

They actually created this service using that (a Deep Learning "as a service"): https://www.floydhub.com/

At FloydHub, we use nvidia-docker in production for running DL jobs. It has been very solid for the past 6 months or so. We have also built a collection of open source DL docker images for various frameworks and we actively maintain them.

[1]: https://hub.docker.com/r/floydhub/ [2]: https://github.com/floydhub/dockerfiles

Exactly what we are using this for. However, I found there was little performance benefit versus running on CPU. Can't figure out why.
Yes, this is a great use case. Search around, there are a few folks putting together dockers to get you out of dl dependency hell
Can I run regular web apps with this on NVIDIA gpu's?
Yes. Ignore the naysayers :-)

Check out VirtuaGL and TurboVNC. It is possible to run these two things inside a docker container, install a DE and accelerate a GUI app. I've been doing it for quite a while.

Also for all those checking the OPs original lib out. It does work and it is great and also probably the most mature option, but my understanding is that future focus for GPU inside containers is with this newer lib, also looked after by Nvidia:

https://github.com/NVIDIA/libnvidia-container

I'm fairly familiar with this stuff, though not an expert, for my startup http://realityzero.one. But, I've not been paying that much attention to this area of late, so if I've got this wrong then please feel free to correct me. Perhaps they're for different things. I know there's been some discussion from the k8n crowd about the newer lib in k8n's github issues.

It allows you to run GPU-accelerated applications (like machine learning, HPC, video/image processing...) inside a Docker container.
No.
I mean, technically I guess you could run it, it just wouldn't make use of the GPU.
Porque no?