Which GPU(s) to Get for Deep Learning: My Experience and Advice for Using GPUs in Deep Learning

Why should I get a GPU?

I have been using GPUs for nearly two years now and for me it is again and again amazing to see how much speedup you get. Compared to CPUs 20x speedups are typical, but on larger problems one can achieve 50x speedups. With GPUs you can try out new ideas, algorithms and experiments much faster than usual and get almost immediate feedback as to what works and what does not. This is a very important aspect when one begins to do deep learning as this rapid gain in practical experience is key to build the expertise with which you can make deep learning work on new problems. Without this rapid feedback it just takes too much time to learn from one’s mistakes and it can be discouraging and frustrating to go on with deep learning.

With GPUs I quickly learned how to apply deep learning on a range of Kaggle competitions and I managed to earn second place in the Partly Sunny with a Chance of Hashtags Kaggle competition, where it was the task to predict weather ratings for a given tweet. In the competition I used a rather large two layered deep neural network with rectified linear units and dropout for regularization and this deep net fitted barely into my 6GB GPU memory. More details on my approach can be found here.

Should I get multiple GPUs?

Excited by what deep learning can do with GPUs I plunged myself into multi-GPU territory by assembling a small GPU cluster with InfiniBand 40Gbit/s interconnect. I was thrilled to see if even better results can be obtained with multiple GPUs.

Setup in my main computer: You can see three GXT Titan and an InfiniBand card. Is this a good setup for doing deep learning?

I quickly found that it is not only very difficult to parallelize neural networks on multiple GPU efficiently, but also that the speedup was only mediocre for dense neural networks. Small neural networks could be parallelized rather efficiently using data parallelism, but larger neural networks like I used in the Partly Sunny with a Chance of Hashtags Kaggle competition received almost no speedup.

However, using model parallelism, I was able to train neural networks that were much larger and that had almost 3 billion connections. But to leverage these connections one needs just much larger data sets than are normally used. I found some uses for that when I trained a language model on the entire Wikipedia corpus – but that’s about it.

On the other hand, one advantage of multiple GPUs is that you can run multiple algorithms or experiments separately on each GPU. This is highly useful if your main goal is to gain deep learning experience as quickly as possible. You gain no speedups, but you get more information of your performance by using different algorithms or parameters at once.

If you use deep learning only occasionally, or you use rather small data sets (smaller than say 10-15GB) and foremost dense neural networks, then multiple GPUs are probably not for you. However, when you use convolutional neural networks a lot, then multiple GPUs might still make sense.

Alex Krizhevsky released his new updated kernels which can run convolutional neural networks on up to four GPUs. Convolutional neural networks – unlike dense neural networks – can be run very efficiently on multiple GPUs because their use of weight sharing makes data parallelism very efficient. On top of that, Alex Krizhevsky’s implementation utilizes model parallelizm for the densely connected final layers of the network.

However, if you want to program similar networks for yourself, be aware that to program efficient convolutional kernels for multiple GPUs is a very difficult undertaking for which expert GPU programming skills are required.

So overall, one can say that one GPU should be sufficient for almost any task and that additional GPUs convey only benefits under very specific circumstances.

So what kind of GPU should I get?

Required memory size

People often ask me if the GPUs with the largest memory are best for them, as this would enable them to run the largest neural networks. I thought like this when I bought my GPU, a GTX Titan with 6GB memory. And I also thought that this was a good choice when my neural network in the Partly Sunny with a Chance of Hashtags Kaggle competition barely fitted in my GPU memory. But later I found out that my neural network implementation was very memory inefficient and much less memory would have been sufficient.

Generally, if you want to know how large a neural network you could fit into a given GPU memory then subtract 400 MB as a buffer, then divide the GPU memory by two for the momentum matrix, then multiply the memory in MB by to get bytes, and finally divide by four get the number of floating point numbers fitting into that memory. For a GTX Titan this would be parameters. For the Kaggle competition I used a 9000x4000x4000x32 network, which are just 52.128.000 parameters (with momentum matrix). So this network would even fit into a small GPU with 1536 MB (space for about 148.897.792 parameters). For comparison, Alex Krizhevsky’s convolutional neural network for the ImageNet competition featured 60.000.000 parameters (120 million with momentum matrix).

So the bottom line is if you just want to use neural networks on some Kaggle data sets and build normal models, you should not worry too much about memory ( is fine). For cutting edge models that are used on very large amounts of data you should have more than 3GB.

Update 2014-09-28: Sander Dieleman made me aware, that this last sentence should be emphasized: If you really want to work on large data sets, you should go for either 4GB or 6GB (depending on your budget). He also pointed out an error in my reasoning with parameters: Convolutional kernels need additional memory to run, i.e. they need additional memory beyond the parameters alone. So Alex’s net – even though it features 120 million parameters, does not fit into 3GB memory. So my calculation above are off for convolutional neural networks (which are arguably the most important neural networks to date). Otherwise this blog post contains all the information you need to make a fully informed choice for your deep learning GPU. I will update the post as more information regard GTX 980 performance is known (first currency mining tests indicate, that the GTX 980 will be the best GPU. Although it’s stats are bad, it’s architecture seems to make more than just up for it! More later).

Fastest GPU for a given budget

Processing performance is most often measured in floating-point operations per second (FLOPS). This measure is often advertised in GPU computing and it is also the measure which determines which supercomputer enters the TOP500 list of the fastest supercomputers. However, this measure is misleading, as it measures processing power on problems that do not occur in practice.

It turns out that the most important practical measure for GPU performance is bandwidth in GB/s, which measures how much memory can be read and written per second. This is because almost all mathematical operations, such as dot product, sum, addition etcetera, are bandwidth bound, i.e. limited by the GB/s  of the card rather than its FLOPS.

Comparison of bandwidth for CPUs and GPUs over time. Bandwidth is one of
the main reasons why GPUs are faster for computing than CPUs are.

To determine the fastest GPU for a given budget one can use this Wikipedia page and look at Bandwidth in GB/s; the listed prices are quite accurate for newer cards (600 and 700 series), but the 400 and 500 series is significantly cheaper than the listed prices – especially if you buy those cards via eBay.

Another important factor to consider however, is that the Fermi architecture (400 and 500 series) is quite a bit faster than the newer Kepler architecture (600 and 700 series), so that for example the GTX 580 is faster than any GTX 600 series GPU. Only high end 700 series GPUs (770, 780, Titan) outpace the GTX 580.

So the only disadvantage a GTX 580 has is its smaller memory of 1.5 or 3GB – which as discussed above – is not so bad after all. With an eBay price of $150-200 the GTX 580 is rather cheap (compared to the 770, 780 and Titan, that is) while delivering excellent performance with a good amount of memory.

For special applications one can easily reason which GPU to choose by balancing ones required memory size, bandwidth in GB/s and the price of the GPU, and this reasoning will be solid for many years to come. But right now and more generally, choosing a GTX 580 will be the best and most cost-effective choice for most deep learning application that does not involve (very) large neural networks.

 

[Image source: NVIDIA CUDA/C Programming Guide]

About these ads