# About running PyFR on multiple GPUs **URL:** \ **Category:** General\ **Created:** [15 April 2015 10:36 UTC](https://pyfr.discourse.group/t/about-running-pyfr-on-multiple-gpus/63 "2015-04-15T10:36:47Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![fdw](https://avatars.discourse-cdn.com/v4/letter/f/a587f6/32.png) [@fdw](https://pyfr.discourse.group/u/fdw)\ **Post date:** [15 April 2015 10:36 UTC](https://pyfr.discourse.group/t/about-running-pyfr-on-multiple-gpus/63/1 "2015-04-15T10:36:47Z")
Hi, Freddie! I am curious about how to use the multiple CUDA GPU cards in a single node. Every-time I use MPI with CUDA, there is only one GPU per node could be used. I have checked out some information online, and find that, this problem seems related with the MPI implementation adopted (For me, I used Intel MPI), as well as how the program itself is written. Thanks for any help! Zhen ---
**Author:** ![fdw](https://avatars.discourse-cdn.com/v4/letter/f/a587f6/32.png) [@fdw](https://pyfr.discourse.group/u/fdw)\ **Post date:** [15 April 2015 10:41 UTC](https://pyfr.discourse.group/t/about-running-pyfr-on-multiple-gpus/63/2 "2015-04-15T10:41:54Z")
Hi Zhen, > I am curious about how to use the multiple CUDA GPU cards in a > single node. Every-time I use MPI with CUDA, there is only one GPU > per node could be used. > > I have checked out some information online, and find that, this > problem seems related with the MPI implementation adopted (For me, > I used Intel MPI), as well as how the program itself is written. By default the CUDA backend uses a 'round-robin' strategy to decide which GPU to use. The strategy tries to create a CUDA context on each CUDA capable device in the system until one succeeds. It is intended to be used when the GPUs are in 'compute exclusive' mode. Alternatively, you can set the device-id key in [backend-cuda] to be 'local-rank'. Here PyFR will use the node-local MPI rank to determine which CUDA device to use. Further information on these options can be found in the user guide. Regards, Freddie. ---
**Author:** ![Zhen\_Zhang](https://avatars.discourse-cdn.com/v4/letter/z/2acd7d/32.png) [@Zhen\_Zhang](https://pyfr.discourse.group/u/Zhen_Zhang)\ **Post date:** [17 April 2015 14:53 UTC](https://pyfr.discourse.group/t/about-running-pyfr-on-multiple-gpus/63/3 "2015-04-17T14:53:17Z")
Hi Freddie, Thanks a lot, but you may misunderstand my idea. Assuming I have two CUDA GPUs on a single node, and I want to partition a mesh into two parts and solve two partition on two GPUs simultaneously. I can set the `devid`, but it is a sole number, which targets at only one GPU. But for local-rank, MPI ( I used MVAPICH2) gives all two processes to a single card. Thank you a lot! ---
**Author:** ![bvermeir](https://avatars.discourse-cdn.com/v4/letter/b/a698b9/32.png) [@bvermeir](https://pyfr.discourse.group/u/bvermeir)\ **Post date:** [17 April 2015 14:56 UTC](https://pyfr.discourse.group/t/about-running-pyfr-on-multiple-gpus/63/4 "2015-04-17T14:56:44Z")
Hi Zhen, It could be the case that you do not have your cards set to “compute exclusive mode”. If you check nvidia-smi -h you should see the following option: -c, --compute-mode= Set MODE for compute applications: 0/DEFAULT, 1/EXCLUSIVE\_THREAD, 2/PROHIBITED, 3/EXCLUSIVE\_PROCESS You should try running nvidia-smi -c 3 to allow only one process per card. ---
**Author:** ![fdw](https://avatars.discourse-cdn.com/v4/letter/f/a587f6/32.png) [@fdw](https://pyfr.discourse.group/u/fdw)\ **Post date:** [17 April 2015 15:12 UTC](https://pyfr.discourse.group/t/about-running-pyfr-on-multiple-gpus/63/5 "2015-04-17T15:12:24Z")
Hi Zhen, > Thanks a lot, but you may misunderstand my idea. Assuming I have two > CUDA GPUs on a single node, and I want to partition a mesh into two > parts and solve two partition on two GPUs simultaneously. This is exactly what you want: [backend-cuda] device-id = local-rank for. (Or use the compute-exclusive solution as outlined by Brian.) > I can set the `devid`, but it is a sole number, which targets at only > one GPU. But for local-rank, MPI ( I used MVAPICH2) gives all two > processes to a single card. Can you attach the config file you're using? Also, as an aside it is perhaps also worth noting that almost all config file options support expansion of environmental variables. So with MVAPICH2: [backend-cuda] device-id = ${MV2\_COMM\_WORLD\_LOCAL\_RANK} is basically the same as local-rank. Regards, Freddie. ---
**Author:** ![p.vincent](https://yyz2.discourse-cdn.com/free1/user_avatar/pyfr.discourse.group/p.vincent/32/225_2.png) [@p.vincent](https://pyfr.discourse.group/u/p.vincent)\ **Post date:** [17 April 2015 15:13 UTC](https://pyfr.discourse.group/t/about-running-pyfr-on-multiple-gpus/63/6 "2015-04-17T15:13:27Z")
Hi Zhen, As Freddie and Brian suggest, I would try using round-robin mode (the default), but with your GPUs in compute exclusive mode. Cheers Peter ---
**Author:** ![Zhen\_Zhang](https://avatars.discourse-cdn.com/v4/letter/z/2acd7d/32.png) [@Zhen\_Zhang](https://pyfr.discourse.group/u/Zhen_Zhang)\ **Post date:** [17 April 2015 15:57 UTC](https://pyfr.discourse.group/t/about-running-pyfr-on-multiple-gpus/63/7 "2015-04-17T15:57:16Z")
Thank you, Brian and Vincent! I tried Brian’s solution (use nvidia-smi to set the mode) and it works. I searched a lot about ranks and binding but the solution turned out to be so simple (and a little hack maybe? since it also eliminated the possibility of run multiple processes evenly on multiple cards) And as a (maybe) off-topic discussion, take this solution as an example, how is the communication between processes done? Will the data from one card be move t the main memory and moved towards another cards? Or the cards can go the PCIe Bus directly? P.S. The direct communication seems to be called the GPUDirect technology. Thanks! Zhen ---
**Author:** ![fdw](https://avatars.discourse-cdn.com/v4/letter/f/a587f6/32.png) [@fdw](https://pyfr.discourse.group/u/fdw)\ **Post date:** [17 April 2015 16:05 UTC](https://pyfr.discourse.group/t/about-running-pyfr-on-multiple-gpus/63/8 "2015-04-17T16:05:44Z")
Hi Zhen, > And as a (maybe) off-topic discussion, take this solution as an > example, how is the communication between processes done? Will the > data from one card be move t the main memory and moved towards > another cards? Or the cards can go the PCIe Bus directly? > > P.S. The direct communication seems to be called the GPUDirect > technology. There is a fork of PyFR that can exploit CUDA-aware-MPI. This permits PyFR to pass CUDA device pointers directly to MPI functions and have the MPI library handle the copying. This may, or may not, use peer-to-peer GPU copies or perhaps even GPUDirect over RDMA. If there is any performance benefit depends heavily on the problem being solved, the platform it is being solved on, and the MPI library. As PyFR already tries very hard to overlap communication with computation, and because for large transfer sizes most MPI libraries will fall back to copying via the host, the benefit is usually quite small. Regards, Freddie.