# MPI run error with cuda **URL:** \ **Category:** General\ **Created:** [20 June 2022 05:17 UTC](https://pyfr.discourse.group/t/mpi-run-error-with-cuda/626 "2022-06-20T05:17:58Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![luli](https://avatars.discourse-cdn.com/v4/letter/l/5f9b8f/32.png) [@luli](https://pyfr.discourse.group/u/luli)\ **Post date:** [20 June 2022 05:17 UTC](https://pyfr.discourse.group/t/mpi-run-error-with-cuda/626/1 "2022-06-20T05:17:58Z")
I install pyfr1.14.0 and run examples/euler\_vortex\_2d case with mpi, and I got an error like this : ![image](https://global.discourse-cdn.com/free1/uploads/pyfr/original/1X/6881e2f2cfe75579fe346b13c57f32ad4e326cf2.png) What’s wrong with me ? ---
**Author:** ![fdw](https://avatars.discourse-cdn.com/v4/letter/f/a587f6/32.png) [@fdw](https://pyfr.discourse.group/u/fdw)\ **Post date:** [20 June 2022 13:08 UTC](https://pyfr.discourse.group/t/mpi-run-error-with-cuda/626/2 "2022-06-20T13:08:03Z")
How many GPUs do you have in your system? Regards, Freddie. ---
**Author:** ![WillT](https://avatars.discourse-cdn.com/v4/letter/w/bc79bd/32.png) [@WillT](https://pyfr.discourse.group/u/WillT)\ **Post date:** [20 June 2022 13:45 UTC](https://pyfr.discourse.group/t/mpi-run-error-with-cuda/626/3 "2022-06-20T13:45:48Z")
Are there cuda enabled device available on the node you are working on? And is Cuda available? Try `nvcc --version` and `nvidia-smi` ---
**Author:** ![luli](https://avatars.discourse-cdn.com/v4/letter/l/5f9b8f/32.png) [@luli](https://pyfr.discourse.group/u/luli)\ **Post date:** [21 June 2022 01:45 UTC](https://pyfr.discourse.group/t/mpi-run-error-with-cuda/626/4 "2022-06-21T01:45:33Z")
In this system I have only one A100 GPU ---
**Author:** ![luli](https://avatars.discourse-cdn.com/v4/letter/l/5f9b8f/32.png) [@luli](https://pyfr.discourse.group/u/luli)\ **Post date:** [21 June 2022 01:47 UTC](https://pyfr.discourse.group/t/mpi-run-error-with-cuda/626/5 "2022-06-21T01:47:53Z")
![image](https://global.discourse-cdn.com/free1/uploads/pyfr/original/1X/6e2d1e406901b06082e958afa5744f6a700513c3.png) ---
**Author:** ![fdw](https://avatars.discourse-cdn.com/v4/letter/f/a587f6/32.png) [@fdw](https://pyfr.discourse.group/u/fdw)\ **Post date:** [21 June 2022 02:05 UTC](https://pyfr.discourse.group/t/mpi-run-error-with-cuda/626/6 "2022-06-21T02:05:02Z")
If you only have a single GPU then you should only be running with a single MPI rank. When two ranks are launched the first rank will request the first GPU and the second rank will request the second GPU. However, if your system only has a single GPU then this will yield an invalid device error. Regards, Freddie. ---
**Author:** ![luli](https://avatars.discourse-cdn.com/v4/letter/l/5f9b8f/32.png) [@luli](https://pyfr.discourse.group/u/luli)\ **Post date:** [21 June 2022 02:18 UTC](https://pyfr.discourse.group/t/mpi-run-error-with-cuda/626/7 "2022-06-21T02:18:56Z")
However, in Pyfr 1.13.0 I can run this example with 2 mpi rank with cuda, is mpi automatically assigned in the new version? ---
**Author:** ![fdw](https://avatars.discourse-cdn.com/v4/letter/f/a587f6/32.png) [@fdw](https://pyfr.discourse.group/u/fdw)\ **Post date:** [21 June 2022 16:41 UTC](https://pyfr.discourse.group/t/mpi-run-error-with-cuda/626/8 "2022-06-21T16:41:23Z")
The behaviour of the CUDA backend was changed in the most recent release to be in line with that of HIP and OpenCL. Running multiple ranks on the same GPU results in very bad performance. Regards, Freddie. ---
**Author:** ![luli](https://avatars.discourse-cdn.com/v4/letter/l/5f9b8f/32.png) [@luli](https://pyfr.discourse.group/u/luli)\ **Post date:** [22 June 2022 10:33 UTC](https://pyfr.discourse.group/t/mpi-run-error-with-cuda/626/9 "2022-06-22T10:33:14Z")
Thank you very much ! ---
**Author:** ![nnunn](https://avatars.discourse-cdn.com/v4/letter/n/c89c15/32.png) [@nnunn](https://pyfr.discourse.group/u/nnunn)\ **Post date:** [27 June 2022 11:41 UTC](https://pyfr.discourse.group/t/mpi-run-error-with-cuda/626/10 "2022-06-27T11:41:15Z")
Hi @luli, **Just a thought** : in case you are trying to develop or validate MPI codes, you can set your single A100 to run as 7 “distinct” GPUs, and then launch 7 MPI ranks: > **[Getting the Most Out of the NVIDIA A100 GPU with Multi-Instance GPU | NVIDIA...](https://developer.nvidia.com/blog/getting-the-most-out-of-the-a100-gpu-with-multi-instance-gpu/)** > > With the third-generation Tensor Core technology, NVIDIA recently unveiled A100 Tensor Core GPU that delivers unprecedented acceleration at every scale for AI, data analytics… This way (if one has an **A100** ) one can develop for exascale on a (10kg micro ATX) shoebox!