# CUBLASInternalError when using multi GPU **URL:** \ **Category:** Errors\ **Created:** [7 November 2022 15:36 UTC](https://pyfr.discourse.group/t/cublasinternalerror-when-using-multi-gpu/714 "2022-11-07T15:36:09Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![hwtang](https://avatars.discourse-cdn.com/v4/letter/h/a87d85/32.png) [@hwtang](https://pyfr.discourse.group/u/hwtang)\ **Post date:** [7 November 2022 15:36 UTC](https://pyfr.discourse.group/t/cublasinternalerror-when-using-multi-gpu/714/1 "2022-11-07T15:36:09Z")
Dear all, I got an error when trying to use 2 different GPUs. The command I used is `mpiexec -n 2 pyfr restart -b cuda -p solutions_01/sd7003.pyfrm solutions_01/sd7003_145.00.pyfrs sd7003_01.ini`. ```auto Traceback (most recent call last): File "/home/tang/anaconda3/envs/pyfr1.15.0/bin/pyfr", line 33, in sys.exit(load_entry_point('pyfr==1.15.0', 'console_scripts', 'pyfr')()) File "/home/tang/anaconda3/envs/pyfr1.15.0/lib/python3.10/site-packages/pyfr-1.15.0-py3.10.egg/pyfr/ __main__.py", line 118, in main File "/home/tang/anaconda3/envs/pyfr1.15.0/lib/python3.10/site-packages/pyfr-1.15.0-py3.10.egg/pyfr/ __main__.py", line 270, in process_restart File "/home/tang/anaconda3/envs/pyfr1.15.0/lib/python3.10/site-packages/pyfr-1.15.0-py3.10.egg/pyfr/ __main__.py", line 247, in _process_common File "/home/tang/anaconda3/envs/pyfr1.15.0/lib/python3.10/site-packages/pyfr-1.15.0-py3.10.egg/pyfr/integrators/base.py", line 115, in run File "/home/tang/anaconda3/envs/pyfr1.15.0/lib/python3.10/site-packages/pyfr-1.15.0-py3.10.egg/pyfr/integrators/std/controllers.py", line 181, in advance_to File "/home/tang/anaconda3/envs/pyfr1.15.0/lib/python3.10/site-packages/pyfr-1.15.0-py3.10.egg/pyfr/integrators/std/steppers.py", line 190, in step File "/home/tang/anaconda3/envs/pyfr1.15.0/lib/python3.10/site-packages/pyfr-1.15.0-py3.10.egg/pyfr/solvers/base/system.py", line 265, in rhs File "/home/tang/anaconda3/envs/pyfr1.15.0/lib/python3.10/site-packages/pyfr-1.15.0-py3.10.egg/pyfr/util.py", line 40, in newmeth File "/home/tang/anaconda3/envs/pyfr1.15.0/lib/python3.10/site-packages/pyfr-1.15.0-py3.10.egg/pyfr/solvers/baseadvecdiff/system.py", line 82, in _rhs_graphs File "/home/tang/anaconda3/envs/pyfr1.15.0/lib/python3.10/site-packages/pyfr-1.15.0-py3.10.egg/pyfr/backends/base/types.py", line 332, in add File "/home/tang/anaconda3/envs/pyfr1.15.0/lib/python3.10/site-packages/pyfr-1.15.0-py3.10.egg/pyfr/backends/cuda/provider.py", line 35, in add_to_graph File "/home/tang/anaconda3/envs/pyfr1.15.0/lib/python3.10/site-packages/pyfr-1.15.0-py3.10.egg/pyfr/backends/cuda/provider.py", line 35, in File "/home/tang/anaconda3/envs/pyfr1.15.0/lib/python3.10/site-packages/pyfr-1.15.0-py3.10.egg/pyfr/backends/cuda/cublas.py", line 103, in add_to_graph File "/home/tang/anaconda3/envs/pyfr1.15.0/lib/python3.10/site-packages/pyfr-1.15.0-py3.10.egg/pyfr/backends/cuda/cublas.py", line 111, in run File "/home/tang/anaconda3/envs/pyfr1.15.0/lib/python3.10/site-packages/pyfr-1.15.0-py3.10.egg/pyfr/ctypesutil.py", line 33, in _errcheck pyfr.backends.cuda.cublas.CUBLASInternalError application called MPI_Abort(MPI_COMM_WORLD, 1) - process 1 ``` Here are the details of my devices: > **Summary** > > ![Screenshot from 2022-11-07 23-37-17](https://global.discourse-cdn.com/free1/uploads/pyfr/original/1X/4f56d5646270ba259398ef74ecce38fff0b7c050.png) The configure file I used: > **Summary** > > [backend] > precision = single > > [backend-cuda] > device-id = local-rank > gimmik-max-nnz = 1024 > mpi-type = standard > > [constants] > gamma = 1.4 > mu = 3.94405318873308E-6 > Pr = 0.72 > M = 0.2 > > [solver-time-integrator] > scheme = rk45 > controller = pi > tstart = 0.0 > dt = 0.00001 > atol = 0.000001 > rtol = 0.000001 > ; safety-fact = 0.5 > ; min-fact = 0.3 > ; max-fact = 1.2 > tend = 185.0 > > [soln-plugin-nancheck] > nsteps = 50 > > [soln-plugin-writer] > dt-out = 5.0 > basedir = ./solutions\_01/ > basename = sd7003\_{t:.2f} > > [soln-plugin-fluidforce-wall] > nsteps = 5 > file = sd7003\_01-wall-forces.csv > header = true > > [solver] > system = navier-stokes > order = 4 > anti-alias = flux > > [solver-interfaces] > riemann-solver = rusanov > ldg-beta = 0.5 > ldg-tau = 0.1 > > [solver-interfaces-line] > flux-pts = gauss-legendre > quad-deg = 11 > quad-pts = gauss-legendre > > [solver-interfaces-quad] > flux-pts = gauss-legendre > quad-deg = 11 > quad-pts = gauss-legendre > > [solver-elements-hex] > soln-pts = gauss-legendre > quad-deg = 11 > quad-pts = gauss-legendre > > [soln-bcs-outlet] > type = char-riem-inv > rho = 1.0 > u = 0.2366431913 > v = 0.0 > w = 0.0 > p = 1.0 > > [soln-bcs-inlet] > type = char-riem-inv > rho = 1.0 > u = 0.2366431913 > v = 0.0 > w = 0.0 > p = 1.0 > > [soln-bcs-wall] > type = no-slp-adia-wall > cpTw = 3.5 > > [soln-ics] > rho = 1.0 > u = 0.2366431913 > v = 0.001 > w = 0.001\*cos(x)\*cos(y) > p = 1.0 I have tested with the [2D Euler Vortex](https://pyfr.readthedocs.io/en/latest/examples.html#d-euler-vortex) example with 2 GPUs, and things are fine. And single GPU is also OK with the same configuration file. So I do not figure out what I did wrong. Could you give me some suggestions, please? OS: ubuntu 20.04 PyFR version: 1.15.0 ---
**Author:** ![fdw](https://avatars.discourse-cdn.com/v4/letter/f/a587f6/32.png) [@fdw](https://pyfr.discourse.group/u/fdw)\ **Post date:** [7 November 2022 18:07 UTC](https://pyfr.discourse.group/t/cublasinternalerror-when-using-multi-gpu/714/2 "2022-11-07T18:07:26Z")
If PyFR works on both GPUs individually the first thing I would check is if you might be running out of memory on one of the GPUs. This not not raise an internal error, but it is something we want to rule out. Regards, Freddie. ---
**Author:** ![hwtang](https://avatars.discourse-cdn.com/v4/letter/h/a87d85/32.png) [@hwtang](https://pyfr.discourse.group/u/hwtang)\ **Post date:** [8 November 2022 01:46 UTC](https://pyfr.discourse.group/t/cublasinternalerror-when-using-multi-gpu/714/3 "2022-11-08T01:46:33Z")
Hi Freddie, Thank you for your reply. I checked the simulation on every single GPU and found the error was caused by GPU 0 (Tesla K40c). The simulation can run successfully on another GPU 1. The simulation takes 9093 MB memory on GPU 1. So neither of the GPUs is running out of memory. I also did a test using PyFR 1.12.3. Surprisingly, the simulation can run successfully using two GPUs. And it is OK using either GPU. The simulation is the sd7003 case obtained from [this paper](https://www.sciencedirect.com/science/article/pii/S0021999116307136). BTW, I have a novice question: if one single GPU has enough memory to do a simulation, can the simulation be accelerated with multi GPUs? The answer seems to be yes if I do not misunderstand [these discussions](https://www.quora.com/When-should-you-go-for-a-multi-GPU-program-vs-a-single-GPU-execution-for-training-running-Deep-Learning-models). And could you also recommend some GPU programming materials for beginners, please? Best regards. ---
**Author:** ![WillT](https://avatars.discourse-cdn.com/v4/letter/w/bc79bd/32.png) [@WillT](https://pyfr.discourse.group/u/WillT)\ **Post date:** [8 November 2022 09:21 UTC](https://pyfr.discourse.group/t/cublasinternalerror-when-using-multi-gpu/714/4 "2022-11-08T09:21:28Z")
I imagine waht the issue is is that the graph API stuff that has ben added is not support on K40 GPUs, which are now quite old. I know for example that the newer nvidia profilers don’t support these. In answer to your second question, generally stong scaling will always speed up your calculation, unless you strong scale it far too far. Each GPU needs enough to do to occupy the warps sufficiently, and for the overhead of launching kernels to not be the major time use, aswell as to fill the time when communication is happening. NVIDIA’s blogs are a good source of material as this paper [https://arxiv.org/abs/1804.06826](https://arxiv.org/abs/1804.06826), but as Volta is a couple generation old now it is less relivant. ---
**Author:** ![hwtang](https://avatars.discourse-cdn.com/v4/letter/h/a87d85/32.png) [@hwtang](https://pyfr.discourse.group/u/hwtang)\ **Post date:** [8 November 2022 09:30 UTC](https://pyfr.discourse.group/t/cublasinternalerror-when-using-multi-gpu/714/5 "2022-11-08T09:30:54Z")
OK, I get it. Thank you very much.