# RocBLASInvalidValue

**URL:** <https://pyfr.discourse.group/t/rocblasinvalidvalue/1170>\
**Category:** Errors\
**Created:** [16 September 2024 12:27 UTC](https://pyfr.discourse.group/t/rocblasinvalidvalue/1170 "2024-09-16T12:27:15Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Zhenyang](https://avatars.discourse-cdn.com/v4/letter/z/ec9cab/32.png) [@Zhenyang](https://pyfr.discourse.group/u/Zhenyang)\
**Post date:** [16 September 2024 12:27 UTC](https://pyfr.discourse.group/t/rocblasinvalidvalue/1170/1 "2024-09-16T12:27:15Z")

</div>

Hi,

I am trying to use pyfr2.0.3 with rocm6.0.3 to run the test case but with error:

```auto
Traceback (most recent call last):
  File "/pfs/lustrep2/projappl/project_465000921/zhenyang/ENV_rocm6/pyfr203_rocm6_lintrip/lib/python3.11/site-packages/pyfr/backends/hip/rocblas.py", line 121, in mul
    algo, dt = self._mul_cache[ckey]
               ~~~~~~~~~~~~~~~ ^^^^^^
KeyError: (<class 'numpy.float64'>, 1.0, 0.0, 768, 16, 16, 768, 64, 768)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/pfs/lustrep2/projappl/project_465000921/zhenyang/ENV_rocm6/pyfr203_rocm6_lintrip/bin/pyfr", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/pfs/lustrep2/projappl/project_465000921/zhenyang/ENV_rocm6/pyfr203_rocm6_lintrip/lib/python3.11/site-packages/pyfr/ __main__.py", line 124, in main
    args.process(args)
  File "/pfs/lustrep2/projappl/project_465000921/zhenyang/ENV_rocm6/pyfr203_rocm6_lintrip/lib/python3.11/site-packages/pyfr/ __main__.py", line 258, in process_run
    _process_common(
  File "/pfs/lustrep2/projappl/project_465000921/zhenyang/ENV_rocm6/pyfr203_rocm6_lintrip/lib/python3.11/site-packages/pyfr/ __main__.py", line 243, in _process_common
    solver = get_solver(backend, rallocs, mesh, soln, cfg)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/pfs/lustrep2/projappl/project_465000921/zhenyang/ENV_rocm6/pyfr203_rocm6_lintrip/lib/python3.11/site-packages/pyfr/solvers/ __init__.py", line 14, in get_solver
    return get_integrator(backend, systemcls, rallocs, mesh, initsoln, cfg)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/pfs/lustrep2/projappl/project_465000921/zhenyang/ENV_rocm6/pyfr203_rocm6_lintrip/lib/python3.11/site-packages/pyfr/integrators/ __init__.py", line 34, in get_integrator
    return integrator(backend, systemcls, rallocs, mesh, initsoln, cfg)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/pfs/lustrep2/projappl/project_465000921/zhenyang/ENV_rocm6/pyfr203_rocm6_lintrip/lib/python3.11/site-packages/pyfr/integrators/dual/phys/controllers.py", line 6, in __init__
    super(). __init__ (*args, **kwargs)
  File "/pfs/lustrep2/projappl/project_465000921/zhenyang/ENV_rocm6/pyfr203_rocm6_lintrip/lib/python3.11/site-packages/pyfr/integrators/dual/phys/steppers.py", line 14, in __init__
    super(). __init__ (*args, **kwargs)
  File "/pfs/lustrep2/projappl/project_465000921/zhenyang/ENV_rocm6/pyfr203_rocm6_lintrip/lib/python3.11/site-packages/pyfr/integrators/dual/phys/base.py", line 23, in __init__
    self.pseudointegrator.commit()
  File "/pfs/lustrep2/projappl/project_465000921/zhenyang/ENV_rocm6/pyfr203_rocm6_lintrip/lib/python3.11/site-packages/pyfr/integrators/dual/pseudo/multip.py", line 127, in commit
    s.system.commit()
  File "/pfs/lustrep2/projappl/project_465000921/zhenyang/ENV_rocm6/pyfr203_rocm6_lintrip/lib/python3.11/site-packages/pyfr/solvers/base/system.py", line 67, in commit
    self._gen_kernels(self.nregs, self.ele_map.values(), self._int_inters,
  File "/pfs/lustrep2/projappl/project_465000921/zhenyang/ENV_rocm6/pyfr203_rocm6_lintrip/lib/python3.11/site-packages/pyfr/solvers/base/system.py", line 202, in _gen_kernels
    kern = kgetter(i)
           ^^^^^^^^^^
  File "/pfs/lustrep2/projappl/project_465000921/zhenyang/ENV_rocm6/pyfr203_rocm6_lintrip/lib/python3.11/site-packages/pyfr/solvers/baseadvec/elements.py", line 74, in <lambda>
    kernels['disu'] = lambda uin: self._be.kernel(
                                  ^^^^^^^^^^^^^^^^
  File "/pfs/lustrep2/projappl/project_465000921/zhenyang/ENV_rocm6/pyfr203_rocm6_lintrip/lib/python3.11/site-packages/pyfr/backends/base/backend.py", line 196, in kernel
    kern = kern_meth(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/pfs/lustrep2/projappl/project_465000921/zhenyang/ENV_rocm6/pyfr203_rocm6_lintrip/lib/python3.11/site-packages/pyfr/backends/hip/rocblas.py", line 143, in mul
    dt = self._benchmark(gemm)
         ^^^^^^^^^^^^^^^^^^^^^
  File "/pfs/lustrep2/projappl/project_465000921/zhenyang/ENV_rocm6/pyfr203_rocm6_lintrip/lib/python3.11/site-packages/pyfr/backends/hip/provider.py", line 47, in _benchmark
    kfunc(stream)
  File "/pfs/lustrep2/projappl/project_465000921/zhenyang/ENV_rocm6/pyfr203_rocm6_lintrip/lib/python3.11/site-packages/pyfr/backends/hip/rocblas.py", line 114, in gemm
    w.rocblas_gemm_ex(
  File "/pfs/lustrep2/projappl/project_465000921/zhenyang/ENV_rocm6/pyfr203_rocm6_lintrip/lib/python3.11/site-packages/pyfr/ctypesutil.py", line 37, in _errcheck
    raise self._statuses[status]
pyfr.backends.hip.rocblas.RocBLASInvalidValue
srun: error: nid007964: task 0: Exited with exit code 1
srun: Terminating StepId=8009219.0

```

The computer system is just updated. I wonder if it is the problem with the cluster.

Regards,  
Zhenyang

---

<div class="post-metadata">

**Author:** ![fdw](https://avatars.discourse-cdn.com/v4/letter/f/a587f6/32.png) [@fdw](https://pyfr.discourse.group/u/fdw)\
**Post date:** [16 September 2024 12:57 UTC](https://pyfr.discourse.group/t/rocblasinvalidvalue/1170/2 "2024-09-16T12:57:45Z")

</div>

This is quite likely. Is there any chance you can swap out rocBLAS versions?

Regards, Freddie.

---

<div class="post-metadata">

**Author:** ![Zhenyang](https://avatars.discourse-cdn.com/v4/letter/z/ec9cab/32.png) [@Zhenyang](https://pyfr.discourse.group/u/Zhenyang)\
**Post date:** [16 September 2024 15:30 UTC](https://pyfr.discourse.group/t/rocblasinvalidvalue/1170/3 "2024-09-16T15:30:05Z")

</div>

Hi,

I double checked with the cluster admin, the version of rocblas is 4.0.0. Here I also quote some of his replies:

```auto
solution_index – [in] [int32_t] if algo is rocblas_gemm_algo_solution_index,
this controls which solution is used. When algo is not
rocblas_gemm_algo_solution_index, or if solution_index <= 0, the default
solution is used. This parameter was unused in previous releases and instead
always used the default solution

 

Now if I edit the code and change the value of the "algo" parameter, i.e.,
changing this piece of code:

w.rocblas_gemm_ex(
> h, opA, opB, m, n, k, byref(alpha_ct), A, rtype, A.leaddim, B,
> rtype, B.leaddim, byref(beta_ct), C, rtype, C.leaddim, C,
> rtype, C.leaddim, rtype, w.GEMM_ALGO_SOLUTION_INDEX, algo, 0
)

to this

w.rocblas_gemm_ex(
> h, opA, opB, m, n, k, byref(alpha_ct), A, rtype, A.leaddim, B,
> rtype, B.leaddim, byref(beta_ct), C, rtype, C.leaddim, C,
> rtype, C.leaddim, rtype, w.GEMM_ALGO_SOLUTION_INDEX, 0, 0
)

PyFR runs.

```

Could you help checking this?

Regards,  
Zhenyang

---

<div class="post-metadata">

**Author:** ![fdw](https://avatars.discourse-cdn.com/v4/letter/f/a587f6/32.png) [@fdw](https://pyfr.discourse.group/u/fdw)\
**Post date:** [16 September 2024 19:46 UTC](https://pyfr.discourse.group/t/rocblasinvalidvalue/1170/4 "2024-09-16T19:46:05Z")

</div>

This change will effectively break the auto-tuning aspect of our rocBLAS integration (ask for a bunch of GEMM algorithms and then use the best one) by forcing us to always use the default algorithm which often performs poorly.

The error is likely in rocBLAS (after all it is telling us what the valid values of `algo` should be and then complaining when we try one of them).

Regards, Freddie.

---

<div class="post-metadata">

**Author:** ![arun-s-p](https://yyz2.discourse-cdn.com/free1/user_avatar/pyfr.discourse.group/arun-s-p/32/733_2.png) [@arun-s-p](https://pyfr.discourse.group/u/arun-s-p)\
**Post date:** [15 September 2025 13:54 UTC](https://pyfr.discourse.group/t/rocblasinvalidvalue/1170/5 "2025-09-15T13:54:54Z")

</div>

@Zhenyang Is this on LUMI? I encounter the same issue now. Is there a way around this without modifyng the source?

---

<div class="post-metadata">

**Author:** ![Zhenyang](https://avatars.discourse-cdn.com/v4/letter/z/ec9cab/32.png) [@Zhenyang](https://pyfr.discourse.group/u/Zhenyang)\
**Post date:** [15 September 2025 18:59 UTC](https://pyfr.discourse.group/t/rocblasinvalidvalue/1170/6 "2025-09-15T18:59:04Z")

</div>

The simplest work around would be:

change [PyFR/pyfr/backends/hip/rocblas.py at 7cdd08dd51b3e5cc21e1c4e97179be238912f4f1 · PyFR/PyFR · GitHub](https://github.com/PyFR/PyFR/blob/7cdd08dd51b3e5cc21e1c4e97179be238912f4f1/pyfr/backends/hip/rocblas.py) as following:

```auto
             # Benchmark suggested algorithms
             for algo in sidx:
- dt = self._benchmark(gemm)
- if best_kern is None or dt < best_kern[-1]:
- best_kern = algo, dt
+ try:
+ dt = self._benchmark(gemm)
+ if best_kern is None or dt < best_kern[-1]:
+ best_kern = algo, dt
+ except RocBLASInvalidValue:
+ pass
 
             # Restore the output matrix
             getattr(out, 'parent', out).set(out_np)

```

and it works for me.

@fdw maybe it is worthwhile to add exceptions in tuning the rocBLAS since rocBLAS can return invalid values. In one example of rocm, it also calls twice and see if the solution is valid, see [https://github.com/ROCm/rocBLAS/blob/dda25a081f187fcef9228e66cce123061805e69a/clients/samples/example\_user\_driven\_tuning.cpp#L93](https://github.com/ROCm/rocBLAS/blob/dda25a081f187fcef9228e66cce123061805e69a/clients/samples/example_user_driven_tuning.cpp#L93) . I can make a pull request if you think this should be fixed.

Regards,

Zhenyang

---

<div class="post-metadata">

**Author:** ![fdw](https://avatars.discourse-cdn.com/v4/letter/f/a587f6/32.png) [@fdw](https://pyfr.discourse.group/u/fdw)\
**Post date:** [15 September 2025 21:33 UTC](https://pyfr.discourse.group/t/rocblasinvalidvalue/1170/7 "2025-09-15T21:33:49Z")

</div>

Can we reproduce the invalid values with rocBLAS from ROCm 6.4? I ask as we’re probably going to merge:

> <https://github.com/PyFR/PyFR/pull/312>
>
> This PR adds full support for HIP graphs, bringing the HIP backend on par with t…he CUDA backend. However, whereas for simple test cases overhead decreased by a factor of ~2 when moving to graphs, for HIP overhead increases by ~2. The performance impact for non-trivial cases still needs to be assessed, however.

soon which will increase the minimum supported version to 6.4.x.

If this is something which is being observed with current releases then we should certainly look to make the code robust against it.

Regards, Freddie.

---

<div class="post-metadata">

**Author:** ![Zhenyang](https://avatars.discourse-cdn.com/v4/letter/z/ec9cab/32.png) [@Zhenyang](https://pyfr.discourse.group/u/Zhenyang)\
**Post date:** [16 September 2025 19:02 UTC](https://pyfr.discourse.group/t/rocblasinvalidvalue/1170/8 "2025-09-16T19:02:31Z")

</div>

On the machines I used, the maximum rocm version is 6.3.3. Not sure if it will appear in the further versions.

For the current release, this fix can bypass the invalid solution issue in the auto tuning according to my test.

Regards,

Zhenyang

---

<div class="post-metadata">

**Author:** ![fdw](https://avatars.discourse-cdn.com/v4/letter/f/a587f6/32.png) [@fdw](https://pyfr.discourse.group/u/fdw)\
**Post date:** [16 September 2025 20:04 UTC](https://pyfr.discourse.group/t/rocblasinvalidvalue/1170/9 "2025-09-16T20:04:26Z")

</div>

That’s a bit too new for comfort. Please go ahead and submit a PR.

Regards, Freddie.