Install pyfr on HPC with AMD gpu

Just Starting
1

Hello everyone,
I am new to PyFR, and tried to intall it on HPC (hlrs hunter: HPE Hunter Hardware and Architecture - HLRS Platforms). The HPC uses AMD mi300a processor.

I followed the instruction on the Installation page, and the package is properly installed with necessary dependencies. However, when I try to run the tutorial cases with hip backend, I get the following error:

(pyfr-venv) iag44026 x1000c0s0b0n0 268$ pyfr -p run -b hip viscous-shock-tube.pyfrm viscous-shock-tube.ini
MPICH ERROR [Rank 0] [job id ] [Mon Mar 17 16:57:23 2025] [x1000c0s0b0n0] - Abort(-1) (rank 0 in comm 0): MPIDI_CRAY_init: GPU_SUPPORT_ENABLED is requested, but GTL library is not linked
 (Other MPI error)

aborting job:
MPIDI_CRAY_init: GPU_SUPPORT_ENABLED is requested, but GTL library is not linked

if I turn the gpu support of by: export MPICH_GPU_SUPPORT_ENABLED=0 I get the following error:

(pyfr-venv) iag44026 x1000c0s0b0n0 270$ pyfr -p run -b hip viscous-shock-tube.pyfrm viscous-shock-tube.ini
Traceback (most recent call last):
  File "/zhome/academic/HLRS/iag/iagyonwu/pyfr-venv/bin/pyfr", line 33, in <module>
    sys.exit(load_entry_point('pyfr', 'console_scripts', 'pyfr')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/zhome/academic/HLRS/iag/iagyonwu/PyFR/pyfr/__main__.py", line 242, in main
    args.process(args)
  File "/zhome/academic/HLRS/iag/iagyonwu/PyFR/pyfr/__main__.py", line 548, in process_run
    _process_common(args, None, Inifile.load(args.cfg))
  File "/zhome/academic/HLRS/iag/iagyonwu/PyFR/pyfr/__main__.py", line 518, in _process_common
    reader = NativeReader(args.mesh, pname=args.pname)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/zhome/academic/HLRS/iag/iagyonwu/PyFR/pyfr/readers/native.py", line 44, in __init__
    self._read_metadata()
  File "/zhome/academic/HLRS/iag/iagyonwu/PyFR/pyfr/readers/native.py", line 152, in _read_metadata
    creator = self.f['creator'][()].decode()
              ~~~~~~^^^^^^^^^^^
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/zhome/academic/HLRS/iag/iagyonwu/pyfr-venv/lib/python3.11/site-packages/h5py/_hl/group.py", line 360, in __getitem__
    oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5o.pyx", line 257, in h5py.h5o.open
KeyError: "Unable to synchronously open object (object 'creator' doesn't exist)"

Here is list of modules that I loaded:

(pyfr-venv) iag44026 x1000c0s0b0n0 271$ module list

Currently Loaded Modules:
  1) craype-x86-genoa     4) cray-pals/1.3.2      7) xpmem/2.3                10) rocm/6.2.2                 13) amd/6.2.2            16) PrgEnv-amd/8.5.0    19) metis/5.1.0-int32
  2) libfabric/1.20.1     5) system/site_names    8) perftools-base/24.07.0   11) HLRS/APU/testing-20250305  14) cray-dsmml/0.3.0     17) cray-python/3.11.7  20) cray-mpich/8.1.30
  3) craype-network-ofi   6) system/ws/20250205   9) craype-accel-amd-gfx942  12) system/wrappers/1.0        15) cray-libsci/24.07.0  18) craype/2.7.32

Anyone knows how should I solve the GTL library problem?

2

Can you please run pyfr import viscous-shock-tube.msh viscous-shock-tube.pyfrm again and then post the output of h5ls viscous-shock-tube.pyfrm.

The GTL error is due to long standing issues with Cray MPI. It is strongly recommended to switch to straight MPICH which is better tested and more robust (the Cray ‘value add’ is dubious at best).

Regards, Freddie.

3

Dear Freddie,
Sorry the h5ls is not installed on our HPC system. I solved the problem by the following

export LD_PRELOAD=/opt/cray/pe/mpich/8.1.30/gtl/lib/libmpi_gtl_hsa.so

and then the tutorial case runs with the hip backend.

Best regards,
Yongxiang