Running PyFR with OpenMP

General
1

I have been trying all day long to get PyFR to run with OpenMP. I have no issues running my case with cuda on GPUs, but I am trying to compare to CPU performance on the same nodes. I am on an HPC cluster and have installed GCC 12.2 and libxsmm according to the documentation. I am getting a similar error to the thread KeyError on 3D case running with WSL Ubuntu openMP - #12 by Zhenyang which is seemingly unresolved. Here is my traceback when running pyfr:(pyfr-venv) [user@p0827 user]$ mpiexec -n 20 pyfr -p run -b openmp part20_refined/3D_isolator_refined.pyfrm /users/PAA0023/user/Downloads/3D_longer_supersonic/3D_case_CPU.ini
Traceback (most recent call last):
File “/users/PAA0023/user/Downloads/Nasa_Test/pyfr-venv/bin/pyfr”, line 8, in
sys.exit(main())
File “/users/PAA0023/user/Downloads/Nasa_Test/pyfr-venv/lib/python3.10/site-packages/pyfr/main.py”, line 134, in main
args.process(args)
File “/users/PAA0023/user/Downloads/Nasa_Test/pyfr-venv/lib/python3.10/site-packages/pyfr/main.py”, line 296, in process_run
_process_common(
File “/users/PAA0023/user/Downloads/Nasa_Test/pyfr-venv/lib/python3.10/site-packages/pyfr/main.py”, line 272, in _process_common
solver = get_solver(backend, rallocs, mesh, soln, cfg)
File “/users/PAA0023/user/Downloads/Nasa_Test/pyfr-venv/lib/python3.10/site-packages/pyfr/solvers/init.py”, line 14, in get_solver
return get_integrator(backend, systemcls, rallocs, mesh, initsoln, cfg)
File “/users/PAA0023/user/Downloads/Nasa_Test/pyfr-venv/lib/python3.10/site-packages/pyfr/integrators/init.py”, line 34, in get_integrator
return integrator(backend, systemcls, rallocs, mesh, initsoln, cfg)
File “/users/PAA0023/user/Downloads/Nasa_Test/pyfr-venv/lib/python3.10/site-packages/pyfr/integrators/std/controllers.py”, line 11, in init
super().init(*args, **kwargs)
File “/users/PAA0023/user/Downloads/Nasa_Test/pyfr-venv/lib/python3.10/site-packages/pyfr/integrators/std/steppers.py”, line 131, in init
super().init(*args, **kwargs)
File “/users/PAA0023/user/Downloads/Nasa_Test/pyfr-venv/lib/python3.10/site-packages/pyfr/integrators/std/base.py”, line 26, in init
self.system.commit()
File “/users/PAA0023/user/Downloads/Nasa_Test/pyfr-venv/lib/python3.10/site-packages/pyfr/solvers/base/system.py”, line 67, in commit
self._gen_kernels(self.nregs, self.ele_map.values(), self._int_inters,
File “/users/PAA0023/user/Downloads/Nasa_Test/pyfr-venv/lib/python3.10/site-packages/pyfr/solvers/base/system.py”, line 202, in _gen_kernels
kern = kgetter(i)
File “/users/PAA0023/user/Downloads/Nasa_Test/pyfr-venv/lib/python3.10/site-packages/pyfr/solvers/baseadvec/elements.py”, line 121, in
kernels[‘evalsrcmacros’] = lambda uin: self._be.kernel(
File “/users/PAA0023/user/Downloads/Nasa_Test/pyfr-venv/lib/python3.10/site-packages/pyfr/backends/base/backend.py”, line 196, in kernel
kern = kern_meth(*args, **kwargs)
File “/users/PAA0023/user/Downloads/Nasa_Test/pyfr-venv/lib/python3.10/site-packages/pyfr/backends/base/kernels.py”, line 177, in kernel_meth
fun = self._build_kernel(name, src, list(it.chain(*argt)), argn)
File “/users/PAA0023/user/Downloads/Nasa_Test/pyfr-venv/lib/python3.10/site-packages/pyfr/backends/openmp/provider.py”, line 150, in _build_kernel
fun = lib.function(name)
File “/users/PAA0023/user/Downloads/Nasa_Test/pyfr-venv/lib/python3.10/site-packages/pyfr/backends/openmp/compiler.py”, line 139, in function
fn = getattr(self.mod, name)
File “/users/PAA0023/user/python3.10/lib/python3.10/ctypes/init.py”, line 387, in getattr
func = self.getitem(name)
File “/users/PAA0023/user/python3.10/lib/python3.10/ctypes/init.py”, line 392, in getitem
func = self._FuncPtr((name_or_ordinal, self))
AttributeError: /users/PAA0023/user/.cache/pyfr/libb9bb435fceb8a5ea365b87102b69b3e2daf5ba1d1a7fa9b9f1a36185ffde0a26.so: undefined symbol: evalsrcmacros

MPI_ABORT was invoked on rank 10 in communicator MPI_COMM_WORLD
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.

slurmstepd: error: *** STEP 32206021.9 ON p0827 CANCELLED AT 2024-10-21T16:24:23 ***
srun: Job step aborted: Waiting up to 32 seconds for job step to finish.
srun: error: p0835: tasks 16-17: Killed
srun: error: p0832: tasks 10-11: Killed
srun: error: p0833: tasks 12-13: Killed
srun: error: p0834: tasks 14-15: Killed
srun: error: p0829: tasks 4-5: Killed
srun: error: p0831: tasks 8-9: Killed
srun: error: p0827: tasks 0-1: Killed
srun: error: p0836: tasks 18-19: Killed
srun: error: p0828: tasks 2-3: Killed
srun: error: p0830: tasks 6-7: Killed
(pyfr-venv) [user@p0827 user]$

Similar to the the thread I linked, whenever I clear my cache, I get a new AttributeError every time. Any help is greatly appreciated.

2

Can you share your .ini file?

3

And could you confirm what version of PyFR you are running. Thanks.

4

What happens if you disable the cache entirely? You can do this by exporting the environment variable PYFR_DEBUG_OMP_DISABLE_CACHE=1.

Regards, Freddie.

6

This seems to change my error to an OSError. I have reinstalled libxsmm from scratch to see if that is the issue. Maybe it could be because I have GCC installed locally (since my HPC cluster did not have version >= 12.0 which OpenMP requires)?

(pyfr-venv) [user@p0306 ~]$ mpiexec -n 20 pyfr -p run -b openmp part20_refined/3D_isolator_refined.pyfrm /users/PAA0023/user/Downloads/3D_longer_supersonic/3D_case_CPU.ini
mpiexec: error while loading shared libraries: libimf.so: cannot open shared object file: No such file or directory

7

Here is my .ini file:
[backend]
precision = double

[backend-openmp]
cc = gcc
gimmik-max-nnz = 8192

[constants]
M = 2
gamma = 1.4
mu = 2.3256e-5
Pr = 0.71
cpTref = 0.6249985
cpTs = 0.3973979
ac-zeta = 3

[solver]
system = navier-stokes
order = 3
shock-capturing = entropy-filter

[solver-entropy-filter]
d-min = 1e-6
p-min = 1e-6
e-tol = 1e-6
e-func = physical
niters = 2

[solver-time-integrator]
formulation = std
scheme = rk45
controller = none
tstart = 0
tend = 20
dt = 5e-5

[solver-interfaces]
riemann-solver = rusanov
ldg-beta = 0.5
ldg-tau = 0.1

[solver-interfaces-quad]
flux-pts = gauss-legendre-lobatto
quad-pts = gauss-legendre-lobatto

[solver-elements-quad]
soln-pts = gauss-legendre-lobatto
quad-pts = gauss-legendre-lobatto

[solver-interfaces-line]
flux-pts = gauss-legendre-lobatto
quad-pts = gauss-legendre-lobatto

[solver-elements-hex]
flux-pts = gauss-legendre-lobatto
soln-pts = gauss-legendre-lobatto

[soln-plugin-writer]
dt-out = 0.05
basedir = .
basename = 20_GPUs_case_smaller_dt-{t:.2f}

[soln-plugin-dtstats]
flushsteps = 1000
file = dtstats3.csv
header = true

[soln-bcs-inlet]
type = sup-in-fa
rho = 1
u = 1
v = 0.0
w = 0.0
p = 0.028571

[soln-bcs-outlet]
type = sub-out-fp
p = 0.828571

[soln-bcs-wall]
type = no-slp-isot-wall
u = 0.0
v = 0.0
w = 0.0
cpTw = 1.074997

[soln-ics]
rho = 1
u = 1
v = 0.0
w = 0.0
p = 0.178571

[soln-plugin-nancheck]
nsteps = 20

8

I am using PyFR version 2.0.3

9

This is an MPI error and is entirely unrelated to what you’ve been seeing previously. The library libimf.so is part of the Intel Compiler which suggests some parts of your stack have been compiled with ICC. Either way it is an environment problem: either make sure libimf.so is available or recompile the relevant dependencies with GCC.

Regards, Freddie.

10

Thanks for the help. Do I have to use GCC for openmp backend? It seems everything on my HPC is compiled with ICC. So if I were to go the GCC route, I will have to recompile almost everything, which will likely cause some messy problems with dependencies.

11

The OpenMP backend requires GCC (some Clang versions might work too assuming they have OpenMP support; ICC is untested). This does not necessarily mean you need to recompile anything; for example on my Mac everything is compiled with Clang and GCC is only used by PyFR for its kernels and this is fine.

Regards, Freddie.

12

How does PyFR know to use GCC? Is this just simply some environmental variable that you set?

13

It does through:

[backend-openmp]
cc = gcc

where you need to specify a working version of GCC as the option value.

Regards, Freddie.