# Running PyFR with OpenMP **URL:** \ **Category:** General\ **Created:** [21 October 2024 20:31 UTC](https://pyfr.discourse.group/t/running-pyfr-with-openmp/1180 "2024-10-21T20:31:46Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![pyfr\_starting\_man](https://avatars.discourse-cdn.com/v4/letter/p/d26b3c/32.png) [@pyfr\_starting\_man](https://pyfr.discourse.group/u/pyfr_starting_man)\ **Post date:** [21 October 2024 20:31 UTC](https://pyfr.discourse.group/t/running-pyfr-with-openmp/1180/1 "2024-10-21T20:31:46Z")
## I have been trying all day long to get PyFR to run with OpenMP. I have no issues running my case with cuda on GPUs, but I am trying to compare to CPU performance on the same nodes. I am on an HPC cluster and have installed GCC 12.2 and libxsmm according to the documentation. I am getting a similar error to the thread [KeyError on 3D case running with WSL Ubuntu openMP - #12 by Zhenyang](https://pyfr.discourse.group/t/keyerror-on-3d-case-running-with-wsl-ubuntu-openmp/404/12) which is seemingly unresolved. Here is my traceback when running pyfr:(pyfr-venv) [user@p0827 user]$ mpiexec -n 20 pyfr -p run -b openmp part20\_refined/3D\_isolator\_refined.pyfrm /users/PAA0023/user/Downloads/3D\_longer\_supersonic/3D\_case\_CPU.ini Traceback (most recent call last): File “/users/PAA0023/user/Downloads/Nasa\_Test/pyfr-venv/bin/pyfr”, line 8, in sys.exit(main()) File “/users/PAA0023/user/Downloads/Nasa\_Test/pyfr-venv/lib/python3.10/site-packages/pyfr/ **main**.py”, line 134, in main args.process(args) File “/users/PAA0023/user/Downloads/Nasa\_Test/pyfr-venv/lib/python3.10/site-packages/pyfr/ **main**.py”, line 296, in process\_run \_process\_common( File “/users/PAA0023/user/Downloads/Nasa\_Test/pyfr-venv/lib/python3.10/site-packages/pyfr/ **main**.py”, line 272, in \_process\_common solver = get\_solver(backend, rallocs, mesh, soln, cfg) File “/users/PAA0023/user/Downloads/Nasa\_Test/pyfr-venv/lib/python3.10/site-packages/pyfr/solvers/ **init**.py”, line 14, in get\_solver return get\_integrator(backend, systemcls, rallocs, mesh, initsoln, cfg) File “/users/PAA0023/user/Downloads/Nasa\_Test/pyfr-venv/lib/python3.10/site-packages/pyfr/integrators/ **init**.py”, line 34, in get\_integrator return integrator(backend, systemcls, rallocs, mesh, initsoln, cfg) File “/users/PAA0023/user/Downloads/Nasa\_Test/pyfr-venv/lib/python3.10/site-packages/pyfr/integrators/std/controllers.py”, line 11, in **init** super(). **init** (\*args, \*\*kwargs) File “/users/PAA0023/user/Downloads/Nasa\_Test/pyfr-venv/lib/python3.10/site-packages/pyfr/integrators/std/steppers.py”, line 131, in **init** super(). **init** (\*args, \*\*kwargs) File “/users/PAA0023/user/Downloads/Nasa\_Test/pyfr-venv/lib/python3.10/site-packages/pyfr/integrators/std/base.py”, line 26, in **init** self.system.commit() File “/users/PAA0023/user/Downloads/Nasa\_Test/pyfr-venv/lib/python3.10/site-packages/pyfr/solvers/base/system.py”, line 67, in commit self.\_gen\_kernels(self.nregs, self.ele\_map.values(), self.\_int\_inters, File “/users/PAA0023/user/Downloads/Nasa\_Test/pyfr-venv/lib/python3.10/site-packages/pyfr/solvers/base/system.py”, line 202, in \_gen\_kernels kern = kgetter(i) File “/users/PAA0023/user/Downloads/Nasa\_Test/pyfr-venv/lib/python3.10/site-packages/pyfr/solvers/baseadvec/elements.py”, line 121, in kernels[‘evalsrcmacros’] = lambda uin: self.\_be.kernel( File “/users/PAA0023/user/Downloads/Nasa\_Test/pyfr-venv/lib/python3.10/site-packages/pyfr/backends/base/backend.py”, line 196, in kernel kern = kern\_meth(\*args, \*\*kwargs) File “/users/PAA0023/user/Downloads/Nasa\_Test/pyfr-venv/lib/python3.10/site-packages/pyfr/backends/base/kernels.py”, line 177, in kernel\_meth fun = self.\_build\_kernel(name, src, list(it.chain(\*argt)), argn) File “/users/PAA0023/user/Downloads/Nasa\_Test/pyfr-venv/lib/python3.10/site-packages/pyfr/backends/openmp/provider.py”, line 150, in \_build\_kernel fun = lib.function(name) File “/users/PAA0023/user/Downloads/Nasa\_Test/pyfr-venv/lib/python3.10/site-packages/pyfr/backends/openmp/compiler.py”, line 139, in function fn = getattr(self.mod, name) File “/users/PAA0023/user/python3.10/lib/python3.10/ctypes/ **init**.py”, line 387, in **getattr** func = self. **getitem** (name) File “/users/PAA0023/user/python3.10/lib/python3.10/ctypes/ **init**.py”, line 392, in **getitem** func = self.\_FuncPtr((name\_or\_ordinal, self)) AttributeError: /users/PAA0023/user/.cache/pyfr/libb9bb435fceb8a5ea365b87102b69b3e2daf5ba1d1a7fa9b9f1a36185ffde0a26.so: undefined symbol: evalsrcmacros MPI\_ABORT was invoked on rank 10 in communicator MPI\_COMM\_WORLD with errorcode 1. ## NOTE: invoking MPI\_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. slurmstepd: error: \*\*\* STEP 32206021.9 ON p0827 CANCELLED AT 2024-10-21T16:24:23 \*\*\* srun: Job step aborted: Waiting up to 32 seconds for job step to finish. srun: error: p0835: tasks 16-17: Killed srun: error: p0832: tasks 10-11: Killed srun: error: p0833: tasks 12-13: Killed srun: error: p0834: tasks 14-15: Killed srun: error: p0829: tasks 4-5: Killed srun: error: p0831: tasks 8-9: Killed srun: error: p0827: tasks 0-1: Killed srun: error: p0836: tasks 18-19: Killed srun: error: p0828: tasks 2-3: Killed srun: error: p0830: tasks 6-7: Killed (pyfr-venv) [user@p0827 user]$ Similar to the the thread I linked, whenever I clear my cache, I get a new AttributeError every time. Any help is greatly appreciated. ---
**Author:** ![p.vincent](https://yyz2.discourse-cdn.com/free1/user_avatar/pyfr.discourse.group/p.vincent/32/225_2.png) [@p.vincent](https://pyfr.discourse.group/u/p.vincent)\ **Post date:** [22 October 2024 12:23 UTC](https://pyfr.discourse.group/t/running-pyfr-with-openmp/1180/2 "2024-10-22T12:23:53Z")
Can you share your .ini file? ---
**Author:** ![p.vincent](https://yyz2.discourse-cdn.com/free1/user_avatar/pyfr.discourse.group/p.vincent/32/225_2.png) [@p.vincent](https://pyfr.discourse.group/u/p.vincent)\ **Post date:** [22 October 2024 12:39 UTC](https://pyfr.discourse.group/t/running-pyfr-with-openmp/1180/3 "2024-10-22T12:39:27Z")
And could you confirm what version of PyFR you are running. Thanks. ---
**Author:** ![fdw](https://avatars.discourse-cdn.com/v4/letter/f/a587f6/32.png) [@fdw](https://pyfr.discourse.group/u/fdw)\ **Post date:** [22 October 2024 12:55 UTC](https://pyfr.discourse.group/t/running-pyfr-with-openmp/1180/4 "2024-10-22T12:55:43Z")
What happens if you disable the cache entirely? You can do this by exporting the environment variable `PYFR_DEBUG_OMP_DISABLE_CACHE=1`. Regards, Freddie. ---
**Author:** ![pyfr\_starting\_man](https://avatars.discourse-cdn.com/v4/letter/p/d26b3c/32.png) [@pyfr\_starting\_man](https://pyfr.discourse.group/u/pyfr_starting_man)\ **Post date:** [22 October 2024 18:12 UTC](https://pyfr.discourse.group/t/running-pyfr-with-openmp/1180/6 "2024-10-22T18:12:18Z")
This seems to change my error to an OSError. I have reinstalled libxsmm from scratch to see if that is the issue. Maybe it could be because I have GCC installed locally (since my HPC cluster did not have version \>= 12.0 which OpenMP requires)? (pyfr-venv) [user@p0306 ~]$ mpiexec -n 20 pyfr -p run -b openmp part20\_refined/3D\_isolator\_refined.pyfrm /users/PAA0023/user/Downloads/3D\_longer\_supersonic/3D\_case\_CPU.ini mpiexec: error while loading shared libraries: libimf.so: cannot open shared object file: No such file or directory ---
**Author:** ![pyfr\_starting\_man](https://avatars.discourse-cdn.com/v4/letter/p/d26b3c/32.png) [@pyfr\_starting\_man](https://pyfr.discourse.group/u/pyfr_starting_man)\ **Post date:** [22 October 2024 18:13 UTC](https://pyfr.discourse.group/t/running-pyfr-with-openmp/1180/7 "2024-10-22T18:13:12Z")
Here is my .ini file: [backend] precision = double [backend-openmp] cc = gcc gimmik-max-nnz = 8192 [constants] M = 2 gamma = 1.4 mu = 2.3256e-5 Pr = 0.71 cpTref = 0.6249985 cpTs = 0.3973979 ac-zeta = 3 [solver] system = navier-stokes order = 3 shock-capturing = entropy-filter [solver-entropy-filter] d-min = 1e-6 p-min = 1e-6 e-tol = 1e-6 e-func = physical niters = 2 [solver-time-integrator] formulation = std scheme = rk45 controller = none tstart = 0 tend = 20 dt = 5e-5 [solver-interfaces] riemann-solver = rusanov ldg-beta = 0.5 ldg-tau = 0.1 [solver-interfaces-quad] flux-pts = gauss-legendre-lobatto quad-pts = gauss-legendre-lobatto [solver-elements-quad] soln-pts = gauss-legendre-lobatto quad-pts = gauss-legendre-lobatto [solver-interfaces-line] flux-pts = gauss-legendre-lobatto quad-pts = gauss-legendre-lobatto [solver-elements-hex] flux-pts = gauss-legendre-lobatto soln-pts = gauss-legendre-lobatto [soln-plugin-writer] dt-out = 0.05 basedir = . basename = 20\_GPUs\_case\_smaller\_dt-{t:.2f} [soln-plugin-dtstats] flushsteps = 1000 file = dtstats3.csv header = true [soln-bcs-inlet] type = sup-in-fa rho = 1 u = 1 v = 0.0 w = 0.0 p = 0.028571 [soln-bcs-outlet] type = sub-out-fp p = 0.828571 [soln-bcs-wall] type = no-slp-isot-wall u = 0.0 v = 0.0 w = 0.0 cpTw = 1.074997 [soln-ics] rho = 1 u = 1 v = 0.0 w = 0.0 p = 0.178571 [soln-plugin-nancheck] nsteps = 20 ---
**Author:** ![pyfr\_starting\_man](https://avatars.discourse-cdn.com/v4/letter/p/d26b3c/32.png) [@pyfr\_starting\_man](https://pyfr.discourse.group/u/pyfr_starting_man)\ **Post date:** [22 October 2024 18:13 UTC](https://pyfr.discourse.group/t/running-pyfr-with-openmp/1180/8 "2024-10-22T18:13:46Z")
I am using PyFR version 2.0.3 ---
**Author:** ![fdw](https://avatars.discourse-cdn.com/v4/letter/f/a587f6/32.png) [@fdw](https://pyfr.discourse.group/u/fdw)\ **Post date:** [22 October 2024 18:22 UTC](https://pyfr.discourse.group/t/running-pyfr-with-openmp/1180/9 "2024-10-22T18:22:36Z")
This is an MPI error and is entirely unrelated to what you’ve been seeing previously. The library `libimf.so` is part of the Intel Compiler which suggests some parts of your stack have been compiled with ICC. Either way it is an environment problem: either make sure `libimf.so` is available or recompile the relevant dependencies with GCC. Regards, Freddie. ---
**Author:** ![pyfr\_starting\_man](https://avatars.discourse-cdn.com/v4/letter/p/d26b3c/32.png) [@pyfr\_starting\_man](https://pyfr.discourse.group/u/pyfr_starting_man)\ **Post date:** [22 October 2024 18:37 UTC](https://pyfr.discourse.group/t/running-pyfr-with-openmp/1180/10 "2024-10-22T18:37:57Z")
Thanks for the help. Do I have to use GCC for openmp backend? It seems everything on my HPC is compiled with ICC. So if I were to go the GCC route, I will have to recompile almost everything, which will likely cause some messy problems with dependencies. ---
**Author:** ![fdw](https://avatars.discourse-cdn.com/v4/letter/f/a587f6/32.png) [@fdw](https://pyfr.discourse.group/u/fdw)\ **Post date:** [22 October 2024 18:43 UTC](https://pyfr.discourse.group/t/running-pyfr-with-openmp/1180/11 "2024-10-22T18:43:17Z")
The OpenMP backend requires GCC (some Clang versions might work too assuming they have OpenMP support; ICC is untested). This does not necessarily mean you need to recompile anything; for example on my Mac everything is compiled with Clang and GCC is only used by PyFR for its kernels and this is fine. Regards, Freddie. ---
**Author:** ![pyfr\_starting\_man](https://avatars.discourse-cdn.com/v4/letter/p/d26b3c/32.png) [@pyfr\_starting\_man](https://pyfr.discourse.group/u/pyfr_starting_man)\ **Post date:** [22 October 2024 18:45 UTC](https://pyfr.discourse.group/t/running-pyfr-with-openmp/1180/12 "2024-10-22T18:45:21Z")
How does PyFR know to use GCC? Is this just simply some environmental variable that you set? ---
**Author:** ![fdw](https://avatars.discourse-cdn.com/v4/letter/f/a587f6/32.png) [@fdw](https://pyfr.discourse.group/u/fdw)\ **Post date:** [22 October 2024 18:50 UTC](https://pyfr.discourse.group/t/running-pyfr-with-openmp/1180/13 "2024-10-22T18:50:41Z")
It does through: ```auto [backend-openmp] cc = gcc ``` where you need to specify a working version of GCC as the option value. Regards, Freddie.