Installing pyfr develop branch. Error: Address not mapped

Errors
1

Hello, I installed the pyfr develop branch, but the errors appear. How could I solve this problem?

 *** Process received signal ***
 Signal: Segmentation fault (11)
Signal code: Address not mapped (1)
 Failing at address: 0x555d00000000
[ 0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420)[0x7f087c454420]
[ 1] /home/jian/libxsmm/lib/libxsmm.so(libxsmm_fsspmdm_create+0x393)[0x7f087641ff63]
[ 2] /lib/x86_64-linux-gnu/libffi.so.7(+0x6ff5)[0x7f08524c2ff5]
[ 3] /lib/x86_64-linux-gnu/libffi.so.7(+0x640a)[0x7f08524c240a]
[ 4] /opt/python3.9/lib/python3.9/lib-dynload/_ctypes.cpython-39-x86_64-linux-gnu.so(+0x12bd1)[0x7f08524dabd1]
 [ 5] /opt/python3.9/lib/python3.9/lib-dynload/_ctypes.cpython-39-x86_64-linux-gnu.so(+0xb8b1)[0x7f08524d38b1]
[ 6] /home/pyfr-develop/bin/python(_PyObject_MakeTpCall+0x8c)[0x555d224bde4c]
[ 7] /home/pyfr-develop/bin/python(_PyEval_EvalFrameDefault+0x7fb8)[0x555d224aecf8]
 [ 8] /home/pyfr-develop/bin/python(+0x12784a)[0x555d2257184a]
[ 9] /home/pyfr-develop/bin/python(_PyFunction_Vectorcall+0x97)[0x555d224bee97]
 [10] /home/pyfr-develop/bin/python(+0x210f28)[0x555d2265af28]
 [12] /home/pyfr-develop/bin/python(_PyEval_EvalFrameDefault+0x20b4)[0x555d224a8df4]
 [13] /home/pyfr-develop/bin/python(+0x12784a)[0x555d2257184a]
[14] /home/pyfr-develop/bin/python(_PyFunction_Vectorcall+0x97)[0x555d224bee97]
[15] /home/pyfr-develop/bin/python(+0x210f28)[0x555d2265af28]
[16] /home/pyfr-develop/bin/python(_PyEval_EvalFrameDefault+0x603a)[0x555d224acd7a]
 [17] /home/pyfr-develop/bin/python(+0x12784a)[0x555d2257184a]
 [18] /home/pyfr-develop/bin/python(_PyFunction_Vectorcall+0x97)[0x555d224bee97]
 [19] /home/pyfr-develop/bin/python(_PyEval_EvalFrameDefault+0x60b9)[0x555d224acdf9]
 [20] /home/pyfr-develop/bin/python(+0x12784a)[0x555d2257184a]
[21] /home/pyfr-develop/bin/python(_PyFunction_Vectorcall+0x97)[0x555d224bee97]
 [22] /home/pyfr-develop/bin/python(_PyEval_EvalFrameDefault+0x72f1)[0x555d224ae031]
[23] /homepyfr-develop/bin/python(+0x5bdcb)[0x555d224a5dcb]
 [24] /home/pyfr-develop/bin/python(_PyEval_EvalFrameDefault+0x72f1)[0x555d224ae031]
[25] /home/pyfr-develop/bin/python(+0x12784a)[0x555d2257184a]
[26] /homepyfr-develop/bin/python(_PyFunction_Vectorcall+0x97)[0x555d224bee97]
 /home/pyfr-develop/bin/python(+
2

As per the develop installation guide you will need to update your version of libxsmm.

Regards, Freddie.

3

Ok, I updated my version of libxsmm. But there were mistakes.

File "/home/pyfr-develop/bin/pyfr", line 33, in <module>
    sys.exit(load_entry_point('pyfr==1.15.0', 'console_scripts', 'pyfr')())
  File "/home/pyfr-develop/lib/python3.9/site-packages/pyfr-1.15.0-py3.9.egg/pyfr/__main__.py", line 125, in main
  File "/home/pyfr-develop/lib/python3.9/site-packages/pyfr-1.15.0-py3.9.egg/pyfr/__main__.py", line 269, in process_run
  File "/home/pyfr-develop/lib/python3.9/site-packages/pyfr-1.15.0-py3.9.egg/pyfr/__main__.py", line 254, in _process_common
  File "/home/pyfr-develop/lib/python3.9/site-packages/pyfr-1.15.0-py3.9.egg/pyfr/solvers/__init__.py", line 14, in get_solver
  File "/home/pyfr-develop/lib/python3.9/site-packages/pyfr-1.15.0-py3.9.egg/pyfr/integrators/__init__.py", line 34, in get_integrator
  File "/home/pyfr-develop/lib/python3.9/site-packages/pyfr-1.15.0-py3.9.egg/pyfr/integrators/std/controllers.py", line 11, in __init__
  File "/home/pyfr-develop/lib/python3.9/site-packages/pyfr-1.15.0-py3.9.egg/pyfr/integrators/std/base.py", line 26, in __init__
  File "/home/pyfr-develop/lib/python3.9/site-packages/pyfr-1.15.0-py3.9.egg/pyfr/solvers/base/system.py", line 67, in commit
  File "/home/pyfr-develop/lib/python3.9/site-packages/pyfr-1.15.0-py3.9.egg/pyfr/solvers/base/system.py", line 213, in _gen_kernels
  File "/hom/pyfr-develop/lib/python3.9/site-packages/pyfr-1.15.0-py3.9.egg/pyfr/solvers/navstokes/inters.py", line 98, in <lambda>
  File "/home/pyfr-develop/lib/python3.9/site-packages/pyfr-1.15.0-py3.9.egg/pyfr/backends/base/backend.py", line 174, in kernel
  File "/home/pyfr-develop/lib/python3.9/site-packages/pyfr-1.15.0-py3.9.egg/pyfr/backends/base/kernels.py", line 157, in kernel_meth
  File "/home/pyfr-develop/lib/python3.9/site-packages/pyfr-1.15.0-py3.9.egg/pyfr/backends/openmp/provider.py", line 73, in _build_kernel
  File "/home/pyfr-develop/lib/python3.9/site-packages/pyfr-1.15.0-py3.9.egg/pyfr/util.py", line 39, in newmeth
pytools.prefork.ExecError: error invoking 'gcc -shared -std=c99 -Ofast -march=native -fopenmp -fPIC -o libtmp.so tmp.c -lm': status 1 invoking 'gcc -shared -std=c99 -Ofast -march=native -fopenmp -fPIC -o libtmp.so tmp.c -lm': tmp.c: In function ‘bccent’:
tmp.c:17:39: error: expected ‘read’, ‘write’, ‘update’, ‘capture’, ‘seq_cst’, ‘acq_rel’, ‘release’, ‘relaxed’ or ‘hint’ clause
   17 | #define atomic_min_fpdtype(addr, val) _Pragma("omp atomic compare") if ((val) < *(addr)) { *(addr) = (val); }
      |                                       ^~~~~~~
tmp.c:77:1: note: in expansion of macro ‘atomic_min_fpdtype’
   77 | atomic_min_fpdtype(&entmin_lhs_v[entmin_lhs_vix[BLK_IDX + X_IDX]], entmin_lhs);
      | ^~~~~~~~~~~~~~~~~~
tmp.c:17:39: error: expected end of line before ‘compare’
   17 | #define atomic_min_fpdtype(addr, val) _Pragma("omp atomic compare") if ((val) < *(addr)) { *(addr) = (val); }
      |                                       ^~~~~~~
tmp.c:77:1: note: in expansion of macro ‘atomic_min_fpdtype’
   77 | atomic_min_fpdtype(&entmin_lhs_v[entmin_lhs_vix[BLK_IDX + X_IDX]], entmin_lhs);
      | ^~~~~~~~~~~~~~~~~~
tmp.c:17:69: error: expected expression before ‘if’
   17 | #define atomic_min_fpdtype(addr, val) _Pragma("omp atomic compare") if ((val) < *(addr)) { *(addr) = (val); }
      |                                                                     ^~
tmp.c:77:1: note: in expansion of macro ‘atomic_min_fpdtype’
   77 | atomic_min_fpdtype(&entmin_lhs_v[entmin_lhs_vix[BLK_IDX + X_IDX]], entmin_lhs);
      | ^~~~~~~~~~~~~~~~~~
tmp.c:17:39: error: expected ‘read’, ‘write’, ‘update’, ‘capture’, ‘seq_cst’, ‘acq_rel’, ‘release’, ‘relaxed’ or ‘hint’ clause
   17 | #define atomic_min_fpdtype(addr, val) _Pragma("omp atomic compare") if ((val) < *(addr)) { *(addr) = (val); }
      |                                       ^~~~~~~
tmp.c:123:1: note: in expansion of macro ‘atomic_min_fpdtype’
  123 | atomic_min_fpdtype(&entmin_lhs_v[entmin_lhs_vix[BLK_IDX + X_IDX]], entmin_lhs);
      | ^~~~~~
4

As per the documentation PyFR now requires a compiler which supports OpenMP 5.1. Practically, this means GCC 12 or later.

Regards, Freddie.

5

Also if I’m not mistaken you’ll need to update to Python >= 3.10. It looks like you might be running 3.9.

6

Hi Freddie,

I’m seeing similar segmentation faults randomly appear causing the solver to crash during runs with the OpenMP backend under macOS (v. 13.4.1). You’ve highlighted the issue being with the compiler; though, I’m not exactly sure how best to install a compiler that supports OpenMP 5.1 on this platform other than attempting to build one from source. The latest Homebrew formula installs gcc-13; however, it only supports the OpenMP 4.5 specification released in Nov 2015. A snippet of the stack trace is below:

(venv) [zdavis@Minerva 2d-inc-cylinder]$ pyfr restart -b openmp -p inc-cylinder.pyfrm inc-cylinder-20.00.pyfrs 
  71.1% [+++++++=============>         ] 53.35/75.00 ela: 00:22:38 rem: 00:14:42
LIBXSMM_VERSION: feature_xsmm_v2-1.17-3645 (25693757)
APPL_M1/DP    TRY    JIT    STA    COL
   0..13     35     35      0      0 
  14..23      3      3      0      0 
  24..64      0      0      0      0 
Registry and code: 13 MB + 1 MB (gemm=38 gemv=2 spmdm=54)
Command: /opt/homebrew/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/Resources/Python.app/Contents/MacOS/Python /Users/zdavis/Applications/PyFR/pyfr/pyfr restart -b openmp -p inc-cylinder.pyfrm inc-cylinder-20.00.pyfrs
Uptime: 1359.814517 s
[Minerva:13918] *** Process received signal ***
[Minerva:13918] *** Process received signal ***
[Minerva:13918] Signal: Segmentation fault: 11 (11)
[Minerva:13918] Signal code: Invalid permissions (2)
[Minerva:13918] Failing at address: 0xffffffffea709c40
[Minerva:13918] Signal: Segmentation fault: 11 (11)
[Minerva:13918] Signal code: Invalid permissions (2)
[Minerva:13918] Failing at address: 0xb7812240
[Minerva:13918] [Minerva:13918] [ 0] [ 0] 0   libsystem_platform.dylib            0x0000000185e76a24 _sigtramp + 56
[Minerva:13918] [ 1] [Minerva:13918] *** Process received signal ***
[Minerva:13918] Signal: Segmentation fault: 11 (11)
[Minerva:13918] Signal code: Invalid permissions (2)
[Minerva:13918] Failing at address: 0xfffffffef7225740
[Minerva:13918] [ 0] [Minerva:13918] *** Process received signal ***
0   libsystem_platform.dylib            0x0000000185e76a24 _sigtramp + 56
0   libsystem_platform.dylib            0x0000000185e76a24 _sigtramp + 56
[Minerva:13918] [Minerva:13918] *** Process received signal ***
[Minerva:13918] Signal: Segmentation fault: 11 (11)
[Minerva:13918] Signal code: Invalid permissions (2)
[Minerva:13918] Failing at address: 0x97011f40
[Minerva:13918] Signal: Segmentation fault: 11 (11)
[Minerva:13918] Signal code: Invalid permissions (2)
[Minerva:13918] Failing at address: 0x42a915040
[Minerva:13918] [Minerva:13918] [Minerva:13918] [ 0] [Minerva:13918] *** Process received signal ***
[ 0] [ 1] 0   libsystem_platform.dylib            0x0000000185e76a24 _sigtramp + 56
0   libsystem_platform.dylib            0x0000000185e76a24 _sigtramp + 56
[ 1] [Minerva:13918] [Minerva:13918] [Minerva:13918] *** Process received signal ***
[ 1] [ 1] [Minerva:13918] *** Process received signal ***
[Minerva:13918] Signal: Segmentation fault: 11 (11)
[Minerva:13918] Signal code: Invalid permissions (2)
[Minerva:13918] Failing at address: 0x3bf8120c0
[Minerva:13918] Signal: Segmentation fault: 11 (11)
[Minerva:13918] Signal code: Invalid permissions (2)
[Minerva:13918] Failing at address: 0xb44483de0
[Minerva:13918] Signal: Segmentation fault: 11 (11)
[Minerva:13918] Signal code: Invalid permissions (2)
[Minerva:13918] Failing at address: 0xfffffffe58970240
[Minerva:13918] [ 0] [Minerva:13918] [ 0] [Minerva:13918] [Minerva:13918] *** Process received signal ***
0   libsystem_platform.dylib            0x0000000185e76a24 _sigtramp + 56
[Minerva:13918] *** Process received signal ***
[Minerva:13918] [Minerva:13918] Signal: Segmentation fault: 11 (11)
[Minerva:13918] Signal code: Invalid permissions (2)
[Minerva:13918] Failing at address: 0x37cedcec0
[ 1] 0   libsystem_platform.dylib            0x0000000185e76a24 _sigtramp + 56
0   lib225f64daec9cb3579017faac6c91d66d 0x0000000130b13b30 bccflux._omp_fn.0 + 96
[Minerva:13918] [ 0] [Minerva:13918] [ 0] [ 1] 0   libsystem_platform.dylib            0x0000000185e76a24 _sigtramp + 56
0   libsystem_platform.dylib            0x0000000185e76a24 _sigtramp + 56
[Minerva:13918] *** Process received signal ***
[Minerva:13918] Signal: Segmentation fault: 11 (11)
[Minerva:13918] Signal code: Invalid permissions (2)
[Minerva:13918] Failing at address: 0x4a1091dc0
[Minerva:13918] *** Process received signal ***
[Minerva:13918] [Minerva:13918] [Minerva:13918] Signal: Segmentation fault: 11 (11)
[Minerva:13918] Signal code: Invalid permissions (2)
[Minerva:13918] Failing at address: 0xfffffffd7f41a2c0
0   lib225f64daec9cb3579017faac6c91d66d 0x0000000130b13b30 bccflux._omp_fn.0 + 96
[Minerva:13918] [Minerva:13918] Signal: Segmentation fault: 11 (11)
[Minerva:13918] Signal code: Invalid permissions (2)
[Minerva:13918] Failing at address: 0x31b284240
[Minerva:13918] *** Process received signal ***
[ 1] [Minerva:13918] [ 2] 0   lib225f64daec9cb3579017faac6c91d66d 0x0000000130b13b30 bccflux._omp_fn.0 + 96
[ 1] [Minerva:13918] [Minerva:13918] [Minerva:13918] Signal: Segmentation fault: 11 (11)
[Minerva:13918] Signal code: Invalid permissions (2)
[Minerva:13918] Failing at address: 0xfffffffe58970240
0   lib225f64daec9cb3579017faac6c91d66d 0x0000000130b13cb4 bccflux._omp_fn.0 + 484
[ 0] [Minerva:13918] [ 0] [Minerva:13918] [Minerva:13918] [Minerva:13918] [ 2] [ 0] 0   libsystem_platform.dylib            0x0000000185e76a24 _sigtramp + 56
[Minerva:13918] [ 2] [ 0] [ 1] 0   libsystem_platform.dylib            0x0000000185e76a24 _sigtramp + 56
0   libsystem_platform.dylib            0x0000000185e76a24 _sigtramp + 56
0   lib225f64daec9cb3579017faac6c91d66d 0x0000000130b13cb4 bccflux._omp_fn.0 + 484
[Minerva:13918] [Minerva:13918] 0   libsystem_platform.dylib            0x0000000185e76a24 _sigtramp + 56
[ 1] [ 2] [Minerva:13918] [Minerva:13918] [ 1] [ 2] [ 1] 0   libgomp.1.dylib                     0x0000000131a10dc4 GOMP_parallel + 84
[Minerva:13918] [ 3] 0   lib225f64daec9cb3579017faac6c91d66d 0x0000000130b13b30 bccflux._omp_fn.0 + 96
[Minerva:13918] [ 2] 0   libgomp.1.dylib                     0x0000000131a18298 gomp_thread_start + 308
[Minerva:13918] [ 3] 0   libsystem_pthread.dylib             0x0000000185e47fa8 _pthread_start + 148
[Minerva:13918] [ 4] 0   libsystem_pthread.dylib             0x0000000185e42da0 thread_start + 8
[Minerva:13918] *** End of error message ***
0   lib225f64daec9cb3579017faac6c91d66d 0x0000000130b13b30 bccflux._omp_fn.0 + 96
[Minerva:13918] [ 2] Segmentation fault: 11

It’s not much of an issue, but more of a hassle, as simply restarting the simulation from the latest restart file allows the simulation to continue.

Also, I realize this would be a very niche use case, but would it be possible at this point to provide a metal backend option? I would be curious if porting the OpenCL kernel to metal would save any time, or if it would be something that would need to be written from scratch.

7

On my M1 Max Mac with GCC 12 from Macports:

OMP_NUM_THREADS=4 pyfr run -bopenmp -p inc-cylinder.pyfrm inc-cylinder.ini
 100.0% [=========================================================================>] 75.00/75.00 ela: 00:13:05 rem: 00:00:00

There is a Metal backend in the latest develop version. It works quite nicely, although note that it only supports single precision and does not confer any benefit for small problem sizes.

Regards, Freddie.

8

I see the same results as Freddie with GCC 12 from homebrew.

9

Thanks for the feedback Freddie & Will. I think the issue may be related to the OMP_NUM_THREADS parameter setting that I had been using for this test case. It works great with a value of 4–not so much with a value of 16…

Thanks for the heads up with the metal backend. It also works well–too bad there is no FP64 support in Metal.

10

The OpenMP implementations on macOS currently seem to be a little finicky. Case in point:

❯ pyfr partition 4 -ebalanced inc-cylinder.pyfrm .
❯ OMP_NUM_THREADS=1 mpirun -np 4 pyfr run -bopenmp -p inc-cylinder.pyfrm inc-cylinder.ini
 100.0% [=============================>] 75.00/75.00 ela: 00:02:45 rem: 00:00:00

which is substantially faster. This is not something we observe to the same degree on Linux, even though the code itself is identical.

Regards, Freddie.

11

Huh. That is strange. I wouldn’t think that would be supported with the openmp backend. Isn’t mpirun an MPI wrapper? Wouldn’t PyFR be using the installed MPI library in this instance instead of OpenMP? Does this performance differential scale for larger problems as well (as in one should always use this approach, rather than using a single partition)?

12

PyFR supports hybrid MPI/OpenMP without issues. Normally on a two socket system we suggest one MPI rank per socket and then using enough OpenMP threads per rank to fully populate a single socket.

The only issues with the MPI approach are slightly higher overheads (in theory), sensitivity to partitioning, and difficulties with heterogeneous systems (E and P cores).

For larger problems the high overheads associated with OpenMP on macOS appear to amortise out. We do not observe these problems on other platforms.

Regards, Freddie.