Problem with "2d-inc-cylinder"

Errors
1

Dear All,

I’m having problem with running “2d-inc-cylinder” test case. The other two cases including 2d-euler and 2d-couette are just fine. Here is the error I get when running with openmp backend:

free(): invalid pointer
[gra-login3:17972] *** Process received signal ***
[gra-login3:17972] Signal: Aborted (6)
[gra-login3:17972] Signal code:  (-6)
[gra-login3:17972] [ 0] /XXX/gentoo/2020/lib64/libpthread.so.0(+0x130f0)[0x2ad8aadc00f0]
[gra-login3:17972] [ 1] /XXX/gentoo/2020/lib64/libc.so.6(gsignal+0x141)[0x2ad8aae0c901]
[gra-login3:17972] [ 2] /XXX/gentoo/2020/lib64/libc.so.6(abort+0x127)[0x2ad8aadf656b]
[gra-login3:17972] [ 3] /XXX/gentoo/2020/lib64/libc.so.6(+0x7b0a8)[0x2ad8aae4f0a8]
[gra-login3:17972] [ 4] /XXX/gentoo/2020/lib64/libc.so.6(+0x8244a)[0x2ad8aae5644a]
[gra-login3:17972] [ 5] /XXX/gentoo/2020/lib64/libc.so.6(+0x83e3c)[0x2ad8aae57e3c]
[gra-login3:17972] [ 6] /XXX/ENV/lib/python3.10/site-packages/numpy/core/_multiarray_umath.cpython-310-x86_64-linux-gnu.so(+0x16de02)[0x2ad8abe5ce02]
[gra-login3:17972] [ 7] /XXX/easybuild/software/2020/avx2/Core/python/3.10.2/lib/libpython3.10.so.1.0(+0x14d579)[0x2ad8aa931579]
[gra-login3:17972] [ 8] /XXX/easybuild/software/2020/avx2/Core/python/3.10.2/lib/libpython3.10.so.1.0(+0x14d579)[0x2ad8aa931579]
[gra-login3:17972] [ 9] /XXX/easybuild/software/2020/avx2/Core/python/3.10.2/lib/libpython3.10.so.1.0(+0x14db44)[0x2ad8aa931b44]
[gra-login3:17972] [10] /XXX/easybuild/software/2020/avx2/Core/python/3.10.2/lib/libpython3.10.so.1.0(+0x13964e)[0x2ad8aa91d64e]
[gra-login3:17972] [11] /XXX/easybuild/software/2020/avx2/Core/python/3.10.2/lib/libpython3.10.so.1.0(+0x152b39)[0x2ad8aa936b39]
[gra-login3:17972] [12] /XXX/easybuild/software/2020/avx2/Core/python/3.10.2/lib/libpython3.10.so.1.0(+0x24218c)[0x2ad8aaa2618c]
[gra-login3:17972] [13] /XXX/easybuild/software/2020/avx2/Core/python/3.10.2/lib/libpython3.10.so.1.0(+0x1d3bfd)[0x2ad8aa9b7bfd]
[gra-login3:17972] [14] /XXX/easybuild/software/2020/avx2/Core/python/3.10.2/lib/libpython3.10.so.1.0(_PyFunction_Vectorcall+0xcab)[0x2ad8aa9db2cb]
[gra-login3:17972] [15] /XXX/easybuild/software/2020/avx2/Core/python/3.10.2/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x32d)[0x2ad8aa9e470d]
[gra-login3:17972] [16] /XXX/easybuild/software/2020/avx2/Core/python/3.10.2/lib/libpython3.10.so.1.0(_PyFunction_Vectorcall+0x24d)[0x2ad8aa9da86d]
[gra-login3:17972] [17] /XXX/easybuild/software/2020/avx2/Core/python/3.10.2/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x623)[0x2ad8aa9e4a03]
[gra-login3:17972] [18] /XXX/easybuild/software/2020/avx2/Core/python/3.10.2/lib/libpython3.10.so.1.0(_PyFunction_Vectorcall+0x24d)[0x2ad8aa9da86d]
[gra-login3:17972] [19] /XXX/easybuild/software/2020/avx2/Core/python/3.10.2/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x623)[0x2ad8aa9e4a03]
[gra-login3:17972] [20] /XXX/easybuild/software/2020/avx2/Core/python/3.10.2/lib/libpython3.10.so.1.0(_PyFunction_Vectorcall+0x24d)[0x2ad8aa9da86d]
[gra-login3:17972] [21] /XXX/easybuild/software/2020/avx2/Core/python/3.10.2/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x623)[0x2ad8aa9e4a03]
[gra-login3:17972] [22] /XXX/easybuild/software/2020/avx2/Core/python/3.10.2/lib/libpython3.10.so.1.0(_PyFunction_Vectorcall+0x24d)[0x2ad8aa9da86d]
[gra-login3:17972] [23] /XXX/easybuild/software/2020/avx2/Core/python/3.10.2/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x623)[0x2ad8aa9e4a03]
[gra-login3:17972] [24] /XXX/easybuild/software/2020/avx2/Core/python/3.10.2/lib/libpython3.10.so.1.0(_PyFunction_Vectorcall+0x24d)[0x2ad8aa9da86d]
[gra-login3:17972] [25] /XXX/easybuild/software/2020/avx2/Core/python/3.10.2/lib/libpython3.10.so.1.0(+0x1fceeb)[0x2ad8aa9e0eeb]
[gra-login3:17972] [26] /XXX/easybuild/software/2020/avx2/Core/python/3.10.2/lib/libpython3.10.so.1.0(+0x1b7081)[0x2ad8aa99b081]
[gra-login3:17972] [27] /XXX/easybuild/software/2020/avx2/Core/python/3.10.2/lib/libpython3.10.so.1.0(PyObject_Call+0xc1)[0x2ad8aa9380a1]
[gra-login3:17972] [28] /XXX/easybuild/software/2020/avx2/Core/python/3.10.2/lib/libpython3.10.so.1.0(_PyEval_EvalFrameDefault+0x3db8)[0x2ad8aa9e8198]
[gra-login3:17972] [29] /XXX/easybuild/software/2020/avx2/Core/python/3.10.2/lib/libpython3.10.so.1.0(_PyFunction_Vectorcall+0x76f)[0x2ad8aa9dad8f]
[gra-login3:17972] *** End of error message ***
/XXX/ENV/lib/python3.10/site-packages/pytools/prefork.py:92: UserWarning: Prefork server exiting upon apparent death of parent
  warn(f"{who} exiting upon apparent death of {partner}")
Aborted

I use PyFR 1.14. How can I fix the problem?

Kind regards
syavash

2

I don’t think this is necessarily a PyFR problem, but can you try reproducing this using the current develop branch from git?

3

Thanks Will.

I am not really sure. I tried to install PyFR from source but I had several problems. I consulted HPC admin and they recommended to install it on a virtual environment via pip from their wheel. They don’t have GCC 12 or other compilers with support of OpenMP 5.1, so they put ver. 1.14 on the wheel.

They also investigated the problem with “2d-inc-cylinder” and told me there is some bug, and suggested to open an issue about it. I am not really sure what is causing the problem either…

Kind regards
syavash

4

In the past when I’ve been in similar situations what I’ve done is compile the dependencies myself (in fact when I’m on a cluster, unless there is some special vendor MPI, I normally compile everything myself).

You can follow the steps below

Compiler etal.

The things you would need in order are:

  1. GCC (this is a useful guide: InstallingGCC - GCC Wiki)
  2. UCX with cuda/hip (this is a communication service, transport and protocol layer)
  3. OpenMPI with UCX ()
  4. HDF5 but set CC=$(which mpicc), CXX=$(which mpic++), and FC=$(which mpif90), this with enable parallel support.
  5. libffi (often this isn’t installed on a system with the development headers)
  6. openssl (needed for python later)
  7. Python 3.11 with openssl

I would suggest compiling these with the prefix $HOME/.local/ or if you have Lmod on the system I would put them in $HOME/.local/pkg/{name}/{version} ie /home/wtrojak/.local/pkg/hdf5/1.14.0

Of course, as you go along you should update your paths etc, again if you have Lmod you can do this nicely with modulefiles.

PyFR

  1. make a venv then:
    7a. cd some_venv && . ./bin/activate && pip install pyfr
    7b. pip uninstall pyfr
    7c. git clone ${pyfr_repo}
    7d. pip install -e PyFR

7a-7b uses pip to install the pyfr and so gets the dependencies.