Running PyFR v2.0.0 in Parallel

Just Starting
1

Running PyFR v2.0.0 requires several extra dependencies to be correctly configured. Namely mpi4py and a partitioner such as METIS or SCOTCH.

mpi4py

Depending on your platform there is a reasonable chance that mpi4py is broken. A simple test for this is to run:

$ mpiexec -np 3 python -c 'from mpi4py import MPI; print(MPI.COMM_WORLD.rank)'
0
1
2

If you do not see the numbers 0, 1, and 2 output to the screen (in any order) then your mpi4py package is broken. The typical cause for this is mpiexec belonging to a different MPI distribution than what mpi4py was compiled against. The solution for this is to recompile mpi4py from scratch (not using a package from conda or pip).

METIS

Between versions 5.1 and 5.2 METIS broke ABI compatibility. Moreover, the METIS library provides no means of determining its version at runtime. This regrettableā€”and needlessā€”change on the part of METIS has caused issues for a variety of projects, including PyFR. As it is not possible to support both versions we made the decision with v2.0.0 to only support 5.2.x. This is newer than what most distributions provide. As such if you wish to use METIS it is likely that you will need to compile it from source.

Furthermore, since the 5.1.x series the METIS build scripts produce a broken shared library. In particular, libmetis.so fails to link against libGKlib.so even though it is a dependency. This goes against best practice and has caused issues for numerous projects, including PyFR. The following patch:

must therefore be applied. If METIS has been built correctly then:

PYFR_METIS_LIBRARY_PATH=/path/to/your/libmetis.so
python -c "from ctypes import CDLL; CDLL('${PYFR_METIS_LIBRARY_PATH}')"

should run without error.

SCOTCH

In order to make partitioning reproducible PyFR v2.0.0 now depends on SCOTCH v7. This, again, is newer than what most distributions ship. As such, you may also need to compile SCOTCH from source. As with METIS, the SCOTCH build scripts go against best practices and yield broken shared libraries by default. Most distributions (including Ubuntu and Gentoo) patch their SCOTCH packages to avoid these issues. On Linux the patchelf:

tool can be used to add libscotcherr.so as a dependency to libscotch.so as:

patchelf --add-needed libscotcherr.so libscotch.so

If SCOTCH has been built correctly then:

PYFR_SCOTCH_LIBRARY_PATH=/path/to/your/libscotch.so
python -c "from ctypes import CDLL; CDLL('${PYFR_SCOTCH_LIBRARY_PATH}')"

should run without error.

Regards, Freddie.

3 Likes