# Simulation with Mesh Partitioning using SCOTCH **URL:** \ **Category:** Cases\ **Tags:** mesh\ **Created:** [22 August 2022 08:46 UTC](https://pyfr.discourse.group/t/simulation-with-mesh-partitioning-using-scotch/677 "2022-08-22T08:46:03Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![rs7](https://yyz2.discourse-cdn.com/free1/user_avatar/pyfr.discourse.group/rs7/32/405_2.png) [@rs7](https://pyfr.discourse.group/u/rs7)\ **Post date:** [22 August 2022 08:46 UTC](https://pyfr.discourse.group/t/simulation-with-mesh-partitioning-using-scotch/677/1 "2022-08-22T08:46:03Z")
I am trying to use PyFR with SCOTCH on my University Compute Cluster. Since I do not have sudo rights, I am not able to use the `sudo apt install` command. Hence, I have to install SCOTCH from their [GitHub repo](https://gitlab.inria.fr/scotch/scotch) code. I followed the instructions in the [INSTALL.txt](https://gitlab.inria.fr/scotch/scotch/-/blob/master/INSTALL.txt) file on the SCOTCH repo, and tried using installation using CMake. I created a Virtual Environment for my pyfr installation: ```auto module load python/3.9.6 python3 -m venv pyfr source pyfr/bin/activate module load openmpi module load paraview module load gcc/10 module load cuda cd pyfr ``` I typed the following commands after cloning the SCOTCH repo: ```auto mkdir build cd build cmake -DCMAKE_INSTALL_PREFIX=/path/to/install/ -DCMAKE_BUILD_TYPE=Release .. make -j 128 env CTEST_OUTPUT_ON_FAILURE=1 ctest make install ``` I installed pyfr using pip: ```auto pip install pyfr ``` The documentation says to specify the SCOTCH path using `PYFR_SCOTCH_LIBRARY_PATH=/path/to/libscotch.so`, but there is no _.so_ file in my _SCOTCH/lib_ folder. In the _euler\_vortex\_2d_ folder from PyFR examples, I run the following commands (as mentioned on the PyFR website: ```auto pyfr import euler_vortex_2d.msh euler_vortex_2d.pyfrm pyfr partition 2 euler_vortex_2d.pyfrm . ``` The partition command does not create any files (and I am not sure if it should create any more files). Regardless, I execute the command: ```auto mpiexec -n 2 pyfr run -b cuda -p euler_vortex_2d.pyfrm euler_vortex_2d.ini ``` I get an error as shown below: ![image](https://global.discourse-cdn.com/free1/uploads/pyfr/original/1X/07cfa0928f93063a25fe958f30602525b48e585d.png) When I run PyFR without partition, it runs correctly and produced _.pyfrs_ files: ```auto pyfr import euler_vortex_2d.msh euler_vortex_2d.pyfrm pyfr run -b cuda -p euler_vortex_2d.pyfrm euler_vortex_2d.ini ``` Note that in the case without partitioning, even if I run `mpiexec -n 1 pyfr run -b cuda -p euler_vortex_2d.pyfrm euler_vortex_2d.ini` , I get a similar error as in the above image. I am not sure if this is the case because of an error with OpenMPI or PyFR. ---
**Author:** ![WillT](https://avatars.discourse-cdn.com/v4/letter/w/bc79bd/32.png) [@WillT](https://pyfr.discourse.group/u/WillT)\ **Post date:** [22 August 2022 09:21 UTC](https://pyfr.discourse.group/t/simulation-with-mesh-partitioning-using-scotch/677/2 "2022-08-22T09:21:32Z")
If I have followed what you’ve done correctly, the libscotch.so should be located in `/path/to/install/lib`. But given that pyfr didn’t throw an error when you ran `pyfr partition ...` this is not likely to be the issue. It seems that there is an issue with your MPI installation. This first thing to check is is the MPI version module you loaded the same as the one you had loaded when you did pip install pyfr? This is becuase a vesion mismatch between the current version and the version used by mpi4py can cause issue. (In case you didn’t know you can do `module save ... ` and `module restore ...` to save and restore modules between sessions. I normally have a module save named the same as the venv.) ---
**Author:** ![rs7](https://yyz2.discourse-cdn.com/free1/user_avatar/pyfr.discourse.group/rs7/32/405_2.png) [@rs7](https://pyfr.discourse.group/u/rs7)\ **Post date:** [22 August 2022 09:37 UTC](https://pyfr.discourse.group/t/simulation-with-mesh-partitioning-using-scotch/677/3 "2022-08-22T09:37:35Z")
Thank you for your answer. Actually there is no file with the extension _.so_. So I am not sure which path to specify for `PYFR_SCOTCH_LIBRARY_PATH`. I think this is the main issue because of which the partitioning is not working, but I’m not sure. I checked the versions for OpenMPI and mpi4py before and after pyfr installation. Both versions are at 3.1.3, so I don’t think there is an issue with the version mismatch. ---
**Author:** ![WillT](https://avatars.discourse-cdn.com/v4/letter/w/bc79bd/32.png) [@WillT](https://pyfr.discourse.group/u/WillT)\ **Post date:** [22 August 2022 10:08 UTC](https://pyfr.discourse.group/t/simulation-with-mesh-partitioning-using-scotch/677/4 "2022-08-22T10:08:46Z")
So when you look in the lib or lib64 directory within the directory you pointed to for the prefix there are no libscotch files? Did you definitely run `make install`? Can you try a simple mpi4py test? Something like: ```python from mpi4py import MPI if __name__ == ' __main__': comm = MPI.COMM_WORLD print(f'rank {comm.rank} of {comm.size}') ``` run with `mpiexec -n 4 python mpi_test.py` ---
**Author:** ![rs7](https://yyz2.discourse-cdn.com/free1/user_avatar/pyfr.discourse.group/rs7/32/405_2.png) [@rs7](https://pyfr.discourse.group/u/rs7)\ **Post date:** [22 August 2022 10:24 UTC](https://pyfr.discourse.group/t/simulation-with-mesh-partitioning-using-scotch/677/5 "2022-08-22T10:24:41Z")
There is a libscotch.a file, but not libscotch.so file in the lib folder. I tried running your script, but I got the error: ![image](https://global.discourse-cdn.com/free1/uploads/pyfr/original/1X/5d05b67994ae62de607038f41711a4a1445ca3e3.png) ---
**Author:** ![WillT](https://avatars.discourse-cdn.com/v4/letter/w/bc79bd/32.png) [@WillT](https://pyfr.discourse.group/u/WillT)\ **Post date:** [22 August 2022 10:28 UTC](https://pyfr.discourse.group/t/simulation-with-mesh-partitioning-using-scotch/677/6 "2022-08-22T10:28:08Z")
Ok, so when you run cmake you need to enable shared and remake scotch. If that simple python scipt doesn’t work then it looks like there might be an issue with your MPI installation. ---
**Author:** ![mlaufer](https://yyz2.discourse-cdn.com/free1/user_avatar/pyfr.discourse.group/mlaufer/32/609_2.png) [@mlaufer](https://pyfr.discourse.group/u/mlaufer)\ **Post date:** [23 August 2022 20:42 UTC](https://pyfr.discourse.group/t/simulation-with-mesh-partitioning-using-scotch/677/7 "2022-08-23T20:42:56Z")
I agree with @WillT, you obviously have an MPI problem. You should consult the HPC admins at your institution and ask what MPI version is recommended for your cluster (MPICH, MVAPICH, Intel MPI, Open MPI, etc.), along with any needed runtime flags. Note: you do not need to use Open MPI as shown in the installation guide. MPI issues aside, I had other issues trying to work with the Scotch partitioner a few months back ([Link to post](https://pyfr.discourse.group/t/pyfr-1-13-0-scotch-mesh-decomposition-error/553/3). In the end I needed to specifically specify the Scotch partitioner with the `-pscotch` flag, otherwise it would fall back to Metis even though the Scotch lib path was defined.