# Freezing at Start-up **URL:** \ **Category:** Errors\ **Created:** [12 September 2024 05:11 UTC](https://pyfr.discourse.group/t/freezing-at-start-up/1163 "2024-09-12T05:11:46Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![pyfr\_starting\_man](https://avatars.discourse-cdn.com/v4/letter/p/d26b3c/32.png) [@pyfr\_starting\_man](https://pyfr.discourse.group/u/pyfr_starting_man)\ **Post date:** [12 September 2024 05:11 UTC](https://pyfr.discourse.group/t/freezing-at-start-up/1163/1 "2024-09-12T05:11:46Z")
Hi, I have been having this issue running my channel flow case where the simulation completely freezes up. I know this because I have modified the source code to print out statements for every few iterations so I know what time step it is on, and the progress bar is completely freezing as well. I am running my case across 10 nodes on HPC cluster with 20 total GPUs. My mesh is somewhat large (~2,000,000 elements). I have been running my case for 4 hours successfully, and need to restart it from my last .pyfrs file, but it is immediately freezing after ~20 iterations or so. This is not making sense to me, as it is the same exact case I ran just a few hours ago with no troubles at all. This has also been happening to me when trying to run other cases as well. I know I saw somewhere on the forum that rebuilding UCX and OpenMPI solved the issue, but that did not work for me. I looked at the backtrace output for my running processes and see that the backtrace indicates that the process is stuck in a low-level MPI communication call involving UCX. Here is some of the backtrace: Loaded symbols for /apps/openmpi-cuda/intel/19.0/4.1.2-hpcx/lib/openmpi/mca\_coll\_han.so Reading symbols from /apps/openmpi-cuda/intel/19.0/4.1.2-hpcx/lib/openmpi/mca\_coll\_inter.so…done. Loaded symbols for /apps/openmpi-cuda/intel/19.0/4.1.2-hpcx/lib/openmpi/mca\_coll\_inter.so Reading symbols from /apps/openmpi-cuda/intel/19.0/4.1.2-hpcx/lib/openmpi/mca\_coll\_libnbc.so…done. Loaded symbols for /apps/openmpi-cuda/intel/19.0/4.1.2-hpcx/lib/openmpi/mca\_coll\_libnbc.so Reading symbols from /apps/openmpi-cuda/intel/19.0/4.1.2-hpcx/lib/openmpi/mca\_coll\_self.so…done. Loaded symbols for /apps/openmpi-cuda/intel/19.0/4.1.2-hpcx/lib/openmpi/mca\_coll\_self.so Reading symbols from /apps/openmpi-cuda/intel/19.0/4.1.2-hpcx/lib/openmpi/mca\_coll\_sm.so…done. Loaded symbols for /apps/openmpi-cuda/intel/19.0/4.1.2-hpcx/lib/openmpi/mca\_coll\_sm.so Reading symbols from /apps/openmpi-cuda/intel/19.0/4.1.2-hpcx/lib/libmca\_common\_sm.so.40…done. Loaded symbols for /apps/openmpi-cuda/intel/19.0/4.1.2-hpcx/lib/libmca\_common\_sm.so.40 Reading symbols from /apps/openmpi-cuda/intel/19.0/4.1.2-hpcx/lib/openmpi/mca\_coll\_sync.so…done. Loaded symbols for /apps/openmpi-cuda/intel/19.0/4.1.2-hpcx/lib/openmpi/mca\_coll\_sync.so Reading symbols from /apps/openmpi-cuda/intel/19.0/4.1.2-hpcx/lib/openmpi/mca\_coll\_tuned.so…done. Loaded symbols for /apps/openmpi-cuda/intel/19.0/4.1.2-hpcx/lib/openmpi/mca\_coll\_tuned.so Reading symbols from /apps/openmpi-cuda/intel/19.0/4.1.2-hpcx/lib/openmpi/mca\_coll\_cuda.so…done. Loaded symbols for /apps/openmpi-cuda/intel/19.0/4.1.2-hpcx/lib/openmpi/mca\_coll\_cuda.so Reading symbols from /apps/openmpi-cuda/intel/19.0/4.1.2-hpcx/lib/openmpi/mca\_coll\_hcoll.so…done. Loaded symbols for /apps/openmpi-cuda/intel/19.0/4.1.2-hpcx/lib/openmpi/mca\_coll\_hcoll.so Reading symbols from /apps/hpcx/2.12.0/hcoll/lib/libhcoll.so.1…done. Loaded symbols for /apps/hpcx/2.12.0/hcoll/lib/libhcoll.so.1 Reading symbols from /lib64/libnsl.so.1…Reading symbols from /usr/lib/debug/usr/lib64/libnsl-2.17.so.debug…done. done. Loaded symbols for /lib64/libnsl.so.1 Reading symbols from /apps/hpcx/2.12.0/hcoll/lib/libocoms.so.0…(no debugging symbols found)…done. Loaded symbols for /apps/hpcx/2.12.0/hcoll/lib/libocoms.so.0 Reading symbols from /apps/openmpi-cuda/intel/19.0/4.1.2-hpcx/lib/openmpi/mca\_osc\_sm.so…done. Loaded symbols for /apps/openmpi-cuda/intel/19.0/4.1.2-hpcx/lib/openmpi/mca\_osc\_sm.so Reading symbols from /apps/openmpi-cuda/intel/19.0/4.1.2-hpcx/lib/openmpi/mca\_osc\_pt2pt.so…done. Loaded symbols for /apps/openmpi-cuda/intel/19.0/4.1.2-hpcx/lib/openmpi/mca\_osc\_pt2pt.so Reading symbols from /apps/openmpi-cuda/intel/19.0/4.1.2-hpcx/lib/openmpi/mca\_osc\_rdma.so…done. Loaded symbols for /apps/openmpi-cuda/intel/19.0/4.1.2-hpcx/lib/openmpi/mca\_osc\_rdma.so Reading symbols from /apps/openmpi-cuda/intel/19.0/4.1.2-hpcx/lib/openmpi/mca\_osc\_ucx.so…done. Loaded symbols for /apps/openmpi-cuda/intel/19.0/4.1.2-hpcx/lib/openmpi/mca\_osc\_ucx.so Reading symbols from /apps/hpcx/2.12.0/hcoll/lib/hcoll/hmca\_rcache\_ucs.so…(no debugging symbols found)…done. Loaded symbols for /apps/hpcx/2.12.0/hcoll/lib/hcoll/hmca\_rcache\_ucs.so Reading symbols from /apps/hpcx/2.12.0/hcoll/lib/hcoll/hmca\_bcol\_basesmuma.so…(no debugging symbols found)…done. Loaded symbols for /apps/hpcx/2.12.0/hcoll/lib/hcoll/hmca\_bcol\_basesmuma.so Reading symbols from /apps/hpcx/2.12.0/hcoll/lib/hcoll/hmca\_bcol\_ucx\_p2p.so…(no debugging symbols found)…done. Loaded symbols for /apps/hpcx/2.12.0/hcoll/lib/hcoll/hmca\_bcol\_ucx\_p2p.so Reading symbols from /apps/hpcx/2.12.0/hcoll/lib/hcoll/hmca\_mcast\_vmc.so…(no debugging symbols found)…done. Loaded symbols for /apps/hpcx/2.12.0/hcoll/lib/hcoll/hmca\_mcast\_vmc.so Reading symbols from /lib64/libcuda.so…(no debugging symbols found)…done. Loaded symbols for /lib64/libcuda.so Reading symbols from /apps/cuda/12.3.0/lib64/libnvrtc.so…(no debugging symbols found)…done. Loaded symbols for /apps/cuda/12.3.0/lib64/libnvrtc.so Reading symbols from /apps/cuda/12.3.0/lib64/libcublasLt.so…(no debugging symbols found)…done. Loaded symbols for /apps/cuda/12.3.0/lib64/libcublasLt.so Reading symbols from /apps/cuda/12.3.0/lib64/libnvrtc-builtins.so.12.3…(no debugging symbols found)…done. Loaded symbols for /apps/cuda/12.3.0/lib64/libnvrtc-builtins.so.12.3 0x00002abafe7bc209 in uct\_rc\_mlx5\_iface\_progress\_cyclic (arg=0x1ad0030) at rc/accel/rc\_mlx5\_iface.c:191 191 rc/accel/rc\_mlx5\_iface.c: No such file or directory. warning: File “/apps/gnu/8.4.0/lib64/libstdc++.so.6.0.25-gdb.py” auto-loading has been declined by your auto-load safe-path’ set to “$debugdir:$datadir/auto-load:/usr/bin/mono-gdb.py”. To enable execution of this file add add-auto-load-safe-path /apps/gnu/8.4.0/lib64/libstdc++.so.6.0.25-gdb.py line to your configuration file. To completely disable this security protection add set auto-load safe-path / line to your configuration file. For more information about this security protection see the “Auto-loading safe path” section in the GDB manual. E.g., run from the shell: info “(gdb)Auto-loading safe path” Missing separate debuginfos, use: debuginfo-install bzip2-libs-1.0.6-13.el7.x86\_64 elfutils-libelf-0.176-5.el7.x86\_64 elfutils-libs-0.176-5.el7.x86\_64 glibc-2.17-326.el7\_9.3.x86\_64 libattr-2.4.46-13.el7.x86\_64 libcap-2.22-11.el7.x86\_64 libffi-3.0.13-19.el7.x86\_64 libibverbs-58mlnx43-1.58307.x86\_64 libnl3-3.2.28-4.el7.x86\_64 libpciaccess-0.14-1.el7.x86\_64 librdmacm-58mlnx43-1.58307.x86\_64 nss-softokn-freebl-3.90.0-6.el7\_9.x86\_64 numactl-libs-2.0.12-5.el7\_9.x86\_64 nvidia-driver-latest-dkms-cuda-libs-535.104.05-1.el7.x86\_64 sssd-client-1.16.5-10.el7\_9.16.x86\_64 systemd-libs-219-78.el7\_9.9.x86\_64 xz-libs-5.2.2-2.el7\_9.x86\_64 zlib-1.2.7-21.el7\_9.x86\_64 (gdb) bt #0 0x00002abafe7bc209 in uct\_rc\_mlx5\_iface\_progress\_cyclic (arg=0x1ad0030) at rc/accel/rc\_mlx5\_iface.c:191 #1 0x00002abacc7301ca in ucs\_callbackq\_dispatch (cbq=) at /build-result/src/hpcx-v2.12-gcc-MLNX\_OFED\_LINUX-5-redhat7-cuda11-gdrcopy2-nccl2.12-x86\_64/ucx-master/src/ucs/datastruct/callbackq.h:211 #2 uct\_worker\_progress (worker=) at /build-result/src/hpcx-v2.12-gcc-MLNX\_OFED\_LINUX-5-redhat7-cuda11-gdrcopy2-nccl2.12-x86\_64/ucx-master/src/uct/api/uct.h:2647 #3 ucp\_worker\_progress (worker=0x15d1740) at core/ucp\_worker.c:2804 #4 0x00002abb06afbed4 in opal\_progress () at runtime/opal\_progress.c:231 #5 0x00002abb03e430a6 in ompi\_request\_default\_wait\_all (count=28114992, requests=0x0, statuses=0x2abafe7bc200 \) at …/opal/threads/wait\_sync.h:83 #6 0x00002abb03e71c63 in PMPI\_Waitall (count=28114992, requests=0x0, statuses=0x2abafe7bc200 \) at pwaitall.c:80 #7 0x00002abb038cca10 in \_\_pyx\_pw\_6mpi4py\_3MPI\_7Request\_27Waitall ( \_\_pyx\_v\_cls=0x1ad0030, \_\_pyx\_args=0x0, \_\_pyx\_nargs=46982621807104, \_\_pyx\_kwds=0x2abb0acd5130 \) at src/mpi4py/MPI.c:143690 #8 0x00000000005c55e3 in \_PyObject\_VectorcallTstate (kwnames=, nargsf=2, args=0x2abbdd9f81d8, callable=0x2abb02fe5e50, tstate=0xb31460) at ./Include/cpython/abstract.h:114 —Type to continue, or q to quit— ---
**Author:** ![fdw](https://avatars.discourse-cdn.com/v4/letter/f/a587f6/32.png) [@fdw](https://pyfr.discourse.group/u/fdw)\ **Post date:** [12 September 2024 11:35 UTC](https://pyfr.discourse.group/t/freezing-at-start-up/1163/2 "2024-09-12T11:35:16Z")
Just for the purposes of debugging please force OpenMPI/UCX to use TCP as a transport rather than Infiniband. Note that this will have a negative impact on performance (maybe 3-4x) but you should still be able to see if the problem still occurs. The syntax for this can vary but passing `--mca btl tcp,self` (or something along those lines) should do it. Regards, Freddie. ---
**Author:** ![pyfr\_starting\_man](https://avatars.discourse-cdn.com/v4/letter/p/d26b3c/32.png) [@pyfr\_starting\_man](https://pyfr.discourse.group/u/pyfr_starting_man)\ **Post date:** [12 September 2024 13:27 UTC](https://pyfr.discourse.group/t/freezing-at-start-up/1163/3 "2024-09-12T13:27:52Z")
Thank you. I am using srun, so I set export OMPI\_MCA\_btl=tcp,self and it is running now. Now that we know what the issue is, is there anything I can do to not take the 3-4x performance penalty? ---
**Author:** ![fdw](https://avatars.discourse-cdn.com/v4/letter/f/a587f6/32.png) [@fdw](https://pyfr.discourse.group/u/fdw)\ **Post date:** [12 September 2024 13:51 UTC](https://pyfr.discourse.group/t/freezing-at-start-up/1163/4 "2024-09-12T13:51:29Z")
The problem is basically guaranteed to be with how Infiniband is configured on your cluster. PyFR is entirely ignorant of how things are actually transported over MPI (the MPI calls we make are always the same irrespective of what transport is being used). You will need to work with your system admin to diagnose the issue. Regards, Freddie.